EEPROM / FLASH access library

In varoius threads, such as

there's talk of an EEPROM library. Is this something that will be done during this sprint?

If not, I'd like to help out.

I'm going to write the library access to the CC3000's eeprom, both with the same high and low level apis found on the Arudino.

The EEPROM in the CC3000 and external Flash both provide non-volatile storage. However, the flash may be more dangerous with some code, since it has to flash a whole a block at a time. (e.g. code writing to increasing addresses one byte at a time.)

Some implementation details: The CC3000 eeprom access would use the 2nd user file descriptor as it's storage location. Since it comes after the spark descriptor, I wonder if we should increase the space in the spark eeprom file to allow additional values to be stored there in later firmware. Otherwise, if a new firmware release comes out that requires an extra byte or two, old content in user space eeprom will be invalid? Increasing the size of the spark core eeprom file to (say) 512bytes will offer room for growth, so firmware upgrades to add more content in eeprom if they need to without affecting the user code's eeprom storage.

By default the spark EEPROM library would write to the spark core's EEPROM. This aligns as closely as possible to the arduino behaviour, but we get several times more EEPROM capacity.

For users that don't care about the block writes, or who want access to even more non volatile storage, they could use the same API, but switch the "back end". eeprom_select(NV_FLASH) can be used to instruct that subsequent EEPROM apis use the external flash rather than the CC3000's eeprom. This would be a spark-specific addition, but means deveolopers on spark now have a choice about where the sketch stores eprom data, while maintaining code-compatibility with existing sources.

For clients wanting directly access to the flash, working with the page-level flash, there is no arduino library to emulate (since the eeprom library doesn't require whole page erases.) Hence, a new API for writing to the user area in the external Flash will be created.

So, in a Nutshell an Eeprom library that can let existing Arduino code target both devices, plus a flash-specific API for working with block-level erases.

Spark team: If you've sketched out designs for this, any hints or pointers would be useful. I will be building this soon, so let me know if I should do this, or wait for the official support in the spark.

2 Likes

Great idea! +1

Frido.

@mdma would be awesome to have your help here. No particular specifics about how we were planning on implementing this, other than we were going to use the EEPROM space thatā€™s available to the user on the CC3000, and replicate the Arduino EEPROM library (EEPROM.read(), EEPROM.write()). This isnā€™t particularly high on our backlog so if youā€™re aching for it, please issue a pull request!

A quick update on this: Iā€™d coded the eeprom library, but this managed to corrupt my core so that first it failed to reconnect with the spark cloud, and later, it now fails to initialize the CC3000 (stuck on white led.)

Iā€™ll continue looking for the cause, but right now eeprom access seems to be unreliable. (Itā€™s also tricky since we need to expand the spark allocated eeprom region from 16 bytes to 368 bytes on existing systems, and itā€™s that function that may have caused problems.)

Until I find how to make eeprom access stable, Iā€™m going to work on a flash memory equivalent.

I wouldnā€™t mess with the CC3000ā€™s EEPROM. When you have all that external flash for use, a few K of slow EEPROM is not worth the potential danger in my opinion.

Thereā€™s also the fact that, if you want to use some of the lower level system calls to program a list of APā€™s (so you donā€™t have to constantly SmartConfig when switching networks) itā€™s going to use the addressing youā€™d be using.

I say stick with the SPI flash and leave the CC3000 EEPROM alone.

1 Like

I agree for the most part. The CC3000 does have documented access points for the user to store files, although itā€™s not clear if the user is the ā€œsystemā€ or the ā€œapplicationā€, so it seemed ok to use it.

In fact, I did get it working and had all the tests pass. What seems unreliable is the initial setup of deleting or initializing files in the eeprom.

The attraction with eeprom is byte-level erase. With flash you have to erase an entire page. I will an access manager so that you donā€™t talk directly to flash. This will allow various schemes to be implemented, such as wear leveling, and schemes using redundancy to provide an efficient random read/write access (i.e. fewer page erases).

Flash memory is also particularly well suited to efficient Stream writes, so I will provide a stream interface to flash. This will be useful for writing logs, implementing circular buffers etc.

1 Like

Can people help here?

The arduino has eeprom, so sketches are used to Eeprom.write(addess,value) api. While thatā€™s possible with flash to a degree, it comes with some costs. Iā€™ll post a couple of entries next - please vote what you feel appropriate.

I need random access to some memory - like Eeprom.write(address,byte) in the arduino

3 Likes

I know itā€™s not eeprom, but I want a convenient access, like a stream, such as to write logs to.

e.g.

Stream str = eeprom.getStream(offset);
2 Likes

I know flash: I know what Iā€™m doing. I have it all under control. I want to manage individual erase/writes according to the page I write to and when it was last written.

Hi @mdma would love to get an update to where you are on this?

Hi, Iā€™ve stopped spark development while I wait for my JTAG shield. As I mentioned above, I got it working, but started to look at initialization, and attempted to reallocate the existing 16 bytes that are already used by the spark and expand them to 384 bytes (or so, the figure suggested by the spark team in another thread) as well as allocat ca 5K for the user.

I had test routines to delete and initialize the two user files in eeprom. After doing this a few times, my device would no longer connect to the cloud. Attempting to load the cc3000 patch provided on the spark github account worsened things, so that now the spark LED shows flashing white - doesnā€™t even get to wifi.

I would like to try to recover this broken core as best I can, or at least write some code so that I can dump the eeprom fully to a place in flash/serial and restore it, so that I can attempt a restore should the same thing happen to one of my other cores.

So thatā€™s where I am - lots of plans, but no forward motion until I get a JTAG shield so I can better debug code and see whatā€™s going on in there!

I recommend taking a look at an ST Application Note AN2594 for this. It deals with Flash having an order of magnitude (or more) less re-rwite life span than EEPROM and provides a working solution.

That said, I just read this ...

So the external Flash already has 10x the endurance of the internal STM32 Flash, IIRC. Still, with so much space available, I reckon it wouldn't hurt to take advantage of the system described in AN2594, all the same.

Someone wrote a libmaple class based on the above app note. I used this successfully on a Maple-mini, which has the same STM32 MCU as the Sparkcore. There is precious little documentation for the class, but I found it straight forward enough to put into use.

@mdma -- Perhaps you could take these ideas and apply them to your work?

By the way, if you have any trouble getting your ST-Link/V2 (assuming you chose such) and Programming shield to work reliably, I can probably help. I've just been through all that myself.


FWIW, AFAIK Leaflab's libmaple was the first serious attempt to bring Arduino style coding to the STM32F1xx chip. They did a great job of writing their system from the ground up, completely replacing the 'standard' ST libraries. IMO however, the Spark team's approach of building on top of the standard ST libraries is cleaner and easier to understand.

1 Like

@gruvin Thanks for these links and info - I read the app note - I was planning to implement something similar and wear levelling.

The scheme I was planning to use wouldnā€™t explicitly store addresses for variables, but instead use 8-way redundancy for each data item (so a 4k page stored 256 bytes of accessible ā€˜eepromā€™ data.) The advantage is that reads are very fast since you donā€™t have to scan an entire page to find a value and can read all the data in one hit.

It may be possible to have different page types so we can implement different algorithms and storage formats. E.g. my 8x redundancy, address+data pair and direct storage (for streamed data.)

I will definitely start with the maple library, which will allow me to hit the ground running with my own particular use case (porting an arduino app that uses eeprom over to the spark.)

Iā€™ll keep you posted!

OK. So long as you won't be erasing Flash banks any more often than absolutely necessary. As you no doubt know, block erasures are where the wear and tear on the Flash memory kicks in, versus actual EEPROM. The somewhat convoluted yet clever scheme in the AN2594 app note is designed specifically to address that issue.

I think I mentioned earlier that, if memory serves, the STM32 MCU Flash is rated to only 10,000 cycles, whereas @zachary noted that the external Flash chip is rated at 100,000, which is much more in line with standard EEPROM. However, re-writes of short data chunks within any given bank can still mess things up pretty quick.

The external Flash chip also requires SPI access, which is shared with the WiFi module. So there be dragons there, me thinks.

In summary then, as far as a general Wiring style EEPROM library goes, I think that an AN2594-based solution in main MCU Flash is best. Larger storage requirements will clearly require something else and there is indeed a lot more of that in the external Flash chip.

Meanwhile, I see that @satishgn has already implemented a 255 byte solution, using spare main MCU Flash ...

He doesn't give any credits to AN2594 in his code, but it appears to be directly based on same.

@gruvin Thanks for reminding about the missing citation but since itā€™s provided by STMicro and for STM32 products, their license is perfectly valid for use on spark products and most of our code is based on their free standard peripheral libraries and application notes. I have now included a link to their App note just in case someone is interested in going through the inner functionality.

@satishgn will I be able to use the library for an FRAM attached to the user SPI?

You're welcome! :stuck_out_tongue: Though, that 'reminder' was really just tongue and cheek on my part. That said, it is good to have the reference to the app note, since it's not immediately obvious what all that complex looking code is trying to achieve. Thanks again.

For my four cents worth ...

That wouldn't be the way to go. Let me explain ...

The Flash memory library written by @satishgn goes to quite some lengths to protect internal MCU Flash memory from excessive block erasure cycles, which is needed because the MCU's internal Flash has quite low erase/re-write life span -- data sheet says min. 10,000 cycles. That sounds like a lot. But with Flash, you have to erase an entire block, just to re-write a single byte. For example, if you change a data byte at address 0, then later a data byte at address 1, you actually have to erase the entire first bank of Flash and then re-write both values back. @satishgn's code works around this issue by maintaining two banks of data, erasing and switching to each bank as the other one become full.

None of that is the case for FRAM though, which has massively higher erase/re-write endurance than Flash and AFAIK can happily update singles words, without needing to erase entire banks.

So, whilst the Flash based 'safety algorithm' could be ported for use with external FRAM, it would IMO be excessive overkill todo so. A much simpler library could be written, instead.


What FRAM chip precisely (if any) are you thinking of using? I'd be keen to read its data sheet. If I buy some myself, perhaps we could get a library written together for it, too.

1 Like

I already bought 2 x Fujitsu FRAM, 64Kb for the shield Iā€™m working on.

You can see more on the other thread. We are having some discussion on the test code :slight_smile:

1 Like