EEPROM/Flash data logging with pointer

I want to use the external flash to log some measurements data, using the flashee-eeprom library ( https://github.com/m-mcgowan/spark-flashee-eeprom ). I figure I need to store a pointer to determine where to continue storing data in the flash memory. However, I’m concerned that, by storing the pointer on each write, the 100,000 write/erase cycle limit would be reached too soon. So, I’m looking for a more efficient solution.

The documentation for the Eeprom library ( http://docs.spark.io/firmware/#other-functions-eeprom ), which I think uses the internal flash, seems to suggest it isn’t subject to write-wear:

EEPROM
The EEPROM emulator allocates 100 bytes of the Spark Core’s built-in flash memory to act as EEPROM. Unlike "true EEPROM, flash doesn’t suffer from write “wear”. The EEPROM functions can be used to store small amounts of data in flash that will persist even after the Core resets after a deep sleep.

From what I thought I knew about flash memory, this doesn’t seem to be accurate. Can anyone confirm either way?

Someone else mentioned that some micro-controllers have a “power-off” interrupt, which can execute while the power ramps down after the power supply is removed. Is there such a thing available on the spark core? If there is, perhaps I could store the pointer to flash only when power is lost?

Has anyone else found a different solution to a similar problem?

Thanks.

Just to be clear, it’s the number of erases that are the limiting factor, not the number of writes. Normally you erase before writing so they are the same. However, flashee has some tricks to allow many writes before needing an erase.

Flashee does wear leveling spread out over the free pages so you get 100,000 erases multiplied by the number of free pages. With 384 pages to start with, that’s over 30,000,000 erases in total.

Plus it has a scheme “Address Erase” where a minimum of 7 writes can be performed without erasing the page. So this increases the number of writes to around 200,000,000.

You could increase this figure even more by writing the pointer in a certain way so you fill a whole page with pointers before erasing. A page in the addressErase scheme holds about 512 bytes. That’s 128 pointers per page, so now we are into billions of writes.

But there is a way to avoid all of this complexity :slight_smile:

  • add an increasing ID to each log you write.
  • store the write pointer in RAM, so you know the next place to write to while the core is powered on.
  • on startup you need to set that pointer. You do this by reading all the log records from memory and finding the one with the largest ID. That was the last log written.

So you know the location of where the next log should go and what ID it will have. With 32-bit IDs, that’s 4 billion logs. Quite a few! :slight_smile:

Another alternative is to use the circular buffer class built into flashee. This allows you to continually write to flash and read it back. The circular buffer manages the write and read pointers, so you could write as data is recieved and read when you’re ready to output the logs. The only caveat is that the read/write pointers are not saved. But you could use the log ID trick above to find the next place to write to.

A bit of a long reply - I hope that gives you some ideas! :slight_smile:

Good luck!
mat.

EDIT: To answer the question about the EEPROM library, it allows you to perform ca. 256 writes before performing an erase. So it does still have the wear, but doesn’t happen with each write.

1 Like

Thanks for your last reply.

I know this is now an old thread, but with changes on the Photon I need to revisit my design.
Specifically, the Photon lacks the external flash that is on the Core, so I’ve added a SPI Flash IC (much like this one that I believe is on the Core).

Is it possible to use Flashee with an external device like this to take advantage of your wear-levelling?
If so, how would I go about doing this?

Yes, it’s possible. We simply need a FlashDevice implementation that can talk to your SPI flash memory. If you have an API that’s already talking to the SPI flash then coding this is trivial. See the existing SparkFlashDevice in flashee for the details.

I hope that’s enough to get you started! :smile:

I’ve just started looking into this again after Christmas and I just need to clarify some things. Is the SparkFlashDevice you referred to supposed to be the SparkExternalFlashDevice implementation (in flashee-eeprom-impl.h)? I have created an implementation based on this to talk to the SPI flash, but I’m unsure how to create a WearLevelErase device around that implementation of Flashdevice (if that makes sense…).

Now I’m wondering if I even need wear levelling, if I just log to consecutive bytes in flash until it is full, then wrap to the first block and erase each block before writing the first value into it?

Yes, sorry, that’s the correct name - I was doing this from memory (plus I know it was the only device name beginning with Spark. :wink: )

If you create your SPI flash object as the default flash object (instead of SparkExternalFlashDevice) then the system will take care of wrapping it for creating wearLevelErase and addressLevelErase devices.

Thanks, I see it now.