How to store large datasets in external flash in P1?

Firmware
1

Hello Particle,
I want to store the samples from a analogPin every millisecond. The analog pin resolution is 12 bits, values ranging from 0-4095 so 2 bytes. The external flash size is 1 MB which is 1,000,000 bytes. Hence I should be able to store 500,000 short integers.

How can I use flashee library to do this? I tried experimenting with circular buffer API’s but I ended up with red flashes.

Is there a way to write to flash on the go, instead of specifying a struct and size and passing an address of it into the write() API?

Like FLASH.write(address, data); //where address can range from 0 - 1000000

On P1, with flash->write(addr, value), I am able to write and read from the address 129000. When I choose the address as 130000, the write/read fails.

2

@dheerajdake I haven't used the EEPROM yet although I will try to take a stab at a solution.

Referring to: GitHub - m-mcgowan/spark-flashee-eeprom: Eeprom emulation using external flash on Particle devices. Includes unit tests cross compiled to regular gcc, and on-device integration tests.

384 pages @ 4096 bytes each are available in the EEPROM. That's 1,048,576 bytes.

Two pieces of information to consider from the readme:

  • The spark external flash has 384 pages, each 4096 bytes in size and is managed as the device provided via the Devices::userFlash() method. Rather than hard-code these page count and page size constants, you can make the code more flexible for future changes by using: Devices::userFlash().pageSize() and Devices::userFlash().pageCount()
  • At present, the maximum contiguous area that the Wear Leveling and Address Erase schemes can occupy is 1MB (256 pages). This is to keep runtime memory overhead to a minimum.

A workaround is to create 2 devices as shown below in separate regions.

    FlashDevice* eeprom = Devices::createAddressErase(0, 256*4096);
    CircularBuffer* logBuffer = Devices::createCircularBuffer(256*4096, 384*4096);

My thinking is that the reason you are getting that error is because your read/write may fall along a page boundary when you create your device.

Each page can store 4096 bytes/2 bytes = 2048 short ints.

The following creates a buffer between page 256 and 384 (~0.5 MB). Its just for demonstration but expand it for your own application.

'''
CircularBuffer* buffer = Devices::createCircularBuffer(4096256, 4096384);
'''

When you read and write, use pointers and increment/decrement the address. Something like below:

short data;
short* ptr;
ptr = &data;

bool success = buffer->write(ptr++, sizeof(data));

I'll edit the post later to make it more clear.

3

Hi,
I think
FlashDevice* eeprom = Devices::createAddressErase(0, 256*4096); //-- 1
does the same thing as
FlashDevice *eeprom = Devices::createDefaultStorage(); // – 2

Call 1, is allocating 0 to 1,048,576 bytes. I believe I should be able to write and read from any byte within that range. But I was only able to read/write between 0-129,000. It fails after 130,000.

Here is my code compiled and flashed using the particle web ide:

#include "flashee-eeprom/flashee-eeprom.h"
using namespace Flashee;

#define ADDRESS 100000
#define VAL 2179
FlashDevice* flash;

void setup() {
        flash = Devices::createAddressErase(0, 256*4096);
        flash->write(VAL, ADDRESS);
}

void loop() {
    short a;
    flash->read(a, ADDRESS);
    Serial.println(a);
    delay(200);
}

The output is as expected for ADDRESS<=129000. The output is 0 for ADDRESS greater than 130000. I am not sure why. This isn’t EEPROM. It’s the external flash.

Thanks
Dheeraj

4

Try:

flash = Devices::createAddressErase(0, 256);

I think the README is wrong.