[Submission] Flashee-eeprom library: eeprom storage in external flash


#82

@mdma Sorry but I have another doubt about Flashee.

I’m maintaining timestamp records in the flash. Each record is 10 bytes long and I’m planning on having 25-100 records. To reset/void a record, all I have to do is set the first byte of the record to a default value.

If I want to reset/void all the records, is it better to erase the entire sector or individually set default values to the first bytes of all 100 records?

Thanks so much already! :blush:


#83

no problem @nitred, I appreciate the questions, and that’s a good one!

The short answer is erase sector. With wear levelling and address erase schemes, you get an endurance of something like 10 million erases, since they are spread out throughout all sectors.

Thats the short answer! Here’s a longer answer:

It depends upon how often you plan to do it and which storage scheme you are using.

If you’re using the wear leveling scheme, then erasing the sector will be best, since you only get one erase happening to the flash device. Whereas if you wrote 255 to each first byte, then that would cause an erase for each write (when the value isn’t already 255.)

If you’re using the address erase scheme, then you can just write the first byte, since the sector is only erased on the 7th destructive write to a value. Depending upon what has been written before, you may be able to set 0xFF in the first byte for all records without requiring an erase in the underlying sector. And even if the underlying sector is erased, it will be erased only once.

I hope that’s clear, but if in doubt, erase the sector!


#84

Those are good thoughts!

Yes, reasonably involved to implement, but easy for the user, who has to just use a special implementation of Vector or String, or some other container class. I don’t think there’s a need to work with primitive variables like uint8_t, since they don’t really take up much space - they can just as well be stored in RAM. Instead, I’m thinking primarily of large data structures/ADTs such as lists, trees, hashtables etc… that can use external memory as their backing store.

You’re definitely correct on performance, it won’t match SRAM speeds. However being able to have simple access to megabytes of data locally may be a key enabling factor to building some solutions, more so than raw performance.

There would be no stack, just a reference in RAM to external memory ranges that are in use, so this should be as robust as the regular memory management (malloc/free etc…)

Thanks for sharing your thoughts, it’s good to have evaluation of ideas bounced around!


#85

@mdma, I’d like to share some code which I tried to use your library with.
I’s a stripped version of a larger code, for a spark based LED display.
Here, a LED is replaced with serial.
What I’m trying to do is send messages via spark function, stack them and then cycle display by pressing a button (keyboard key here). I’m using AddressErase, since it compiles from the cloud,
and every other method I tried make my spark red SOS, and I don’t want to compile locally.
So, I’m writing up to 10 messages to different addresses (multiplications of 1024), messages are to be up to 1kB long - here, for testing, I don’t assemble messages, just send single short ones.
I move the old messages down the “stack” when a new arrives, and can also delete any message, moving the old up to have them in order.
I’m not claiming this is an elegant code by no means, but it still can be useful, since -
STRANGE THINGS HAPPEN !!!
The code is ready for debugging - everything is printed to the serial, so it’s enough to send few sms’es and then delete them, sooner or later (rather sooner) you will see a lot of garbage…
There is also a second spark function, allowing to Devices::userFlash().eraseAll() and .eraseAll(), I erased the whole flash and rebooted before every test, just to be sure.
And one thing more - if there is a bug in my code, causing this behavior, please feel free to call me an idiot, I tested it really long, so maybe I just can’t see an obvious problem…

// This #include statement was automatically added by the Spark IDE.
#include "flashee-eeprom/flashee-eeprom.h"
using namespace Flashee;
//---------------------------------flash
FlashDevice* flash1;
// +++++++++++++++++++++++++++++++++++++++++++++++++ GLOBAL VARIABLES
byte display = 0; // 1 display message1, 2 display message 2 etc
String message[11]; // message[0] not used, to be compatible with the display variable - messages are counted like display, 1-10
//************************************************  SETUP
void setup() {
flash1 = Devices::createAddressErase();
Serial.begin(9600);
//=================================================SPARK FUNCTIONS
Spark.function("sms", sms);
Spark.function("cmd", cmd);
}
//**************************************************************************************    main loop
//***************************************************
//***************************************************
void loop() 
{
checkButton(); 
if (display == 0)
{
Serial.println("main menu - press 1 to cycle messages, 2 to delete current message");
delay(1000);
}
else 
{
if(message[display]=="") return;
Serial.print("message" + String(display) + ":  ");
delay(1000);
Serial.println(message[display]);
delay(1000);
}
}
//=================================================== SPARK FUNCTION cmd
int cmd(String co)
{
if(co == "deleteflash") 
{
Devices::userFlash().eraseAll();
for (byte x=1; x<=10; x++) message[x] = "";
}
else if(co == "delete") 
{
flash1->eraseAll();
for (byte x=1; x<=10; x++) message[x] = "";
}
return 1;
}
//*************************************************************** sms
int sms(String line)
{
for (byte x=10; x>1; x--) 
{
readflash1(x-1);
message[x]=message[x-1];
writeflash1(x);
message[x]="";
}
message[1]=line;
writeflash1(1);
Serial.println("new messages in flash:");
        for (byte y = 1; y <= 10; y++)
        {
        readflash1(y);
        Serial.println("message" + String(y) +": " + message[y]);
        message[y] = "";
        }        
        Serial.println("press any key...");
        while(!Serial.available())
        {}
display = 1;
return 1;
}
//*************************************************************** writeflash1
void writeflash1(byte toflash)
{
int l = message[toflash].length();
String ll;
//
//normally messages are up to 999 characters long
//
if (l == 0) ll = "000";
else if (l <10) ll = "00" + String(l);
else if (l< 100) ll ="0" + String(l);
else if (l >= 100) ll = "" + String(l);
String  f = ll + message[toflash];
char f1[f.length()+1];
f.toCharArray(f1, f.length()+1);
delay(20);
flash1->writeString(f1, (toflash)*1024);
delay(20);
}
//*************************************************************** readflash1
void readflash1(byte toread)
{
char buf[3];
flash1->read(buf, (toread)*1024, 3);

int buf1 = String(buf).toInt();
if (buf1 > 0)
{
char f1[buf1+1];
delay(20);
flash1->read(f1,(toread)*1024+3,buf1+1);
delay(20);
message[toread]=f1;
}
else 
{
message[toread] = "";
return;
}
}
//************************************************ checkButton
void checkButton()
{
byte key;
key = Serial.read();
if (key == '1') 
{
 message[display] = "";
 display++;
 if (display==11)
 {
     display=0;
     return ;
 }
 readflash1(display);
 if (message[display]=="")
 {
 Serial.println("no messages, send sms !");
 display=0;
 }
} 
if(key == '2')
{
if(display==0) return;
        Serial.println("message to delete - " +  String(display));
        Serial.println("old messages in memory:");
        for (byte y = 1; y <= 10; y++)
        {
        Serial.println("message" + String(y) +": " + message[y]);
        }  
        Serial.println("old messages in flash:");
        for (byte y = 1; y <= 10; y++)
        {
        readflash1(y);
        Serial.println("message" + String(y) +": " + message[y]);
        message[y] = "";
        }
        Serial.println("deleting message " + String(display));
        for (byte x=display; x < 10; x++) 
        {
        readflash1(x+1);
        message[x]=message[x+1];
        writeflash1(x);
        message[x]="";
        }
        message[10]="";
        writeflash1(10);
        Serial.println("new messages in flash:");
        for (byte y = 1; y <= 10; y++)
        {
        readflash1(y);
        Serial.println("message" + String(y) +": " + message[y]);
        message[y] = "";
        }        
        Serial.println("message "+String(display) + " deleted, press any key...");
        while(!Serial.available())
        {}
        Serial.println("returning to menu");
        display = 0;
}
}

#86

@kennethlimcp, FSMC peripheral is available only with higher pin count versions of STM32 (100 pins devices and above such as STM32F103Vx). I doubt Spark will go for such large footprint packages in future versions.


#87

You know, this is exactly the kind of problem that could be solved by the idea I sketched out above for STL vector and other classes backed by external memory! You’d then not have to write all the code to shuffle messages, read/write to from flash etc. You’d just have to put them in a vector and use the vector api to insert/remove elements. Anyway, I digress!

The deleteFlash command should also reset the spark with System.reset() since it’s necessary to recreate the flash device again. deleteFlash should only be necessary once, and only if you were using the previous version of the library (1.7.7) - I’m assuming you’re using the latest version 1.7.8?

Just taking a look through the code and writing down things as I see them:

  • in writeFlash1, you’ll end up consuming a lot of RAM.
    String f = ll + message[toflash]; char f1[f.length()+1];
    Assuming the message[toflash] is 1024 bytes, then string f will be at least that long, and then the char buffer f1 is also at least 1024 bytes long. Of course, this might just be debug code, written for your test with small strings. (Also, btw, rather than copying the bytes out with toCharArray, you could use c_str() to get read-only access to the buffer inside the String object.)

I don’t see anything wrong with the code from first glance. Could you perform a deleteAll then a System.reset(). and then try to reproduce the problem, making a note of the steps you take?

I will then code up the same calls to flashee in a test to see if I can reproduce the problem here.

I really appreciate your patience and assistance with this!


#88

@mdma, I’m using the latest version, since I only compile from the cloud, and it says 1.7.8.
You are absolutely right about the RAM, at this moment I just hope it will fit in RAM, I still have more than 5kB free after compiling the whole code, and I’ll think about optimizing it later on.

Just for now, problem reproduction follows >>>>with my comments:

~ $ spark call
Listfunctions called
polling server to see what cores are online, and what functions are available
Retrieving cores... (this might take a few seconds)
xxxxxxxxxxxxxxxxx has 2 functions 
  int sms(String args) 
  int cmd(String args) 
~ $ spark call kefir-core1 cmd "deleteflash"
1
~ $ spark call kefir-core1 cmd "delete"
1
~ $ 

>>>reset button on core

>>>SPARK CLI:

~ $ spark call kefir-core1 sms "55555555555555555555555"
1
~ $

>>>Serial

main menu - press 1 to cycle messages, 2 to delete current message
new messages in flash:
message1: 55555555555555555555555
message2: 
message3: 
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
press any key...
main menu - press 1 to cycle messages, 2 to delete current message
>>> key 1 pressed
message1:  55555555555555555555555
message1:  55555555555555555555555
message1:  55555555555555555555555

>>>SPARK CLI:
~ $ spark call kefir-core1 sms "44444444444444444444444"
1
~ $ 
>>>Serial
message1:  55555555555555555555555
message1:  55555555555555555555555
new messages in flash:
message1: 44444444444444444444444
message2: 55555555555555555555555
message3: 
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
press any key...
main menu - press 1 to cycle messages, 2 to delete current message
>>> key 1 pressed
message1:  44444444444444444444444
message1:  44444444444444444444444
message1:  44444444444444444444444
>>> key 1 pressed
message2:  55555555555555555555555
message2:  55555555555555555555555
message2:  55555555555555555555555

>>> and so on to message 11111111111111111111111:

new messages in flash:
message1: 11111111111111111111111
message2: 22222222222222222222222
message3: 33333333333333333333333
message4: 44444444444444444444444
message5: 55555555555555555555555
message6: 
message7: 
message8: 
message9: 
message10: 
press any key...

>>> now I can cycle messages pressing 1:

main menu - press 1 to cycle messages, 2 to delete current message
main menu - press 1 to cycle messages, 2 to delete current message
message1:  11111111111111111111111
message2:  22222222222222222222222
message3:  33333333333333333333333
message4:  44444444444444444444444
message5:  55555555555555555555555

>>> now lets delete some messages by pressing 2

main menu - press 1 to cycle messages, 2 to delete current message
message1:  11111111111111111111111
message1:  11111111111111111111111
message1:  11111111111111111111111
message1:  11111111111111111111111
old messages in memory:
message1: 11111111111111111111111
message2: 
message3: 
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
old messages in flash:
message1: 11111111111111111111111
message2: 22222222222222222222222
message3: 33333333333333333333333
message4: 44444444444444444444444
message5: 55555555555555555555555
message6: 
message7: 
message8: 
message9: 
message10: 
display - 1
new messages in flash:
message1: 22222222222222222222222
message2: 33333333333333333333333
message3: 44444444444444444444444
message4: 55555555555555555555555
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
message 1 deleted, press any key...

>>>> so far so good

main menu - press 1 to cycle messages, 2 to delete current message
message1:  22222222222222222222222
message1:  22222222222222222222222
old messages in memory:
message1: 22222222222222222222222
message2: 
message3: 
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
old messages in flash:
message1: 22222222222222222222222
message2: 33333333333333333333333
message3: 44444444444444444444444
message4: 55555555555555555555555
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
display - 1
new messages in flash:
message1: 33333333333333333333333
message2: 44444444444444444444444
message3: 55555555555555555555555
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
message 1 deleted, press any key...

>>> still good


message1:  33333333333333333333333
message1:  33333333333333333333333
message1:  33333333333333333333333
old messages in memory:
message1: 33333333333333333333333
message2: 
message3: 
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
old messages in flash:
message1: 33333333333333333333333
message2: 44444444444444444444444
message3: 55555555555555555555555
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
display - 1
new messages in flash:
message1: 44444444444444444444444
message2: 55555555555555555555555
message3: 
message4: 
message5: 
message6: 
message7: 
message8: 
message9: 
message10: 
message 1 deleted, press any key...

>>>>>> and so on, successfully deleted all messages, so the algorithm works correctly. Lets repeat all steps.

>>>>> SPARK CLI

~ $ spark call kefir-core1 sms "55555555555555555555555"
1
~ $ spark call kefir-core1 sms "44444444444444444444444"
1
~ $ spark call kefir-core1 sms "33333333333333333333333"
1
~ $ 


>>>>>>> SERIAL

main menu - press 1 to cycle messages, 2 to delete current message
main menu - press 1 to cycle messages, 2 to delete current message
main menu - press 1 to cycle messages, 2 to delete current message
main menu - press 1 to cycle messages, 2 to delete current message
new messages in flash:
message1: 
message2: 
message3: 5555555555555555555
message4: 
message5: 555555555555555
message6: 
message7: 55555555555
message8: 
message9: 5555555
message10: 
press any key...
no messages, send sms !
main menu - press 1 to cycle messages, 2 to delete current message
main menu - press 1 to cycle messages, 2 to delete current message
new messages in flash:
message1: 44444444444444444444444
message2: 
message3: 4444444444444444444
message4: 5000
message5: 444444444444444
message6: 5000
message7: 44444444444
message8: 5000
message9: 4444444
message10: 5555555
press any key...
no messages, send sms !
main menu - press 1 to cycle messages, 2 to delete current message
main menu - press 1 to cycle messages, 2 to delete current message
new messages in flash:
message1: 33333333333333333333333
message2: 4000
message3: 3333333333333333333
message4: 40045000
message5: 333333333333333
message6: 40045000
message7: 33333333333
message8: 40045000
message9: 3333333
message10: 4444444
press any key...

AND THIS IS THE WHOLE PROBLEM. SOMETIMES IT HAPPENS WHILE RECEIVING A MESSAGE, SOMETIMES WHILE DELETING.

You can easily reproduce this, by flashing a core with the code, it happens very quickly and every time. Randomly, there is no rule where it happens. Sometimes, after receiving the message, the core goes red SOS and reboots. I had the previous version of the code with all messages in RAM (working ok with short messages), and never got red SOS with it.

So, if you need me to test anything more with it, I’ll be more than happy to do so !


#89

What’s the SOS count when it fails? Is it always the same?

I think there is still an unresolved issue with compiling flashee in the cloud, that I am unable to explain. (It looks like a heap error, but running the same code compiles locally works fine.) Shall I compile the code locally for you to test?


#90

@mdma yes, its always one flash after sos.
I don’t have any code to compile at the moment, as I said, with devices::createAddressErase() it compiles correctly form the cloud.
However, I tried earlier a streaming device Devices::createWearLevelErase(); and it never compiled, always caused red error, and required factory reset. So I also believe this issue still exists.
But whatever - AddressErase, if working correctly, is absolutely a good tool for my project.
(If we can get it working, I can try later on other solutions from Flashee, maybe better for RAM optimization).


#91

The crazy thing is that addressErase actually uses the wearLevel device, so really strange that addressErase works when wearLevel fails!

I’m very sure that it’s not a RAM exhaustion issue, even with a empty sketch I get a SOS. It could be heap corruption, which might manifest itself in the same way. I get 8 flashes with an empty sketch (out of memory), while you get 1 (hard fault) - both of these could be caused by heap corruption in some form.

I can try using the cli to compile and see if that creates the same problems with an empty sketch, if so, I should be able to debug and track where and why the error is happening.


#92

Can you please clarify what you mean with empty sketch ?

  1. Can you compile it, and get SOS while trying to do something (like sending a message via spark function) ?
  2. or you can’t compile it at all using cloud ?

If 1, can you reproduce the problem with random messages in memory after few read and writes (like in my problem reproduction from previous post) ?
If 2, it’s super weird, because I can compile from web. What’s the difference, is it between cores ? I got a black one.

Also, a heap corruption is a software or hardware problem (broken core or something) ?


#93

@mdma - hi, any news about the problem we discuss ? thanks…


#94

Hi @kefir135,

I think he means a program / project like:

void setup() {

}

void loop() {

}

The web IDE uses a different branch (currently compile-server2), than when you compile locally, so that might be the difference as well. The server is also using a recent copy of the arm GCC embedded toolchain which might matter, make sure you’re using a recent 4.8 release.

Thanks!
David


#95

Sorry for the delay @kefir135 - been busy the past few days. I’ll set aside time this week to compile your code and reproduce the failures you’ve been experiencing, both with online and local builds, to see if there is a difference.


#96

Great ! thanks a lot for your time @mdma, can’t wait to get it working :smile:


#97

I don’t know if I fixed it, but first time around I got strange results. I did 2 things:

  1. I added a Serial.println() as part of the deleteflash command to be sure it was happening.
  2. Fixed a buffer overflow here:
void readflash1(byte toread)
{
char buf[3];
flash1->read(buf, (toread)*1024, 3);

int buf1 = String(buf).toInt();

Do you see it?

The buffer should be size 4 since you are reading 3 chars, plus a null terminator is needed for String::toInt() to work correctly.

char buf[4];
buf[3] = 0;

Without this, the string that String::toInt() sees could be anything depending upon the contents of the stack, which could in some cases result in toInt() returning a large unexpected value. This is probably the cause of the memory problems you’re seeing, since the following line tried to dynamically allocate a large buffer on the stack, corrupting the heap.


#98

Yeah, I see it, and I’l try it with buf[4], but I’ll begin with current code adding Serial.println(String(buf1)) to see what it actually reads, this way I’ll be able to test the difference. I’ll do it in the evening (I’ve got morning here, Poland).


#99

@mdma, sorry to say that, but no difference between buf[3] and buf[4].
I also added the println for buf1

char buf[4];
buf[3] = 0;
flash1->read(buf, (toread)*1024, 3);
int buf1 = String(buf).toInt();
Serial.println("readflash -> buf1 = " + String(buf1));

And, both the buf[3] and buf[4] versions print correct buf1 values, also, when data read from flash is scrambled, buf1 shows the correct length of the scrambled data.
So, unfortunately, no progress :frowning:


#100

Can you take out the flash write functions, and for flash read, just write dummy data to the buffer, e.g.

// instead of reading length and data from flash, just do a dummy read each time with different data
static char c = 'A';
length = (c-'A')+10;
memset(buf, c++, length);
buf[length] = 0;

That will narrow the problem down to the flash or something else in the code, like memory management. Obviously your program wont function the way you expect, but it shouldn’t crash. If it does, or you see garbled data, you know the error is not in flash.


#101

So I did it, and I got a crash every time with the second message sent. I’ll try to narrow it a little bit, and let you know later on.