I am trying to use the FileSystem API to implement a data caching feature on my devices. The devices are simple; they read data from sensors and publish an event with the data every 5 minutes. This feature would have the devices cache data locally if they lose connection. They will then try to repost the cached data when connection is resumed. Each datum is approximately 200 bytes.
Due to Serial logging, I have identified the issue to be with my code that reads the data:
// calls stat() fn to get file size
int cacheBytes = cacheSize();
cache = open("cache.txt", O_RDWR);
if (cache != -1) {
Serial.println("data cache: reading");
// init buffer to take data from cache
std::string rawData;
rawData.reserve(cacheBytes);
// read to rawData, then close cache
read(cache, &rawData, cacheBytes);
Serial.println("data cache: successfully read");
close(cache);
} else {
Serial.println("data cache: error reading");
}
I’ve tried a couple things here. Originally I was using the String class from Particle, but changed to std::string (mostly out of superstition). I expect the issue is around the creation of rawData. Anyone have any insights here? Appreciate it!
I believe you need to use rawData.data() not &rawData. Taking a pointer to the std::string object gets the pointer to the object, not a pointer to the variable-length data buffer.
However also be careful because read() won’t necessarily null-terminate the buffer, which will result in an invalid c-string.
It depends on how big the data is, how long it needs to be valid for, and if it’s string-like data.
Assuming it’s string data you want to keep around for a while, you can use std::string but I think you need to manually terminate the string using rawData.data()[cacheBytes] = 0; when you read the string that way.
If the data is binary, you might want to use new uint8_t[cacheBytes] or malloc(cacheBytes).
Hmm, this doesn’t seem to work because std::string.data() returns a const void* instead of void*. I also tried reading like this:
// init buffer to take data from cache
char * rawDataBytes = new char[cacheBytes];
// read to rawData, then close cache
read(cache, &rawDataBytes, cacheBytes);
Serial.println("data cache: successfully read");
close(cache);
std::string rawData;
rawData.assign(rawDataBytes, cacheBytes);
But I’m having the same issue. The devices crashes, and blinks red before it hits that println. Is there an example of the read fn being used in any open source projects you know of? I appreciate you taking the time to help me with this.
And just for reference on my previous comment, the code below results in the compiler error: error: invalid conversion from 'const void*' to 'void*' [-fpermissive] read(cache, rawData.data(), cacheBytes);
// init buffer to take data from cache
std::string rawData;
rawData.reserve(cacheBytes + 1);
// read to rawData, then close cache
read(cache, rawData.data(), cacheBytes);
Serial.println("data cache: successfully read");
close(cache);
Sorry, I just noticed that that overload only exists in C++17. You should use malloc or new instead. Just remember if you need a c-string, malloc/new an extra byte for the null terminator.
// init buffer to take data from cache
char * rawDataBytes = new char[cacheBytes];
// read to rawData, then close cache
read(cache, &rawDataBytes, cacheBytes);
Serial.println("data cache: successfully read");
close(cache);
std::string rawData;
rawData.assign(rawDataBytes, cacheBytes);
This causes the same error (I also tried adding the null terminator manually).
yes, incredible! that works wonderfully. For reference the code I used to get this to work is (just like you said):
Serial.println("data cache: in read block");
char * rawData = new char[cacheBytes];
Serial.println("data cache: trying to read");
read(cache, rawData, cacheBytes);
Serial.println("data cache: successfully read");
close(cache);
I appreciate help on this - I’m a web developer by profession so the finer details of pointers and strings in c++ are not intuitive to me yet. Thanks!
Mind sharing an example of how you used the stat() function to get the file size? Having trouble determining the right amount of buffer to account for.
struct stat statbuf;
int result = stat(path, &statbuf);
if (result == 0) {
// do something with statbuf.st_size
}
The declaration creates a struct stat on the stack, with all of the necessary space allocated. You pass &statbuf to stat() because it requires a pointer, not the object itself.
Thanks Rick! I am giving that a try now, thank you for the direct link.
On a related note, and this could completely be my own issue and something I need to better understand, but I find it a little confusing that there are so many version of the docs available publicly. And I’m not referring to device specific diagrams vs DeviceOS, etc.
For example I did not see the example you linked to because I was reading these docs listed under the directory “cards”. Its not clear to me what “cards” are or why this documentation is seemingly different. The link I shared does not seem to have as much information in them as the link you shared, but they did seem complete enough that I neglected to search elsewhere for more information.
On a different example, the other day I was having trouble finding something using the search tools within the site and so I turned to Google. I was extra confused because the docs I was reading seemed very out of date. It took me a few minutes to realize what I was reading was on a subdomain of “prerelease-docs”.
That said the information is super helpful when I am able to track it down
The difference is that /reference/device-os/firmware is a single page for the entire firmware API reference, and /cards/firmware is one page per level-3 heading. The information under the heading should be exactly the same.
The reason for this is that the single page is always how it worked, however this works really poorly for search. The single topic per card generally works much better for search and mobile.
I’ve been meaning to remove the prerelease docs from Google, however I ran into a problem in that I didn’t have access to do it at the time, and then forgot about it.
