I posted this question to the community earlier in the year on a conceptual method to reduce data operations. I finally had some quiet time to focus on it. The prior post is closed so I figured I'd start a new thread. I thought others might find this beneficial for similar use cases and if you see a way to make this better, cleaner feel free to post here as well.
Background:
Basically What I want to do is instead of publishing something like this using PublishQueue.publish() 12 times where each reading is 5 minutes apart:
{ "Sensor1": 1234, "Sensor2": 5678, DateTime:1623000246}
{ "Sensor1": 1234, "Sensor2": 5678, DateTime:1623000546}
{ "Sensor1": 1234, "Sensor2": 5678, DateTime:1623000846}
…
I would publish a single JSON array like this once:
[ {"Sensor1": 1234, "Sensor2": 5678, DateTime:1623000246}, {"Sensor1": 1234, "Sensor2": 5678, DateTime:1623000546}, {"Sensor1": 1234, "Sensor2": 5678, DateTime:1623000846}...]
I'd update my backend to then process the array of data instead of individual web hooks of every data point. The benefit is: 1) reduce data operations from the Particle Cloud now that Data Operations is the unit of measure. 2) Reduced load on my backend as it only needs to process 1 web hook that contained an array of data rather than 12 separate web hooks.
Here is the code I used:
#include <fcntl.h>
#include <sys/stat.h>
Step 1 (Create a JSON and write it to the file system): Call this as many times as needed (once every 5 minutes) as the fileCnt++ increments the file name. 1 file per JSON of Sensor Data.
JsonWriterStatic<256> jwInner;
{
JsonWriterAutoObject obj(&jwInner);
jwInner.insertKeyValue("Sensor1", 1234);
jwInner.insertKeyValue("Sensor2", 5678);
jwInner.insertKeyValue("DateTime", 1623000846);
}
JsonWriterStatic<256> jwOuter;
{
JsonWriterAutoObject obj(&jwOuter);
jwOuter.insertKeyJson("Data", jwInner.getBuffer());
}
String Data = jwOuter.getBuffer();
//Increment the file counter by 1 for each new file that gets written
//Index is reset to 0 when all data is published
fileCnt++;
//Open the file, if it doesn't exist create it, truncate it to length 0, then write the data to the file and close it.
int fd = open(String(fileCnt), O_RDWR | O_CREAT | O_TRUNC);
if (fd != -1) {
write(fd, Data.c_str(), Data.length());
close(fd);
}
//Publish the data: COMMENTED OUT AS NOW WRITING A FILE TO THE FILE SYSTEM
// Log.info("Lora message: %s", jwInner.getBuffer());
// publishQueue.publish("Data", jwInner.getBuffer(), 60, PRIVATE, NO_ACK);
Step 2: Create a function to publish all files. This function checks for the max publish length (Requires Device OS3.1 or you could just hardcode it. It'll keep reading files and appending them to the JSON array until the max publish length is reached, once it's reached, it'll publish the event and clear the buffer, if it is never reached, it'll publish the event once all files are read. This will continue until all files are published.
/*******************************************************************************
* Function Name : publishFiles()
* Description :
* Return : Publishes all files from the File System as a JSON Array
*******************************************************************************/
int publishFiles(){
//Obtain the maxEventDatasize. This can change based on firmware version, device and modem firmware
int maxEventDataSize = Particle.maxEventDataSize();
int remainingBytes;
Log.info("eventDataSize=%d", Particle.maxEventDataSize());
JsonWriterStatic<1024> jw;
{
JsonWriterAutoObject obj(&jw);
jw.init();
jw.startArray();
//For each file, open it, read the contents, add the contents to the JSON array and close the file.
for(int i = 0; i < fileCnt; i++){
int fd = open(String(i+1), O_RDONLY);
if (fd != -1) {
char fileData[256];
memset(fileData, 0, sizeof(fileData));
//Determine the length of the file we are reading
struct stat statbuf;
int result = fstat(fd, &statbuf);
if (result == 0) {
//Read the entire file
read(fd, fileData, statbuf.st_size);
}
close(fd);
Log.info("maxEventDataSize: %i | Current Write Offset: %i | Current File Size: %i", maxEventDataSize, jw.getOffset(), statbuf.st_size);
remainingBytes = maxEventDataSize-jw.getOffset()-statbuf.st_size;
//If less than 10 bytes are left after adding the current file, before adding the file, publish the Array to the queue and then re-initialize the buffer
if(remainingBytes < 10 ){
Log.info("Buffer is full: Publish Buffer and reset it");
jw.finishObjectOrArray();
publishQueue.publish("DataArray", jw.getBuffer(), 60, PRIVATE, NO_ACK);
jw.init();
jw.startArray();
}
jw.insertCheckSeparator();
jw.insertJson(fileData);
}
}
jw.finishObjectOrArray();
}
//Now publish the JSON Array to the cloud.
publishQueue.publish("DataArray", jw.getBuffer(), 60, PRIVATE, NO_ACK);
fileCnt = 0;
return 1;
}
Once the device is connected to the cloud I simply call the function to publish all the files:
publishFiles();
Now instead of 12 individual publish events like this:
I now receive 1 or 2 of these properly formatted JSON arrays: (If total bytes is > 1012, it is split up into multiple publish events):
I'll probably limit the number of files to something like 100. If it ever gets that full without publishing the data to the cloud first, I'm OK with just overwriting prior data. That way I shouldn't be able to fill the file system.
Next step for me is to update my backend to ingest the array of JSON data rather than individual elements. Should be fairly easy with a Python for Loop.
A few questions for those smarter than me:
- Is there an easier way to do this? I liked the file system as it seemed easy to keep track and can store the entire JSON right to it.
- Is there an impact on using the file system repeatedly like this. I.e. read/write a file called the same name over and over again (once every every 5 minutes).
- Should the file number be something other than just an integer?
- Should I make the files more of a "Ring Buffer" I.e. write new files continually from 0 --> 100 before looping back around to 0? I currently start over at 0 every time I publish data. So file 1 gets a lot more used than file 20 and file 50+ may never get used. Would a ring buffer be better on longevity of reading/writing to memory?
- Would simply deleting after reading it but an OK method instead of a ring buffer?
- Any other thoughts for improvement?
