PublishQueueAsyncRK and Large Datasets

Only the P1 has SPI flash onboard, and there is not enough space on the Photon and P1 in Device OS to fit LittleFS. It might fit on the Electron and E Series, but since the devices don’t have on-board SPI flash, it doesn’t make sense for it to be in Device OS. As part of user firmware as a library it would work fine on Gen 2, and realistically that’s the only way it would ever be implemented on Gen 2.

Whether it makes sense to store the events in flash depends mainly on how often you send events, and how full the flash is. Because the wear is mostly distributed across all free sectors (for both Little FS and SPIFFS) if you have a lot of free sectors the wear is distributed across a large number of sectors and you don’t have to worry about it. If the flash is almost full, then a small number of sectors get rewritten frequently.

I was thinking of LittleFS as a user firmware library - need to get past some linker issues or maybe missing includes for arm binaries.

I see your point about flash memory - just seems FRAM is better suited with a 10^10 writes guaranteed life - I am comparing to SD card using SDFat where I have been seeing some issues (aside from the power down problem).

I have made some changes to your library PublishQueueAsyncRK to support SPI FRAM and implemented a replacement for MB85RC256V-FRAM-RK called MB85RS-FRAM-AA which supports and has been tested with MB85RS1MT and MB85RS2MT. It should work with MB85RS4MT - giving a massive 512KB send buffer. The SPI really speeds up writing to the FRAM, erase takes 1.8 seconds for the 2Mbit version.

I must say that the quality of documentation and the way you constructed the library made this reasonably straight forward - the only issue I had was with an error in the datasheets for the SPI FRAM.

Is this something you would be interested to roll into the library for benefit of anyone else who wants to use SPI FRAM.

Why Device OS 2.0.0-rc3 or later? I am looking to use it on 1.5.2 with B523.

Device OS 1.5.2 does not have support for the POSIX filesystem from user applications. There’s no way to use it, except from monolithic debug builds.

1 Like

Hopefully this thread isn’t tooooo dead…

Basic setup, taking GPS cords and throwing them onto a server I’m hosting using TCPClient. Setup hasn’t changed much since my previous post but briefly, its an Argon with an SD card (which I have for this purpose but haven’t really been using). Running 1.5.4rc2 (holding on to my mesh as long as I can). I have a wifi hotspot mounted that’s just always on - I already had it and adding a boron just seemed like an unnecessary complication/expense.

I’m having trouble turning the SDFat example into something my monkey brain understands. Basically, after a bunch of logic that decides whether or not to actually do somethin with the GPS data (stuff like, don’t send when it’s sitting in the garage because I already know it’s in the garage… or stop sending if the car is off etc.), there’s a point where the TCP client (ignoring the obvious security hole for the moment) sends the payload to my server. It looks something like this:

  client.connect(INFLUXDB_HOST, INFLUXDB_PORT);
  client.println("POST /write?db=" + String(INFLUXDB_DB) + "&u=" + String(DB_USER) +"&p=" + String(DB_PASS) + " HTTP/1.1");
  client.println("Host: " + String(INFLUXDB_HOST));
  client.println("User-Agent: Argon/1.0");
  client.println("Connection: close");
  client.println("Content-Type: application/json");
  client.printlnf("Content-Length: %d", payload.length());
  client.println();
  client.print(payload);
  client.println();
  client.stop();

The payload is made up of a bunch of concatenated string data formatted so the server can read it and is only getting the data I want it to have; lat, lon, alt, speed, some other stuff, the kitchen sink, etc.

Now, I’m sure there’s a way more “correcterish” way of doing all that, but, well… see the monkey brain bit.

I’ve included the bits mentioned many posts back for setup but I’m trying to figure out if I need to include the buffer size (retained uint8_t publishQueueRetainedBuffer[somenumber]; where somenumber is… what, the size of the memory card? Unnecessary? Do I wrap up the code above for sending the payload in… something…? I think I need a very simplified example as I’m missing something that I think is supposed to be obvious.

The payload would end up looking about like this if I print the whole thing out (made up values)

GPSTracking,device=monkeybrain count=123,lat=37.123456,lon=-121.1234567,alt=100,speed=50,sat=12,doors=locked,dist=100,distKM=123,distPrev=100

Some of that data doesn’t actually do anything at the moment but the db doesn’t like it when I change certain things and I got tired of fiddling with it, so ‘doors’ usually equals the same thing all the time, and yes, sending distKM is totally unnecessary (monkey brain).

I’m not totally opposed to starting over, but if I just have it sending GPS data all the time, I think the size of the SD card would be the least of my concerns…

Suffice to say, most of the time, I’m in range of a cellular signal, but when I go out of range, I’m usually waaay out of range for days at a time, so there’d likely be 2-3 days worth of data piling up. Just storing all that raw GPS data would just be a mess to deal with. Ok, the cat has “that look” on her face that makes me think I might need to stock up on band aids soon… think this about covers it anyway… thoughts?

PublishQueueAsync supports multiple data storage systems. If you want to use the SD card to store data, you don’t use the retained buffer at all - that’s used for retained memory.

Use the example in the more-examples/SdFatExample to save the queue on the SD card

Ok, I guess I’m trying to figure out what exactly is being stored - or should be stored, the payload or the payload + the TCPClient commands? Most of what’s talking to the particle cloud is really just making certain commands and variables available, not the actual GPS data. Stuff like, rebooting, setting a debug mode, a few misc variables etc. The actual GPS data is going directly to my server.

If I assume I’m just letting it take the payload and stick it on the SD card, how does it know what to do with that data when a connection is restored? Am I over-thinking this?

Oh, in that case you don’t want to use PublishQueueAsync at all. That’s only for storing Particle.publish data to be sent when cloud connected. You just want to use the SdFat API directly to write files containing the data you want to upload to your server. You should start a new thread for that, as it’s completely unrelated to the original post.

Well crap! Sorry for de-railing the thread then… Live and hopefully learn! lol

Ok, I don’t know why I wasn’t running through the particle cloud for this before - I think I was having trouble with webhooks but I got that bit working so I may just ditch the direct to my server bit as ‘speed’ isn’t really very important here unless were talking unexpected delays of minutes or something…

Anyway, trying to get the queue working and I’m not having any luck. I’m also not getting any errors, but I don’t have the device physically in front of me at the moment so I’ll have to fiddle with the serial output later if I need to.

This is a very chopped version with junk data but probably does a better job of illustrating what I’m attempting to do (and my level of understanding…)

#include "../lib/AdafruitDataLoggerRK/src/AdafruitDataLoggerRK.h"
#include "../lib/SdFat/src/SdFat.h"
#include "../lib/PublishQueueAsyncRK/src/PublishQueueAsyncRK.h"

SYSTEM_THREAD(ENABLED);
RTCSynchronizer rtcSync;

const int SD_CHIP_SELECT = D5;

SdFat sdCard;
PublishQueueAsyncSdFat publishQueue(sdCard, "events.dat");
bool pub_status=false;


//This is sort of a holdover from an older version that kept exceeding the limit, but I can't really get useful GPS data in sub 1 second intervals anyway.
elapsedMillis rateLimiter;
unsigned int rateInterval = 1000;

//For the data and this example
String theLoad = "";

void setup()
{
  Serial.begin(115200);
  rtcSync.setup();
  rtcSync.syncTime();

  waitFor(Serial.isConnected, 10000); //Do I need this?
  if(sdCard.begin(SD_CHIP_SELECT, SPI_FULL_SPEED))
  {
    publishQueue.setup();
  }
  else
  {
    Log.info("failed to initialize sd card");
    Particle.publish("SDCARD","failed to initialize sd card",PRIVATE);
  }

  rateLimiter = 0;
}

void loop()
{
	rtcSync.loop();
	if(rateLimiter >= rateInterval)
  	{
  		getGPSdata();
  		rateLimiter = 0;
  	}
}

void getGPSdata()
{
	//Junk data below since I've pulled the actual functions that get real GPS data.
	theLoad = "GPSTracking,device=monkeybrain count=123,lat=37.123456,lon=-121.1234567,alt=100,speed=50,sat=12,doors=locked,dist=100,distKM=123,distPrev=100";
	pub_status = publishQueue.publish("GPS", theLoad, PRIVATE, WITH_ACK);
	//The below works, but obviously only if the cloud is available
	//Particle.publish("GPS",String(theLoad),PRIVATE);
}

If it looks like something glaringly obvious is missing (like a missing #include), I probably missed it in copy/pasting it into this version. I also haven’t actually tested this version… Before I actually strip down the much longer code, I wanted to make sure I’m understanding the usage correctly. The example included with the library is doing several things and I think I’m getting distracted by the extra things going on (or I’m just misunderstanding). Seems like it should be simple enough if I’m doing it right… Am I even close?

Yeah… glaringly obvious ended up being hardware, as in I had the logger board sitting in a position it shouldn’t have been - am using one of the feather side by side boards and forgot I’d soldered some stuff in place and yeah… anyway…

New question, @rickkas7 , does your library handle the rate limit automatically?

Yes, it handles rate limiting of sending events to one per second.