I am reconsidering using PublishQueueAsyncRK library on a new P2 project that has no external storage. Previous Photon based projects have had SD card storage where I implemented a publish queue using a text file which had a fairly infinite size limit and therefore could use a list with a head and tail and when the head and tail were the same could be truncated. With the P2 there is now considerably more RAM available to be used for a queue - albeit this is volatile.
Could someone suggest the best way to implement PublishQueueAsyncRK using RAM or flash memory to provide the best balance between storing data whilst offline and publishing later and having a buffer in the publishing of events?
The current process (because of the large storage capacity) when back online throws away certain regular events older than a few hours as these are no longer relevant.
Ideas:
Custom develop a hierarchical storage mechanism which maintains data whilst offline and continuously thins out older data. This is separate from the publish queue and creation of publish events. The publish queue could be fairly small.
A publish queue that is thinned out whilst offline and operates normally whilst online.
Multiple publish queues (for event groups) that can behave differently.
@armor, @rickkas7's PublishQueuePosixRK uses flash for queueing outgoing publishes using the flash file system. You can set a RAM queue which acts before data goes to flash if you wish. I have been using it on a Boron with much succes, publishing data every 10mins with no RAM queue so that publishes are always "safe".
In your case, the choice of queueing exclusively in PublishQueuePosixRK depends on how many events you want to queue and their size (ie, will it fit in flash). The library can automatically overwrite older events as the queue max size is reached, trimming older events automatically.
Personally, I prefer simplicity so in my case I use PublishQueuePosixRK exclusively to decouple the data publishing in my code from the state of the cloud connection. If you do chose to use @rickkas7's library, reach out to me as there are some caveats you may want to know about.
You should definitely use PublishQueuePosixRK on the P2 instead of PublishQueueAsyncRK. You should not use PublishQueueAsyncRK unless you need to use a specific storage method that's only supported by that library.
You shouldn't use retained RAM on the P2 because it will often be cleared when the device is reset.
Why would retained memory be cleared on a reset? I am assuming this is for a soft reset, triggered by System.reset(). Will retained memory be cleared occur if System.backupRamSync() is called before reset
The first paragraph of the PublishQueueAsyncRK README is:
If you are using a Particle Gen 3 device (Argon, Boron, B Series SoM, Tracker One, Tracker SoM, E404X) or a P2, I highly recommend using the PublishQueuePosixRK library instead. That library has a better internal design that is simpler, less likely to have bugs, and is better tested. Since Gen 2 devices are deprecated and no longer supported by current versions of Device OS, there will be limited maintenance of this library going forward.
If you are using the older library in order to use a SPI NOR flash chip as the storage method, you should use PublishQueueSpiFlashRK instead.
@gusgonnet, in my use of PublishQueuePosixRK on a Boron 404x, the background publish thread created by the library would stall (deadlock or other) occasionally. This would cause queued publishes to accumulate and the queue to fill and lose older events. In discussions with @rickkas7, we could not identify a cause for this.
The solution I found was to leverage the Boron's hardware watchdog and use the library's
callback functionality whenever a (queued) publish was completed. The callback "pets" the watchdog so that if the callback is never called due to the thread stalling, the device resets. The code has rarely, if at all, lost any unqueued events during the resets and (queued) publishes restart after the cloud connection is reestablished. To be super safe, the PublishQueuePosixRK RAM queue size is to zero so all published data goes directly to flash.
Note that in our application, the Boron never sleeps, data is collected every second for displaying and publishes (about 900 bytes) are done every 10 minutes so the watchdog timeout is set for 15 minutes. Data is also written to SD every 30s.
What if I called System.backupRamSync(); before I hit the reset button? Would that result in cleared and reinitialization of retained memory or would my saved values persist?
Yes, if you call backupRamSync() before hardware reset the saved state will be restored at boot. The problem is that there's no good way to do that before pressing the hardware reset button in the general case.