Did not find this in the docs. When using PublishQueueAsyncRK with LFS, is there a storage limit built into PublishQueueAsyncRK, or is it up to the user to ex. check nof items and purge the oldest event when saving a new one?
There is no limit for file system based PublishQueueAsyncRK (POSIX, SPIFFS, SdFat). However, because of the way they work, deleting the oldest event is tricky.
Events are stored in a single file, at the end. There’s a pointer to keep track of how much has already been sent, saved in the beginning of the file. That way, at worst a 512 byte sector need to be written to disk.
Moving all of the contents of the file to remove the beginning is a really expensive operations, so it waits until the entire file has been sent, then truncates it, which is efficient. In the normal case of periodically publishing and having it work, it has very low overhead.
Using the method to delete the oldest or second oldest even won’t actually save any disk space, because it doesn’t actually delete the data, it just marks at is no longer waiting to be sent.
1 Like
Thanks. Makes a lot of sense. For LFS being a shared system, I want to mitigate potential (future) risks related to maxing it out.
As described, it seems better for me to stop adding items when publishQueue.getNumEvents() hits a predefined max nof events comparable to ex. 512KBytes. Makes sense for users too.
That way the file should worst case never grow beyond that limited size?
1 Like
Yes, that’s a great option and a good idea. The getNumEvents() method will return the current number of events that are stored in the file, which is a good proxy for how large the file is.
As feedback I did not get this to work with OS2.0.1.
One issue is when periodically adding 3 items to the queue (2x 620 bytes and 1x ~300 bytes) , only one or two of them are there when sending, despite return values when adding were ok. It works with FRAM.
I will probably look more into it later to save FRAM cost, but will use FRAM for now.
It would be good to know if some of you guys have PublishQueueAsyncRK successfully running with LittleFS/POSIX for long periods ?
I got saving to work with POSIX, but after days of running fine, deletion of succesful uploads stops working and number of saved uploads hit the max I set (OS2.0.1).
I am using the library in paused mode, do uploads myself like the library, and delete succesful uploads with publishQueue.discardOldEvent(false); upon confirmation from our server.
With FRAM I got corrupted data in a matter of days, and when uploaded the webhook throw away the data without an error response, so the corrupted upload is not deleted and gets resent later, and so on.
Retained memory is working fine for weeks with the same code. I will report back when I can recreate the bug with a minimal code example.
Going to long term test PublishQueueAsyncRK with FRAM on 2.1.0 as the enhancement "Improve I2C reset procedure to be less destructive ..." sounds promising.
Unfortunately not. One upload with malformed data was read back from FRAM after one and a half week, and had to be discarded.
I have a new library coming out soon for Gen 3 devices. It uses a different technique of a small queue in RAM, with a set of files for the queue on the file system. It’s significantly simpler because the publish and queuing are factored out into separate libraries, and there is none of the circular buffer stuff from PublishQueueAsync. I’ve sent tens of thousands of events through it successfully so far.
PublishQueuePosixRK is now available:
This library works a bit differently than PublishQueueAsyncRK:
- It only works with the POSIX flash file system
- It can keep a number of events in regular memory for efficiency and reduced flash wear
- Or it can always write events to the file system for maximum prevention of event loss
- The file system queue supports discarding the oldest events when the size limit is exceeded
1 Like
With the FRAM, running two devices for two weeks 1/1500 to 1/1800 uploads fail on each unit.
The new lib sounds promising. I will see if I can fit in this projects hard requirement to only discard uploads on positive web hook response …
In case it is not intentional the GitHub link is 404 for me https://github.com/rickkas7/PublishQueuePosixRK
It’s fixed now. I thought I had made it public, but either I didn’t or the change didn’t take.
In the meantime I have made a fallback with the old lib. The 3 new uploads at a time, are always stored in 3 retained RAM stores, and the publish engine prioritise to send those first. At first web hook response error or response timeout (5 sec), all RAM entries are put to PublishQueueAsyncRK w. FRAM.
So the RAM stores are exposed to a power cycle for a few secs every upload, but the 1/3000 FRAM fails per device is only activated in the very rare error situations, instead of affecting normal operation. It should be years between any fails.
The correct solution is still to use internal POSIX, with web hook response confirmed discards, so will still try and accomplish that with the new lib.
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.