@rickkas7 Dude this new library is totally Awesome!
Would this be relatively easy to use to send to something that isn’t particle? My particular case is storing GPS cords and then sending them back to a server I have privately hosted and skips the cloud - for the data anyway, I’ve got some cloud functions to grab the current cords or to reboot the device etc. For the sake of simplicity though, lets just assume I’m not going to publish to the particle cloud etc.
Or is there a library more suited for the task? I’m assuming that I will be frequently out of cellular range, but want it to queue up data to be sent - potentially to an SD card (whatever works really). This isn’t for an electron though, its going through a hotspot (it has multiple uses :-P)
2 Likes
@rickkas7 Just had an idea while looking through the library, is there any issue with forking the library and replacing the following
-
Particle.connect()->Mesh.connect() -
Particle.publish()->Mesh.publish()
It would be great to have a similar library for Mesh only messages.
@emile, the primary reason for the library is to maintain a 1-per-second publish rate limitation set by Particle. Such a limitation does not exist with Mesh publishes. Also, Mesh publishes have no ACK mechanism so they are fast. What is your use case for the new library?
1 Like
@peekay123 Ah ok, nevermind. I was confused, and thought Particle.publish & Mesh.publish operated in a similar fashion.
I am only using Mesh.publish right now to pass data from the edge devices to the gateway. Having the ACK would be great because I’m currently publishing and then reseting my counters (not ideal).
@emile, you can create an ACK by having the gateway do a Mesh.publish() with an “ACK” message back to the node. I believe this was covered in a topic somewhere but I can’t recall where.
1 Like
Great idea, it does add a little data-overhead on the network, but I’ll take it right now.
Thank you again!
Not sure how to frame this question - my previous post hasn’t changed though (back in nov 2019).
Without having done the math, lets say I’ve got a weeks worth of gps data stored every 2 seconds to an sd card (thats a lot more than it’ll ever be - days most likely). Not because thats just where I store it, but because I’ve been out in the boonies and thats the time between having signal and not having signal etc. If my max publish rate is 1 per second with (I’d have to look it up) some max amount of data per publish, I’m guessing going through the cloud isn’t always going to be practical unless I don’t care how long it takes (I do).
Is using a library such as this feasible? My plan was to take what I have now which sends everything back to my system at home and stores it on an SD card that I can go and look at later, but I haven’t come up with a good way to sync those up. In my mind I would think the logic would be something to the effect of:
Have Signal? Send home.
Don’t have Signal? Store on SD
Have Signal again? Send SD data home and resume sending home.
Thats the basic idea - alternately, storing everything on SD and queuing up the data when out of range, then sending it all when I’m back in range “works” too. This actually seems to me more along the lines of what you’re doing with this library (thus my interest).
Am I biting off more than I can chew here? Has no one done this without the cloud? I can go into more details about the setup if need be, but thats the gist of it.
@variable, your approach to storing then sending is good. A Particle.publish() has a max payload of 622 characters (NOT 8-bit bytes) which you can use in a webhook (to send data to a service like Ubidots or others) or to your own subscribed server (using Particle’s javascript library for example). This approach will give you an effective 622 chars/s transfer rate or 37,320 chars/min. However, you do gain full end-to-end encryption.
Another approach, if encrypted data transmission is not a requirement, is to use TCPClient to send the data to a target server directly. This method allows much faster data transfer and also allows you to use full 8-bit byte data if that is required. You can create a FIFO or circular buffer on your SD to store the data and pull it off in “chunks” of whatever size payload you want.
You could also use one of the available TLS encryption libraries on the web IDE to add encryption to the TCPClient approach but the libraries are large and have large memory requirements which may not suit your application.
4 Likes
Yes - I have done this - my model is to create a file on the SD - write all messages to that file as records. The message is a struct with one of the members being a “sent” boolean flag. I have a separate process in loop() that check if we are online and if so, then cycles through the SD records to find any “not sent” and removing them from the file when sent. I use PublishQueueRK library to manage the actually publish as it does it so well.
1 Like
Is there a “for dummies” version of that (or recommended reading material - also for dummies)? Conceptually I completely understand what you’re describing and how that works… syntactically I feel like I’m trying really hard to concentrate on something after having downed a dozen shots of <insert strong alcoholic beverage of choice>
Obviously I don’t want you to do the work for me (well, maybe a little), I’m just trying to figure out how best to budget my time (you really don’t want to see my code… its embarrassing, but at the same time a little astonishing that it works as well as it does for being the bloated mess that it is - I hate starting over on things). Maybe some snippets? I think the part that I’m struggling with at the moment, mostly because I haven’t spent much time doing it, is actually reading/writing to the sd card.
I excel at biting of more than I can chew and then waiting till I’m ready to bury my head in the sand before I ask for help 
<sigh> On the bright side, I know several thousand ways how not to build a light bulb… 
If your issue is storing the events on SD for later publish, the PublishQueueAsynkRK library does that for you. You just need to instantiate the publishQueue object via the correct constructor and let it do the rest
2 Likes
Hey guys,
I hope you can help me figure this out.
I have succesfully used this library in an Electron, I have my own Queue in RAM that helps me store thousands of encoded datapoints before publishing them. I just wait until the publishQueue.getNumEvents() is empty so I am sure the events always get published.
I tried declaring the publishQueueRetainedBuffer both in retained and regular RAM, no difference.
retained uint8_t publishQueueRetainedBuffer[2048];
PublishQueueAsync publishQueue(publishQueueRetainedBuffer, sizeof(publishQueueRetainedBuffer));
if (publishQueue.getNumEvents()== 0) { //If queue is empty, send publish
Serial.printlnf("Publishing %d datapoints",data_points_queued);
Serial.printlnf("body_buffer: %s",body_buffer);
pub_status=publishQueue.publish("idb", body_buffer, 6000, PRIVATE, WITH_ACK);
}
else {
Serial.printlnf("Waiting for publishQueue");
break;
}
This works flawlessly in an Electron, but I tried the same bit of code on an Argon, and I’ve seen that the call to publishQueue.getNumEvents() always causes a Hard Fault, if I remove that, it works well. I tried in V1.4.4 and 1.5.2 with same results.
Thanks!
You are calling the publishQueue.setup() method from setup(), correct?
If you call publishQueue.getNumEvents() before publishQueue.setup() returns it will almost certainly hard fault because the mutex hasn’t been allocated yet.
1 Like
That was it Rick! You are a genius!
I was thrown off by the fact that without that publishQueue.setup() in the Electron there is no error at all. I have added it and all works well now.
Thanks for such amazing work.
2 Likes
@rickkas7 Quick question.
Now that we have access to the 2MB of external memory on the Gen3 Devices I’m wondering if your awesome library can put it to good use via the flash options you have already added to the library or if new code would be required to make it work?
The library would need new code to allow events to be saved in the LittleFS file system. It’s not a big change, just a subclass that uses the POSIX file system API instead, but I haven’t written it yet.
2 Likes
Good to hear its a simple update.
I’m going to put this to use on a Argon to collect data all day and then upload to the web when Wifi is available.
It’s really, really nice to finally be able to access this extra 2MB of flash! Even better when it works with your library here.
I’m assuming you used this same code on the Tracker One to store and send data when there is no Cellular signal?
There is a new version of PublishQueueAsyncRK that supports the POSIX (LittleFS) file system on the Argon, Boron, B Series SoM, and Tracker SoM. Device OS 2.0.0-rc.3 or later is required.
0.2.0 (2020-11-06)
- Fixed a bug in all file-based implementations (Spiffs, SdFat) where events were not published after a reboot.
- Added a new test suite function (7) to disconnect, post events to the queue, then reboot.
- Added support for storing events on the POSIX file system on Gen 3 devices (Argon, Boron, B Series SoM, and Tracker SoM) in 2.0.0-rc.3 and later.
Rick, is there any fundamental reason why LittleFS could not be made to work on Gen2? Also, if using the LittleFS/POSIX file system on Gen3 means having the event send queue stored on flash memory is this a good idea in terms of wear?