PublishQueueAsyncRK and Large Datasets

@rickkas7 Dude this new library is totally Awesome!

Would this be relatively easy to use to send to something that isn’t particle? My particular case is storing GPS cords and then sending them back to a server I have privately hosted and skips the cloud - for the data anyway, I’ve got some cloud functions to grab the current cords or to reboot the device etc. For the sake of simplicity though, lets just assume I’m not going to publish to the particle cloud etc.

Or is there a library more suited for the task? I’m assuming that I will be frequently out of cellular range, but want it to queue up data to be sent - potentially to an SD card (whatever works really). This isn’t for an electron though, its going through a hotspot (it has multiple uses :-P)

2 Likes

@rickkas7 Just had an idea while looking through the library, is there any issue with forking the library and replacing the following

  • Particle.connect() -> Mesh.connect()
  • Particle.publish() -> Mesh.publish()

It would be great to have a similar library for Mesh only messages.

@emile, the primary reason for the library is to maintain a 1-per-second publish rate limitation set by Particle. Such a limitation does not exist with Mesh publishes. Also, Mesh publishes have no ACK mechanism so they are fast. What is your use case for the new library?

1 Like

@peekay123 Ah ok, nevermind. I was confused, and thought Particle.publish & Mesh.publish operated in a similar fashion.

I am only using Mesh.publish right now to pass data from the edge devices to the gateway. Having the ACK would be great because I’m currently publishing and then reseting my counters (not ideal).

@emile, you can create an ACK by having the gateway do a Mesh.publish() with an “ACK” message back to the node. I believe this was covered in a topic somewhere but I can’t recall where.

1 Like

Great idea, it does add a little data-overhead on the network, but I’ll take it right now.

Thank you again!

Not sure how to frame this question - my previous post hasn’t changed though (back in nov 2019).

Without having done the math, lets say I’ve got a weeks worth of gps data stored every 2 seconds to an sd card (thats a lot more than it’ll ever be - days most likely). Not because thats just where I store it, but because I’ve been out in the boonies and thats the time between having signal and not having signal etc. If my max publish rate is 1 per second with (I’d have to look it up) some max amount of data per publish, I’m guessing going through the cloud isn’t always going to be practical unless I don’t care how long it takes (I do).

Is using a library such as this feasible? My plan was to take what I have now which sends everything back to my system at home and stores it on an SD card that I can go and look at later, but I haven’t come up with a good way to sync those up. In my mind I would think the logic would be something to the effect of:

Have Signal? Send home.
Don’t have Signal? Store on SD
Have Signal again? Send SD data home and resume sending home.

Thats the basic idea - alternately, storing everything on SD and queuing up the data when out of range, then sending it all when I’m back in range “works” too. This actually seems to me more along the lines of what you’re doing with this library (thus my interest).

Am I biting off more than I can chew here? Has no one done this without the cloud? I can go into more details about the setup if need be, but thats the gist of it.

@variable, your approach to storing then sending is good. A Particle.publish() has a max payload of 622 characters (NOT 8-bit bytes) which you can use in a webhook (to send data to a service like Ubidots or others) or to your own subscribed server (using Particle’s javascript library for example). This approach will give you an effective 622 chars/s transfer rate or 37,320 chars/min. However, you do gain full end-to-end encryption.

Another approach, if encrypted data transmission is not a requirement, is to use TCPClient to send the data to a target server directly. This method allows much faster data transfer and also allows you to use full 8-bit byte data if that is required. You can create a FIFO or circular buffer on your SD to store the data and pull it off in “chunks” of whatever size payload you want.

You could also use one of the available TLS encryption libraries on the web IDE to add encryption to the TCPClient approach but the libraries are large and have large memory requirements which may not suit your application.

4 Likes

Yes - I have done this - my model is to create a file on the SD - write all messages to that file as records. The message is a struct with one of the members being a “sent” boolean flag. I have a separate process in loop() that check if we are online and if so, then cycles through the SD records to find any “not sent” and removing them from the file when sent. I use PublishQueueRK library to manage the actually publish as it does it so well.

1 Like

Is there a “for dummies” version of that (or recommended reading material - also for dummies)? Conceptually I completely understand what you’re describing and how that works… syntactically I feel like I’m trying really hard to concentrate on something after having downed a dozen shots of <insert strong alcoholic beverage of choice>

Obviously I don’t want you to do the work for me (well, maybe a little), I’m just trying to figure out how best to budget my time (you really don’t want to see my code… its embarrassing, but at the same time a little astonishing that it works as well as it does for being the bloated mess that it is - I hate starting over on things). Maybe some snippets? I think the part that I’m struggling with at the moment, mostly because I haven’t spent much time doing it, is actually reading/writing to the sd card.

I excel at biting of more than I can chew and then waiting till I’m ready to bury my head in the sand before I ask for help :roll_eyes: <sigh> On the bright side, I know several thousand ways how not to build a light bulb… :stuck_out_tongue:

If your issue is storing the events on SD for later publish, the PublishQueueAsynkRK library does that for you. You just need to instantiate the publishQueue object via the correct constructor and let it do the rest

2 Likes