How to send bulk data to ThingSpeak? (webhooks? JSON/CSV?)

General
1

I’m having trouble with the bulk transfer of data from my particle electron to ThingSpeak, which I want to use to process the data. In brief, what I want to do with the device is to have it mostly in deep sleep mode, but periodically (every 10 minutes or so) wake up with GSM module off and take measurements. It will then publish the recorded data much less frequently - probably once a day.

I can use a retained array or string to record the data while in deep sleep mode, and have been able to use example code and webhook to immediately publish data points to ThingSpeak, but this only publishes one data point and it plots the timestamp of the publish rather than the data.

I have done a lot of reading around the subject, and am hampered slightly by my lack of experience. I believe that the best method is to use the bulk transfer given here, but I don’t know how to implement it: https://uk.mathworks.com/help/thingspeak/bulk-update-a-channel-feed-1.html#bvqi_8p

All the example code for this sort of thing seems to be written for the Photon, and I am struggling with even knowing if the basic methods used are compatible with the Electron, let alone converting it. I am also concerned about something I have seen on a support forum about there being a 255 char limit on webhooks, which would not be big enough for my purposes.

Can anyone give me some pointers on implementing the bulk transfer in the link (the firmware side in particular, neither JSON nor CSV are even mentioned in the Particle firmware reference), or else point me towards a different method that will fulfil my requirements?

Sorry this may seem a rather general post, I’ve just found myself getting very bogged down in http protocol details and similar and I don’t feel like I’m getting anywhere, any direction would be much appreciated.

2

Are you wanting to perform a Bulk Data update with Saving Data or Saving Power in mind?

There are several great posts on this forum with detailed use cases for Data Usage.
It is likely that you will use less data by publishing every 10 minutes, instead of performing a full Modem handshake once a day.

It’s very possible to update (8) ThingSpeak fields every 10 minutes and stay between 1 to 2 MB per month.

3

Thanks for your reply,

I’m doing it with power saving in mind, hence the deep sleep mode, and keeping the cellular module on all the time is not preferable because it will need to run for a long time (months to years) on a LiPo battery and a small(ish) solar panel.

My question with the data usage is: is it possible to send the quantity of data I want to send and have it picked up by one webhook? Or would it require maybe a looped iteration through an array with a publish for each set of data?

4

Each publish is somewhat limited in size at 255bytes so the simplest solution is indeed to store these values in something like an array in SRAM with timestamps, then once a day send them one at a time. Don’t forget publish is rate limited so it is going to take just over 2 minutes to send a days records, you might also want to run sync as well to make sure your clock is reasonably accurate.