Uploading locally logged data (stored on an SD) to a cloud server

Hi guys - I’m just starting my IoT journey, so apologies if this is a really basic question.

I need to create a system to sample data every second from a range of sensors, and log it locally on an SD card (in a time-stamped csv txt file) then send the data file to a cloud server (either when requested by the server (webhook?), or every hour or so).

Any advice or examples would be really appreciated. The logging to SD I’m sure I can muddle through, but I’m really out of my depth on the subsequent sending of the file (not just the values) to the cloud server.

[Logging the data directly in the cloud - thanks for the great examples out there - isn’t an option as the system is for use in remote locations, and wifi coverage (and eventually cellular with the electron) is not guaranteed.]

Thanks,

There is a wider range of possibilities for you, but I guess searching the docs and forum for TCPServer, TCPClient and HTTPClient might come up with some interesting reads that might suit your needs.


After I got a pointer of my highly admired friend @peekay123, I’d like to add that using TCP might not be the best thing to do, if your data is confidential, since it’s not encrypted.
If this is OK for you, that’d be an easy path, if not there might be other options to be discussed.

2 Likes

You could use Spark.publish to the Particle Cloud and use webhooks if you need to send it to another server after that.

You’ll just have to be careful and consider some of the edge cases. You could figure out the max size publishable using the Particle APIs and publish chunks of the CSV file…deleting them once you receive a successful response from the server. Additionally, you’ll have to stay within the bounds of the throttles to be a good cloud citizen…you’ll want to design things so that you don’t just push as fast as possible blindly.

To stay within publish throttles and minimize data you have to push you can be smart about what type of sensor data you actually need. For example…if it is a temperature sensor maybe you take a absolute reading once an hour and afterward you only record a new reading if the temperature changes value sufficiently…allowing the server to interpolate the intermediate values. (No point in wasting your publish throttles pushing the temperature values every minute…70, 70, 70, 70, 70, …).

You’ll need to build in retries as well so that if you get a failure you stop and retry that chunk after some time.

TCPClient and HTTPClient as suggested already would be fine if you aren’t willing to use the Particle Cloud but I didn’t see that as a requirement in your post. Everything above would apply to those as well.

1 Like

Thanks both, great pointers

@ScruffR Data confidentiality isn’t an biggie at this point, I’d settle for just getting the file there (and worrying about encryption later). I found lots of good examples for setting up the TCP connection (Using a Camera and Sending JPEG to cloud, and [TCP Server and Client Example Socket Program] 2), but I’m still a little lost as to how to reference, chunk, and ‘send’ the file from the SD - any chance you could point me in the direction of an example?

@chrisp Thanks for the general tips - especially on the data-logging. Very happy to use the Particle Cloud - Im trying to keep everything as simple as possible though, and @zachary mentioned [here] 3 its easier to send to a private server. Again though, any examples/project shares of sending stored data through the particle cloud would be of huge help.

As I understand you already can store your data on SD, so I guess you’re using an SD library like SD-CARD-LIBRARY on Particle Build (aka Web IDE).

In that library you find the sample SPARK-READWRITE.cpp and in there you find this bit of code

   // read from the file until there's nothing else in it:
    while (myFile.available()) {
      Serial.write(myFile.read());
    }

So if you replace the while() with a for() loop that only reads chewable chunks, you are almost there.
And sending is just as easy as Serial.write(), just replace Serial with your respective client object.