Uploading locally logged data (stored on an SD) to a cloud server

Hi guys - I’m just starting my IoT journey, so apologies if this is a really basic question.

I need to create a system to sample data every second from a range of sensors, and log it locally on an SD card (in a time-stamped csv txt file) then send the data file to a cloud server (either when requested by the server (webhook?), or every hour or so).

Any advice or examples would be really appreciated. The logging to SD I’m sure I can muddle through, but I’m really out of my depth on the subsequent sending of the file (not just the values) to the cloud server.

[Logging the data directly in the cloud - thanks for the great examples out there - isn’t an option as the system is for use in remote locations, and wifi coverage (and eventually cellular with the electron) is not guaranteed.]

Thanks,

There is a wider range of possibilities for you, but I guess searching the docs and forum for TCPServer, TCPClient and HTTPClient might come up with some interesting reads that might suit your needs.


After I got a pointer of my highly admired friend @peekay123, I’d like to add that using TCP might not be the best thing to do, if your data is confidential, since it’s not encrypted.
If this is OK for you, that’d be an easy path, if not there might be other options to be discussed.

2 Likes

You could use Spark.publish to the Particle Cloud and use webhooks if you need to send it to another server after that.

You’ll just have to be careful and consider some of the edge cases. You could figure out the max size publishable using the Particle APIs and publish chunks of the CSV file…deleting them once you receive a successful response from the server. Additionally, you’ll have to stay within the bounds of the throttles to be a good cloud citizen…you’ll want to design things so that you don’t just push as fast as possible blindly.

To stay within publish throttles and minimize data you have to push you can be smart about what type of sensor data you actually need. For example…if it is a temperature sensor maybe you take a absolute reading once an hour and afterward you only record a new reading if the temperature changes value sufficiently…allowing the server to interpolate the intermediate values. (No point in wasting your publish throttles pushing the temperature values every minute…70, 70, 70, 70, 70, …).

You’ll need to build in retries as well so that if you get a failure you stop and retry that chunk after some time.

TCPClient and HTTPClient as suggested already would be fine if you aren’t willing to use the Particle Cloud but I didn’t see that as a requirement in your post. Everything above would apply to those as well.

1 Like

Thanks both, great pointers

@ScruffR Data confidentiality isn’t an biggie at this point, I’d settle for just getting the file there (and worrying about encryption later). I found lots of good examples for setting up the TCP connection (Using a Camera and Sending JPEG to cloud, and [TCP Server and Client Example Socket Program] 2), but I’m still a little lost as to how to reference, chunk, and ‘send’ the file from the SD - any chance you could point me in the direction of an example?

@chrisp Thanks for the general tips - especially on the data-logging. Very happy to use the Particle Cloud - Im trying to keep everything as simple as possible though, and @zachary mentioned [here] 3 its easier to send to a private server. Again though, any examples/project shares of sending stored data through the particle cloud would be of huge help.

As I understand you already can store your data on SD, so I guess you’re using an SD library like SD-CARD-LIBRARY on Particle Build (aka Web IDE).

In that library you find the sample SPARK-READWRITE.cpp and in there you find this bit of code

   // read from the file until there's nothing else in it:
    while (myFile.available()) {
      Serial.write(myFile.read());
    }

So if you replace the while() with a for() loop that only reads chewable chunks, you are almost there.
And sending is just as easy as Serial.write(), just replace Serial with your respective client object.

Hello, I have a similar project and need some help!

I would like to store sensor data (currently just U, S, F, V) into SD card every 1 min and publish the stored data every hour. While the Particle Electron collects and stores the sensor data, the cellular is off. And then, the cellular is on every hour to publish the stored data.

I was successfully able to do this using PublishQueueAsyncRK library with Webhook. However, I have been trying to do this using HTTPCLIENT library, but I could not make this work…

So far, I was able to turn the cellular off, while the particle collects and stores the data. Then, turn the cellular on every hour. While the cellular is on, it successfully publish the data at the moment. However, I am not sure how to publish the stored data…

If you could help with this, I really appreciate it!

Below is the code that I have so far.

#include <SdFat.h>
#include <HttpClient.h>
#include "application.h"

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED);

int SD_INITIALISED = 0;
int counter1 = 0;
bool CLOUD_HAS_CONNECTED = FALSE;

SdFat sd; 
SdFile file;	
File myFile;

int U = 111;
int S = 222;
int F = 333;
int V = 444;

HttpClient http;

// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers[] = {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    { "Accept" , "*/*"},
    { "User-agent", "Particle HttpClient"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

http_request_t request;
http_response_t response;

#define SPI_CONFIGURATION 0
#define SPI_SPEED SD_SCK_MHZ(4)
// Setup SPI configuration.
#if SPI_CONFIGURATION == 0
// Primary SPI with DMA
// SCK => A3, MISO => A4, MOSI => A5, SS => A2 (default)
const uint8_t chipSelect = SS;
#elif SPI_CONFIGURATION == 1
// Secondary SPI with DMA
// SCK => D4, MISO => D3, MOSI => D2, SS => D1
SdFat sd(1);
const uint8_t chipSelect = D1;
#elif SPI_CONFIGURATION == 2
// Primary SPI with Arduino SPI library style byte I/O.
// SCK => A3, MISO => A4, MOSI => A5, SS => A2 (default)
SdFatLibSpi sd;
const uint8_t chipSelect = SS;
#elif SPI_CONFIGURATION == 3
// Software SPI.  Use any digital pins.
// MISO => D5, MOSI => D6, SCK => D7, SS => D0
SdFatSoftSpi<D5, D6, D7> sd;
const uint8_t chipSelect = D0;
#endif  // SPI_CONFIGURATION


void setup() {
    Serial.begin(9600);
    while (!Serial) {
    SysCall::yield();
  }
  
  if (!sd.begin(chipSelect, SD_SCK_MHZ(4))) {
    sd.initErrorHalt();
  }
  
  if (!myFile.open("test.txt", O_RDWR | O_CREAT | O_AT_END)) {
    sd.errorHalt("opening test.txt for write failed");
  }
  myFile.close();

  if (!myFile.open("test.txt", O_READ)) {
    sd.errorHalt("opening test.txt for read failed");
  }
  Serial.println("test.txt content:");

  int data;
  while ((data = myFile.read()) >= 0) {
    Serial.write(data);
  }
  myFile.close();
  }


void loop() {



    if(counter1>=60){
        connectCloudAndCell(5);
        if(counter1>=65){
            disconnectCloudAndCell();
            counter1=0;
        }
    } else {
          
    }
    
      myFile.open("test.txt", O_RDWR | O_CREAT | O_AT_END);
      Serial.print(U);
      Serial.print("   ");
      Serial.print("S");
      Serial.print("   ");
      Serial.print(S);
      Serial.print("   ");
      Serial.print("F");
      Serial.print("   ");
      Serial.print(F);
      Serial.print("   ");
      Serial.print("V");
      Serial.print("   ");
      Serial.print(V);
      Serial.print("   ");
      Serial.print(counter1);
      Serial.println("");
      
      myFile.print(U);
      myFile.print(",");
      myFile.print(S);
      myFile.print(",");
      myFile.print(F);
      myFile.print(",");
      myFile.print(V);
      myFile.println("");
      myFile.close();
     

    if(counter1<100){

    Serial.println();
    Serial.println("Application>\tStart of Loop.");
    request.hostname = "abc.def.com"; //the address is confidential    
    request.port = 80;
    char data[256];
    snprintf(data, sizeof(data), "/subaddress?U=%d&S=%d&F=%d&V=%d", U, S, F, V); //the address is confidential  
    request.path = data;     
    http.get(request, response, headers);
    Serial.print("Application>\tResponse status: ");
    Serial.println(response.status);
    Serial.print("Application>\tHTTP Response Body: ");
    Serial.println(response.body);
}

  delay(60000); 
  counter1++;
}

void connectCloudAndCell(int mins){
    Serial.println(" ");
    Serial.println("CONNECT FUNCTION:");
    Serial.println("----->TURNING ON CELL...");
    Cellular.on();
    Serial.println("----->CONNECTING TO CELL...");
    Cellular.connect();
    if (waitFor(Cellular.ready, mins*60000)) {
        Serial.println("----->TURNING ON CLOUD...");
        if (!Particle.connected()) Particle.connect();
        if (waitFor(Particle.connected, mins*60000)) {
            connectionStatus();
        } else {
            Serial.println("------------------------>ERROR COULD NOT CONNECT TO CLOUD");
            connectionStatus();
        }
    } else {
        Serial.println("------------------------>ERROR COULD NOT CONNECT TO CELLULAR");
        connectionStatus();
    }
    if (Particle.connected()) CLOUD_HAS_CONNECTED = TRUE;
    Serial.println(" ");
}

void disconnectCloudAndCell(){
    if (Particle.connected()) {
        Serial.println(" ");
        Serial.println("DISCONNECT FUNCTION:");
        Serial.println("-----> TURNING OFF CLOUD...");
        Serial.println("-----> TURNING OFF CELL...");
        while(Particle.connected()){
            Particle.disconnect();
            delay(500);
        }
    }
    Cellular.off();
    delay(500);
    connectionStatus();
    Serial.println(" ");
}

void connectionStatus(){
    Serial.println("CELL CONNECTED: " + String(Cellular.ready()));
    Serial.println("CLOUD CONNECTED: " + String(Particle.connected()));
}

First you may want to have a look at the SDFat samples for reading back data.
e.g. Particle Web IDE

Once you can read the data into a local buffer, you can send that buffer's contents whatever way you want.

Once you have that you may want to work out a strategy on how to mark already sent entries so that you can pick up after the last sent record in case you cannot send all data in during one upload cycle.

BTW, the snprintf() scheme that you already use for building the request.path would also fit well to replace all these individual print statements here

Thank you, @ScruffR!

I am still not able to make this work yet. For some reason, the above “ReadCsvArray” example keeps giving me an error message about the below line.

#define errorHalt(msg) {Serial.println(F(msg)); SysCall::halt();}

Just another thought is that is it possible to put the values in the dlist, then publish this?

int dlist[4] = {0,0,0,0};
void loop() {
    myFile.open("test.txt", O_RDWR | O_CREAT | O_AT_END);
    myFile.print(U);
    myFile.print(",");
    myFile.print(S);
    myFile.print(",");
    myFile.print(F);
    myFile.print(",");
    myFile.print(V);
    myFile.println("");
    myFile.close();
     
    if (cnt <= 3){
    dlist[cnt]=S;
    }
    else{
        Serial.println("sending a data");
        SendData();
        cnt = 0;
    }
      
    cnt = cnt +1;
    delay(10000);
}

Updated: I was able to make the SDfat sample library work now. What should I do for the next step?

#include <HttpClient.h>
#include <SdFat.h>
#include "application.h"

// 5 X 4 array
#define ROW_DIM 5
#define COL_DIM 4

int SD_INITIALISED = 0;
int counter1 = 0;
byte cnt = 0;

SdFat sd; 
SdFile file;	
File myFile;

int U = 91720;
int S = 0;
int F = 75;
int V = 0;

HttpClient http;
http_header_t headers[] = {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    { "Accept" , "*/*"},
    { "User-agent", "Particle HttpClient"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

http_request_t request;
http_response_t response;

#define SPI_CONFIGURATION 0
#define SPI_SPEED SD_SCK_MHZ(4)
// Setup SPI configuration.
#if SPI_CONFIGURATION == 0
// Primary SPI with DMA
// SCK => A3, MISO => A4, MOSI => A5, SS => A2 (default)
const uint8_t chipSelect = SS;
#elif SPI_CONFIGURATION == 1
// Secondary SPI with DMA
// SCK => D4, MISO => D3, MOSI => D2, SS => D1
SdFat sd(1);
const uint8_t chipSelect = D1;
#elif SPI_CONFIGURATION == 2
// Primary SPI with Arduino SPI library style byte I/O.
// SCK => A3, MISO => A4, MOSI => A5, SS => A2 (default)
SdFatLibSpi sd;
const uint8_t chipSelect = SS;
#elif SPI_CONFIGURATION == 3
// Software SPI.  Use any digital pins.
// MISO => D5, MOSI => D6, SCK => D7, SS => D0
SdFatSoftSpi<D5, D6, D7> sd;
const uint8_t chipSelect = D0;
#endif  // SPI_CONFIGURATION


size_t readField(File* file, char* str, size_t size, const char* delim) {
  char ch;
  size_t n = 0;
  while ((n + 1) < size && file->read(&ch, 1) == 1) {
    // Delete CR.
    if (ch == '\r') {
      continue;
    }
    str[n++] = ch;
    if (strchr(delim, ch)) {
        break;
    }
  }
  str[n] = '\0';
  return n;
}

// //------------------------------------------------------------------------------
// #define errorHalt(msg) {Serial.println(F(msg)); SysCall::halt();}
// //------------------------------------------------------------------------------

void setup() {
    Serial.begin(9600);
 //   Time.zone(-4);
    
    pinMode(trigPin, OUTPUT);
    pinMode(echoPin, INPUT);
    attachInterrupt(echoPin, echo_interrupt, CHANGE); 


    while (!Serial) {
    SysCall::yield();
  }
  
  if (!sd.begin(chipSelect, SD_SCK_MHZ(4))) {
    sd.initErrorHalt();
  }
  
  if (!myFile.open("test.txt", O_RDWR | O_CREAT | O_AT_END)) {
    sd.errorHalt("opening test.txt for write failed");
  }
  myFile.close();

  if (!myFile.open("test.txt", O_READ)) {
    sd.errorHalt("opening test.txt for read failed");
  }
  Serial.println("test.txt content:");


  // Rewind file so test data is not appended.
  myFile.rewind();

  // Write test data.
  myFile.print(F(
    "11,12,13,14\r\n"
    "21,22,23,24\r\n"
    "31,32,33,34\r\n"
    "41,42,43,44\r\n"
    "51,52,53,54"     // no delimiter
    ));

  // Rewind the file for read.
  myFile.rewind();


  // Array for data.
  int array[ROW_DIM][COL_DIM];
  int i = 0;     // First array index.
  int j = 0;     // Second array index
  size_t n;      // Length of returned field with delimiter.
  char str[20];  // Must hold longest field with delimiter and zero byte.
  char *ptr;     // Test for valid field.

  // Read the file and print fields.
  
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      n = readField(&myFile, str, sizeof(str), ",\n");
      if (n == 0) {
       // errorHalt("Too few lines");
      }
      array[i][j] = strtol(str, &ptr, 10);
      if (ptr == str) {
//errorHalt("bad number");
      }
      while (*ptr == ' ') {
        ptr++;
      }
      if (*ptr != ',' && *ptr != '\n' && *ptr != '\0') {
      //  errorHalt("extra characters in field");
      }
      if (j < (COL_DIM-1) && str[n-1] != ',') {
    //    errorHalt("line with too few fields");
      }
    }
    // Allow missing endl at eof.
    if (str[n-1] != '\n' && myFile.available()) {
    //  errorHalt("missing endl");
    }    
  }

  // Print the array.
  for (i = 0; i < ROW_DIM; i++) {
    for (j = 0; j < COL_DIM; j++) {
      if (j) {
        Serial.print(' ');
      }
      Serial.print(array[i][j]);
    }
    Serial.println();
  }
  Serial.println("Done");
  myFile.close();
  
  Serial.println(myFile);
  
  }

int dlist[5] = {0,0,0,0,0};

void loop() {
      myFile.open("test.txt", O_RDWR | O_CREAT | O_AT_END);
      myFile.print(U);
      myFile.print(",");
      myFile.print(S);
      myFile.print(",");
      myFile.print(F);
      myFile.print(",");
      myFile.print(V);
      myFile.println("");
      myFile.close();
     
    if (cnt <= 3){
    dlist[cnt]=S;
    }
    else{
        Serial.println("sending a data");
        SendData();
        cnt = 0;
    }
      
    cnt = cnt +1;
    delay(10000);
}

 

//Send data  
void SendData(){

    Serial.println();
    Serial.println("Application>\tStart of Loop.");
    request.hostname = "abc.def.com"; //the address is confidential    
    request.port = 80;
    char data[256];
    snprintf(data, sizeof(data), "/Cabs/asdf?U=%d&S=%d&F=%d&V=%d", U, S, F, V); //the address is confidential  
    request.path = data;     
    http.get(request, response, headers);
    Serial.print("Application>\tResponse status: ");
    Serial.println(response.status);
    Serial.print("Application>\tHTTP Response Body: ");
    Serial.println(response.body);

    delay(1000);
    Serial.print("sent all data, all done!");
    }
  
  }

Try to understand how that example works, distill the info needed for your application and apply that acquired knowledge to your project.

Another thing would be to apply what was already said to show the ability to convert verbal descriptions into running code :wink:

So for the issue at hand:

Writing data:

  • you want to write the data as you later want it to be published - one line per record

Reading data:

  • read all bytes till you find a '\n'
  • once you got a complete record -> send
  • rinse, repeat

Thank you, @ScruffR.

I will work on it based on your suggestion!

Also, could you please check if I transfer this httpclient code into webhook correctly? Below are the code and webhook setup.

    Serial.println("Application>\tStart of Loop.");
    request.hostname = "abc.def.com"; //the address is confidential    
    request.port = 80;
    char data[256];
    snprintf(data, sizeof(data), "/hello/def?U=%d&S=%d&F=%d&V=%d", U, S, F, V); //the address is confidential  
    request.path = data;     
    http.get(request, response, headers);
    Serial.print("Application>\tResponse status: ");
    Serial.println(response.status);
    Serial.print("Application>\tHTTP Response Body: ");
    Serial.println(response.body);

image

Not sure how these two should act together?
If you want your Electron to act as server for that webhook, then you are out of luck as it cannot (easily) work with HTTPS
Second, your webhook uses POST but your code uses GET - that is also incompatible.
Finally, if you want to test a webhook you can use services like requestbin.com to see whether your requests are well formed. Once you have ensured of that you can use the webhook to see what your actual target server responds and work from that.

Thank you, @ScruffR, for comments.

Thanks to your comments, I think this is going well. I was able to put the numbers in array, then send them to server via Httpclient.

I have a question for you, is there anyway I can put date and time (“9/25/2020 12:11:10”) into array?

For numbers, I was able to put them in a float as shown in below code. I am not sure how I can put the time(string) into array…

String time = Time.format(Time.now(), “%m/%d/%Y %H:%M:%S”);

Thank you for your help!

float temp[256];

void loop() {

if (cnt <= 3){

float temperature = analogRead(A0)*1.23;
temp[i] = temperature;
cnt++;

}
else{

SendData();
cnt = 0;

}

If you want to have a combination of a float with a timestamp, I’d create a struct that packages the related data into a single entity and then have an array of that type.

Since you are already transferring binary data, I’d not use the string representation for the timestamp but rather the integer value of Time.now() directly.

With all that I’d do something along this line

struct tsf_t { // timestamped float
  int   ts;
  float val;
};
tsf_t data[256]

...
  data[i].ts  = Time.now();
  data[i].val = temperature;

On the receiving side you’d unpack the data and reformat as needed.

Alternatively you could create a JSON formatted packet from the binary data on the device’s side. This would avoid the receiving side from needing to know how to unpack the binary data.