Best way to combine data for Particle.Publish?

Cloud
1

Hello Particle Community,

I’m trying to get a photon to send a json array to a google sheets endpoint and have created a structure that I save data from a sensor into.

struct pms5003data {
  uint16_t framelen;
  uint16_t pm10_standard, pm25_standard, pm100_standard;
  uint16_t pm10_env, pm25_env, pm100_env;
  uint16_t particles_03um, particles_05um, particles_10um, particles_25um, particles_50um, particles_100um;
  uint16_t unused;
  uint16_t checksum;
};

I then put all that data into an array.

 int PMArray[] ={  data.pm10_standard, data.pm25_standard, data.pm100_standard, 
                          data.pm10_env, data.pm25_env, data.pm100_env,
                          data.particles_03um, data.particles_05um, data.particles_10um, data.particles_25um, 
                          data.particles_50um, data.particles_100um 
        };

Becuase it apears the Partcle.publish() funtion can only take strings I then convert the int data into char and then to strings.

char PM1_0_S[32];
            sprintf(PM1_0_S,"{\"PM1_S\":%d}", PMArray[0]);
            char PM25_S[32];
            sprintf(PM25_S,"{\"PM2.5_S\":%d}", PMArray[1]);
            char PM10_S[32];
            sprintf(PM10_S,"{\"PM10_S\":%d}", PMArray[2]);
            char PM1_E[32];
            sprintf(PM1_E,"{\"PM1_E\":%d}", PMArray[3]);
            char PM25_E[32];
            sprintf(PM25_E,"{\"PM2.5_E\":%d}", PMArray[4]);
            char PM10_E[32];
            sprintf(PM10_E,"{\"PM10_E\":%d}", PMArray[5]);
            char PM03[32];
            sprintf(PM03,"{\"PM0.3\":%d}", PMArray[6]);
            char PM05[32];
            sprintf(PM05,"{\"PM0.5\":%d}", PMArray[7]);
            char PM1[32];
            sprintf(PM1,"{\"PM1.0\":%d}", PMArray[8]);
            char PM25[32];
            sprintf(PM25,"{\"PM2.5\":%d}", PMArray[9]);
            char PM5[32];
            sprintf(PM5,"{\"PM5.0\":%d}", PMArray[10]);
            char PM10[32];
            sprintf(PM10,"{\"PM10\":%d}", PMArray[11]);

But I’m stuck on a good way to combine all the strings of data into one string of data that can be sent to the cloud to be used in a webhook. The goal is to only have to use one Particle.publish("jsonData", PM03, 60, PRIVATE); function to get all the data into a web server for post-processing.

Any recomendation would be appreciated, thanks.

1 Like
2

You can either use snprintf() to feed all data into one big string or you use JsonParserGeneratorRK to create a JSON object and then send it’s buffer (e.g. Particle.publish("jsonData", json.getBuffer(), PRIVATE);).
No need for the intermediate int array nor the dedicated (sub)strings in both cases.

4 Likes
3

Scruff, is this an appropriate implementation of your suggestion? The NEW and the OLD code are listed. Both work, but I assume there must be some advantages to the NEW code implementation.

// NEW CODE ********************************************
//
// Publishes the status, in my case: specifically sends it to a Google spreadsheet and the PoolController Android app
bool publishAllStatus() {
    char buffer[622];
    int cx;

    cx = snprintf(buffer, 622,
        "{\"T0\":"  + String(f_current_temps[0], 1) +    // "Header name" of identified Google SHEET column where this data is recorded 
        ",\"T1\":"  + String(f_current_temps[1], 1) +   
        ",\"T2\":"  + String(f_current_temps[2], 1) +
        ",\"T3\":"  + String(f_current_temps[3], 1) +
        ",\"T4\":"  + String(f_current_temps[4], 1) +
        "}"
    );
    return publishNonBlocking("temp" /*identifies Google spreadsheet */, buffer);                                   
}


// OLD CODE ********************************************
//
// Publishes the status, in my case: specifically sends it to a Google spreadsheet and the PoolController Android app
 bool publishAllStatus() {
    return publishNonBlocking(
        "temp",                                         // identifies the specific Google spreadsheet SHEET to record this data
        "{\"T0\":"  + String(f_current_temps[0], 1) +    // "Header name" of identified Google SHEET column where this data is recorded
        ",\"T1\":"  + String(f_current_temps[1], 1) +   
        ",\"T2\":"   + String(f_current_temps[2], 1) +
        ",\"T3\":"   + String(f_current_temps[3], 1) +
        ",\"T4\":"   + String(f_current_temps[4], 1) +
        "}");}


// A wrapper around Partical.publish() to check connection first to prevent
// blocking. The prefix "pool-" is added to all names to make subscription easy.
bool publishNonBlocking(String name, String message) {
    // TODO replace with a failure queue?
    if (Particle.connected()) {
        bool success = Particle.publish("pool-" + name, message, PUBLIC); // TODO, need to understand ramifications of making this PRIVATE
//        Serial.printlnf("Published \"%s\" : \"%s\" with success=%d",
//                        name.c_str(), message.c_str(), success);
        return success;
    } else {
//        Serial.printlnf("Published \"%s\" : \"%s\" with success=0 no internet",
//                        name.c_str(), message.c_str());
    }
    return false;
}

I guess I didn't end up using "int cx" anywhere so I guess I could eliminate that variable...but eliminating that seems to eliminate the need for the snprintf command entirely...now I'm a bit confused, possibly I don't need to do this for this case because I don't need the option to add more variables to the string before I publish it? So possibly the original, OLD code is ok?

Here is an example of the data being published:

{"T0":79.3,"T1":97.6,"T2":79.1,"T3":80.5,"T4":79.6}

For a Followup:

What about particle variables?

Particle.variable("GetStatP", statPage);

I am combining a bunch of variables into statPage as follows: Should I be combining those using snprintf() before setting the particle variable? I actually use many more variables than are listed.

    statPage = 
        String(currentPumpStatus) + ";" +       // 0
        currentPumpMode + ";" +                 // 1
        currentPumpCountdown + ";" +            // 2
        currentPumpRPM + ";" +                  // 3
        currentPumpRPMSetpoint + ";" +          // 4
        currentPumpRPMPercent + ";" +           // 5
        currentPumpWatts + ";" +                // 6
        String(currentPumpEFactor,1) + ";" +    // 7
        currentPumpGPM;                   // 8
4

Not quite. It's still using String which is something I always try to avoid. The purpose of using snprintf() is the f in the function name which stands for formatted.
Your first example above would rather look like this

snprintf(buffer, sizeof(buffer)
        , "{ \"T0\":%.1f"
          ", \"T1\":%.1f"
          ", \"T2\":%.1f"
          ", \"T3\":%.1f"
          ", \"T4\":%.1f"
          "}"
        , f_current_temps[0]
        , f_current_temps[1]
        , f_current_temps[2]
        , f_current_temps[3]
        , f_current_temps[4]
        );

Although I'd probably build that in a loop.

No, not before. With Particle.variable() you only register statPage with the cloud for requesting its then current value some time in the future. Particle.variable() doesn't push the value to the cloud.
And yes, I'd use snprintf() as shown above to construct the string and statPage should be declared as a global char statPage[nSize] (nSize being big enough to hold the longest expected string).

I'd also drop String as function parameters and rather go with something like this

const char evtPrefix[] = "pool-";

bool publishNonBlocking(const char* name, const char* message) {
  char evtName[sizeof(evtPrefix) + strlen(name)];
  snprintf(evtName, sizeof(evtName), "%s%s", evtPrefix, name);
  ...
}
2 Likes
5

Thanks for the help and the tutorial ! I think I “got” everything…

6

I'm curious how you'd build a formatted string in a loop efficiently, especially if you didn't know ahead of time how many items you'd be outputting. I can see how you could manually build the format string, keeping track of the index in the char array, but how would you dynamically save up all the values to be used with that (ie f_current_temps[0], f_current_temps[1], etc) and recombine them all at the end for the snprintf call?

7

Not to step on ScruffR’s toes, but here is one approach to start a discussion (unbuilt, untested)

char * buildPublishString(float * temps, size_t num_temps, char * dest_buffer, size_t dest_buffer_size)
{
   char    temp_buffer[30];

   strcpy(dest_buffer, "{ ");

   for (int index = 0; index < num_temps; ++index)
   {
      snprintf(temp_buffer, sizeof(temp_buffer), "\"T%d\":%.1f", index, temps[index]);
      strncat(dest_buffer, temp_buffer, dest_buffer_size - strlen(dest_buffer) - 1);

      if (index < (num_temps - 1))
      {
         strncat(dest_buffer, ", ", dest_buffer_size - strlen(dest_buffer) - 1);
      }
      else
      {
         strncat(dest_buffer, "}", dest_buffer_size - strlen(dest_buffer) - 1);
      }
   }

   return (dest_buffer);
}

// You must carefully select and manage the buffer size.  The build function shouldn't overflow, but could truncate.

char    publish_buffer[200];

buildPublishString(f_current_temps, 5, publish_buffer, sizeof(publish_buffer));
8

If you have an indeterminate number of variables, then you have to make some decisions on how to store them. You could use an array but you will have to guesstimate the maximum number of elements since you cannot dynamically change the array size (assuming the array is declared as a global variable.) There are C objects that are dynamic in size (i.e. vectors) but they can be problematic as they can cause memory fragmentation over time. So which will it be? Guesstimate the maximum number of elements, or risk memory fragmentation over time?

I guess a third option would be to use larger storage hardware such as external Flash, EEPROM or FRAM.

After you determine how to store the data, you can estimate what you largest string to publish might be. The current Particle.publish() limitation is 622 bytes so that's a hard plateau you have to plan for. Otherwise, if you are transmitting another way you may have a different limitation to plan for. (i.e. IIRC UDP has to be broken up into 512-byte chunks when sending from an Electron, TCP packet size is limited to the MTU setting of the network, serial has a 64-byte buffer but you can send unlimited data as long as the transmitter and receiver are programmed to handle the stream.)

Maybe @ScruffR, the seasoned C veteran, has a better solution but that's my take on it.

1 Like
9

Thanks, hadn’t even thought of doing the replacement piece by piece and just combining those outputs.

Thanks, @ninjatill. I was thinking of a set of sensors, so the max would be relatively low, probably 8. And the output string would always be well within the limit. Up until now I’ve been doing static checks; ie multiple output options and I pick the one for the sensors currently connected. Or the very inefficient way of using string concatenation; ie “x:” + String(x) + “,y:” + String(y).

Always looking to improve my methods, thanks both.

10

That depends on the kind of efficiency you are after. I often go with the "shortest" code rather than fastest execution as it may be more efficient for maintenance.
For that I'd do something along this line

  char buf[maxLen] = "{";
  for (int i=0; i < nSensors; i++) {
    snprintf(buf, sizeof(buf)
            , "%s\"T%d\":%.1f%s"
            , buf
            , i
            , f_current_temps[i]
            , (i < nSensors-1) ? "," : "}"
            );
  }

If speed would play a role, I'd avoid the "self-reference" in the format string and keep track of the end of the string by catching the return value of snprintf() and adding to the string length myself (saving the time strncat() and strlen() would need to count the characters and copy the data).

If you need to be really dynamic at runtime, then @ninjatill's arguments are to be considered.
But if you mean dynamic as keeping the code agnostic to the size of your array but define a fixed number of sensors you can do this

const int pinSensors[] = { D1, D2, D4, A4, A6, A7 }; // assuming one pin per sensor
const int nSensors     = sizeof(pinSensors) / sizeof(pinSensors[0]);
const int maxLen       = nSensors * 16 + 4;          // allow 16 byte per sensor reading 
                                                     // plus some for prefix and suffix
float     f_current_temps[nSensors];

This way the only thing you'd need to change if you add or remove a sensor would be to add/remove the sensor pin in pinSensors[] - the rest is done for you.

4 Likes