Storing very infrequent data while Photon offline (not cloud connected)

Getting Started
1

Hello Particle friends.

I am building a device with a Photon as the MCP and moisture sensors. Very simple. I want to put it in a location where it is likely it will go for multiple days without internet connectivity. I have code in place that logs to the cloud and it works fine, but as soon as it disconnects, it will stop logging (as it stands).

Based on what it is for (gardening - soil moisture to be exact) and the data it collects (a timestamp and a voltage reading from up to 6 sensors), I believe my simplest answer is to store the data in array if Particle.connected() returns false. I have looked all over the internet and the forum for answers but most people appear to want to store LOTS of data points. I will check the readings at most 2x per day (could even go to 1 TBH, but 2 is better) so if I am offline for even 72 hours, I would have 6 timestamps and worst case 36 4-digit voltage readings (it ignores disconnected sensors) - even with formatting it would be under 512kb at worst (ish). Oh and I very much want to avoid SD cards and all that as I already have PCBs and want this prototype to be simpler than all that would entail.

So all that said, the root of the question is: I believe that I shouldn’t overrun the memory on the photon with that little amount of data, but I cannot for the life of me figure out the way to store the data itself in C code. I have spent HOURS searching Stack, Google, here, and other sites for a simple “dynamic arrays” type tutorial, but none of them are quite what I am looking for (I know very little C/C++ - I am a Python primary) so I figured I would ask here as there is no way I am the only one who’s done this?

Example of one of the reading blocks from my code:

// Check the value of the probe - if > 4000 probe not connected so skip reading it
if (SMVolts1 < 4000) {
    String voltage = "Voltage " + String(SMVolts1);
    String publishdata = timestamp + " " + voltage;
        if (Particle.connected()) {
          Particle.publish("S1_Reading_Data", publishdata, PRIVATE);
        }
        else {
          INSERT CODE HERE THAT STORES THIS READING INTO AN ARRAY THAT IS DYNAMIC
        }
  }
  delay(1000);

In the final production version I will most likely reverse the loops you see here so that it checks connectivity first, THEN takes the reading or not. That way i can store a “not connected” or something like it for missing probes. Does not change the “I need to store it in an array” part of the equation though.

Thanks in advance.

2

Why do you think you need a dynamic array? Just create an array big enough to hold all the data in your worst case scenario. It should still be well under the size of the memory available to you.

1 Like
3

To expand on @Ric’s sound advice, as long your data stays below 3K I’d even suggest to put the data in Backup RAM as retained int16_t buf[1024] (=2K leaving you with short 1K for other stuff) which will even survive a deep sleep or system reset as long the device doesn’t lose power (which can be prevented via a simple coin cell on VBAT).

I guess the reason why you “only” found questions about “big” data on the forum is due to the fact that little data never really is a matter of much head scrathing :wink:

BTW, as mentioned above, with such infrequent readings as long you won’t do a lot else in between, you may want to consider sending the device to sleep after its short task is done.

1 Like
4

@ScruffR and @Ric thank you both for the advice.

I have used the “retained” variable type (as a test so far, but I see how to apply it). N00b question:

I think I get how to define the array (retained pubarray buf[1024] - right? Assuming I want to call pubarray as my array name?)

2 follow-on questions. How do I “insert” the values into the array on each loop? This is where my “redim arrPub” background comes in - if I just needed to redefine it as what it was plus the new data that is a dynamic array to me, but from the advice it sounds like I can just append at the end the new values?

second ? - since these are not just int values, but also strings, how do I insert the timestamp? I can keep the string stuff pretty short by making some educated assumptions and applying some logic on read-out and not on the storing.

using my same code example:

// Check the value of the probe - if > 4000 probe not connected so skip reading it
if (SMVolts1 < 4000) {
    String voltage = "Voltage " + String(SMVolts1);
    String publishdata = timestamp + " " + voltage;
        if (Particle.connected()) {
          Particle.publish("S1_Reading_Data", publishdata, PRIVATE);
        }
        else {
          pubarray() = pubarray() + Timestamp + SMVolts1
//(  ^^^^  HERE IS WHERE I AM LOST - I know this isn't right, just trying to get down the right road)
        }
  }
  delay(1000);

Thanks in advance!

5

Actually no :wink:
buf would be the name and the place where you got pubarray would actually be the place for the datatype you want to store.
Since you said you wanted 4digit numbers I suggested int16_t which is a 16 bit signed integer (-32768 .. +32767)

And as this is a static array with 1024 "slots" you won't redim it but only fill the slots one by one like this

retained int16_t pubarray[1024];
retained int16_t globalIndex = 0;

void loop() { 
  int16_t newValue = getValue();

  if (newValue < 4000) {
    pubarray[globalIndex++] = newValue;
    if (globalIndex > 1023) globalIndex = 0; // when buffer is full overwrite old data
  }
}

If you need to store the timestamp too, you may use a struct instead of int16_t for the array, which will reduce the max number of retainable entries.

2 Likes
6

There are many options for how to store and send your data; the choice on how to do it depends a lot on how you're using that data on the receiving end. Are you sending it to your own server, a third party service (like Ubidots, ThingSpeak, etc), an app, or web page of your own making? One complication of using an array, is that you can't send it in a Particle.publish; you'll have to loop through the array, and publish each data point separately, while respecting the publish limit of 1 per second. Third party services also usually have a limit on how fast you can send the data (at least in their free versions). This is one reason I often store my data as a comma separated c-string, rather than an array (at least in cases where I'm sending it to my own app where I have control over how to parse that string).

2 Likes
7

Thank you so much for this tidbit.

Currently I am using Particle.publish (and have a 1-second delay in the code to accommodate the required API delay) because it is easy for testing. When I go forward my intention is to have my own “base station” unit that can parse the incoming string however I decide to do it, so in that case a simple string as you describe makes a ton more sense. I can put a simple CSV string together with a recognized (by the code) terminator for each reading and simply loop the string and break out the values for storage and action.

MUCH easier than messing about with an array and far less likely to break since I can use a retained variable type for the string, and reboot if needed to clean up fractured memory. (I have read elsewhere on the forum that using Strings over long timespans or “endless” loops can cause memory issues, but having a soft restart embeded in the code every x loops should fix that)

Thanks again for the guidance!

8

You can use that string to send data via a Particle Publish to services like Losant and Ubidots.

Here is an example dashboard I set up in Losant that I feed with a simple Particle Publish in a 220 byte strings.

I like Losant because I can send Particle Publish data separated by simple colons : : and then break it down on the backend of Losant.

image.png2496×1384 236 KB

3 Likes
9

It would be safer to use a c-string (char array) instead of a String object. You can use sprintf() to convert your data to a char array, and then strcat() to concatenate that new string to the end of the larger string that you're building.

1 Like
10

“Safer” but significantly steeper learning curve and difficulty for me to figure out how to actually implement it. I go cross-eyed reading about c-strings and it seems a very clunky way to do things. It sounds much easier how you explain it, but the devil’s in the code…

11

@charrold303, it’s not as steep as you think and if you give it a shot, the community can help you get there.

1 Like
12

@charrold303, you cannot use a String object as retained variable.
And using an array is neither more likely to break (actually the opposite) nor is preventing you from having the array retained.

Parsing a C string is definetly way more complicated than wrapping the numeric values stored in an array into a string.

1 Like
13

@charrold303 The C-String used to confuse me also but after playing around a bit I now see how simple they are to work with to format data before sending it in a Particle Publish.

Here is how I setup a C-String (Char array) in my code to send GPS Data & Battery Voltage to Losant.

First define your Char Array and change publishStateString to what ever you want it to be:

The 256 is the max byte size of the array which is the max size of a Particle Publish payload +1

Now I use the snprintf function to format the data that will be put into the char array before sending it out as the Particle Publish data payload.

snprintf(publishStateString, sizeof(publishStateString), "%.6f, %.6f:%.2f:%u:%.2f:%.2f:%.u:%.2f:",gps.location.lat(), gps.location.lng(), gps.altitude.feet(), gps.satellites.value(), (gps.hdop.value()/100.0), fuel.getSoC(), fuel.getVCell()  );
Particle.publish("GPS", publishStateString);

Pay attention to this first line of code below because it contains the commands that actually format the data:

"%.6f, %.6f:%.2f:%u:%.2f:%.2f:%.u:%.2f:"

,gps.location.lat(), gps.location.lng(), gps.altitude.feet(), gps.satellites.value(), (gps.hdop.value()/100.0), fuel.getSoC(), fuel.getVCell()  );

Now the %.6f is a snprintf formatting function for turning the data returned from gps.location.lat() into a float data point with 6 numbers after the decimial point.

So you can see we use the same %.6f for the gps.location.lng() data point also because it’s also a float data point with 6 digits after the decimal point.

For the next data point we use %.2f because gps.altitude.feet() the accuracy of the altitude only goes to 2 points past the decimal point and it’s a floating point number.

Now for the next data point we use %u because it’s not a float but a nubmer from 0-99 showing how many satellites are in view: gps.satellites.value()

And we just keep doing this as we go down the list of data variables we want to send.

Here is a reference for the different types of data formatting you can do using the snprintf function.

image.png1380×902 94.6 KB

http://www.cplusplus.com/reference/cstdio/printf/

When I send a Particle Publish the snprintf function formats the data payload like this:

 {"data"39.123456 , -85.123456:856.45:14:1.12:85:3.88:":","ttl":60,"published_at":"2017-07-10T20:28:13.934Z","coreid":"460042001051353338363333","name":"GPS"}

Then on the back end of Losant I can just pull data out between the colons and define what that data is so it can be databased and then made available to view in custom dashboards.

That may or may not make sense and I’m a newbie but after playing around with this and looking at what actually get’s Published will help you figure out this formatting function rather quickly.

The help is here if you need it.

6 Likes
14

from one newbie to another, thank you. This was easy to read and follow and gave me exactly what I needed to know. I will be trying it out this week and will let you know what I come up with.

1 Like
15

Glad I can help out after so many others have helped me out on here :slight_smile:

4 Likes
16

Thanks for the help!

17

First issue struck immediately - was just testing storing a string into a very short char array.

Here is the “greatly simplified” code:

retained char dev_name[12];    <- Pretty sure this is a character array of 12 characters (12 was random - could be more or less as needed)
sprintf(dev_name, sizeof(dev_name), "%.s", "Tomato_1"); <- if I read your post right, this should write "Tomato_1" as a string into the aforementioned character array

instead i get a compiler error of: “invalid conversion from ‘unsigned int’ to ‘const char*’ [-fpermissive]”

Is there nothing approximating:

retained char dev_name[12] = “Tomato_1” ?

I have seen some really (what I hope at least is) archaic string array things where I would have to literally walk down each char spot in the array and store the letter in that spot?

18

@charrold303, you were SO close! The command you wanted is snprintf() not sprintf() - note the missing “n”. :wink:

2 Likes
19

ALWAYS THE TYPOS! :stuck_out_tongue:

Thanks @peekay123

1 Like
20

If you want to copy a string literal into a char[] you’d use either

  strcpy(dev_name, "Tomato_1"); 
  // or
  strncpy(dev_name, "Tomato_1", sizeof(dev_name));
  // or less commonly since you are not actually formatting the string but just setting it
  snprintf(dev_name, sizeof(dev_name), "Tomato_1");

I should keep it shorter, then I wouldn’t be beaten by @peekay123 :wink:

3 Likes