Encoding JSON in firmware

Social Project Share
1

JSON is something relatively new to me. I have found the site http://www.jsoneditoronline.org/ to be VERY useful as I design JSON files and as I write code to encode JSON data.

On the encoding side, I followed some sample code to get started. That code worked, but it used char[] logic, and it seemed overly complex to me.

As an experiment, I used 100% “string logic” in the following function that maintains a Particle.variable named dStateJSON. The function works, and the code seems simpler to me. I’m curious to get other people’s thoughts, and to know if there might be an even simpler way to encode JSON.


// ----------------------------------------------------------------------------------------------------------------

void update_dStateJSON() {
	
    bool firstDoor = TRUE;
	String jsonString = "{\"gnode\": {\"version\": \"1.0\", \"message\": \"" + gnodeMessage + "\", \"doors\": [ ";
    for (int i = 0; i < NUMDOORS; i++) {
        if (! door[i].state == UNKNOWN) {
            if (firstDoor) {
                firstDoor = FALSE;              
            } else { 
                jsonString = jsonString + ",";   // each array element after the first must start with a comma
            }
    		jsonString = jsonString  // construct a json string for the door object
    		    + "{"
    			    + "\"id\":" + door[i].id + ","
    			    + "\"state\":\"" + doorStateText[door[i].state] + "\","
    			    + "\"statetime\":\"" + String(door[i].stateTime) + "\","
    			    + "\"message\":\"" + door[i].actionMessage + "\","
    			    + "\"button\":\"" + doorNextAction[door[i].state] + "\""
    		    + "}";
        }
	}
	jsonString = jsonString + "]}}";
	dStateJSON = jsonString;            // dStateJSON is the Particle.variable
	return;
  }
    
// ----------------------------------------------------------------------------------------------------------------

In HTML, I use the jquery command “$.getJSON” to obtain the JSON produced by the above function. A prettified version of of the command’s output follows:

{
  "gnode": {
    "version": "1.0",
    "message": "",
    "doors": [
      {
        "id": 1,
        "state": "OPEN",
        "statetime": "2017-05-24T00:10:55Z",
        "message": "",
        "button": "Close Door"
      },
      {
        "id": 2,
        "state": "CLOSED",
        "statetime": "2017-05-24T00:10:55Z",
        "message": "",
        "button": "Open Door"
      },
      {
        "id": 3,
        "state": "CLOSED",
        "statetime": "2017-05-24T00:10:55Z",
        "message": "",
        "button": "Open Door"
      },
      {
        "id": 4,
        "state": "CLOSED",
        "statetime": "2017-05-24T00:10:55Z",
        "message": "",
        "button": "Open Door"
      }
    ]
  }
}
2

Hi @Bear

I edited your post to use the tags for embedding C/C++ code and other literal things like your JSON.

To do this yourself next time, use this for code:

 ```cpp
   Code goes here

 
You can leave off the cpp part for HTML and other things but there are lots and lots of supported languages.

**As to your question**

I find the String type to be not so good on embedded platforms like this--it can cause memory fragmentation.  If you search around you will several examples of creating JSON's using the more old-fashioned but less memory intensive C style strings and `snprintf`.
4 Likes
3

I assume the issue you raise relates to the ever growing string variable: jsonString. Each time jsonString is extended, the Photon needs to allocate space for the new string. By using a fixed character buffer memory only needs to be allocated once … is that correct?

1 Like
4

Yes, that is correct. All of the String operator “+” overloads that have non-constant inputs also generate lots of temporary String objects that can lead the heap fragmentation, so it is not just the main jsonString that can cause problems.

You can use the String object with a more static allocation strategy but not with the syntax you are using. You would need to preallocate the String as large as you need it in the constructor to a global or static var, and then use the concatenate operator in a different way.

I find it much easier to just use snprintf with a global or static char array instead.

1 Like
5

I am not thrilled with any of the options so far. Is there a library with JSON encoding functions that we could call?

6

After 30 days of testing my firmware, The experimental string logic for encoding JSON that I attached at the beginning of this thread worked flawlessly. That said, this code is clearly not the best solution for IOT platforms with limited resources. I therefore decided to take a close look at the “SparkJson” library … which is a port of the ArduinoJson library documented at the following link:
https://bblanchon.github.io/ArduinoJson/doc/index.html

After adding the SparkJson library to my firmware, I developed the following code to replace the aforementioned string logic. If we discount for the learning curve, this code was simpler to write and should be simpler to update and maintain going forward.

void update_dStateJSON() {
    StaticJsonBuffer<1024> jsonBuffer;
    JsonObject& root = jsonBuffer.createObject();
    JsonObject& gnode = root.createNestedObject("gnode");
    gnode["version"] = "1.0";
    gnode["message"] = gnodeMessage.c_str();
    JsonArray& doors = gnode.createNestedArray("doors");
    for (int n = 0; n < NUMDOORS; n++) {
        if (! door[n].state == UNKNOWN) {
            JsonObject& jsonDoor = doors.createNestedObject();
            jsonDoor["id"] = n + 1;
            jsonDoor["state"] = doorStateText[door[n].state].c_str();
            jsonDoor["statetime"] = door[n].lastSensorStateTime.c_str();
            jsonDoor["message"] = door[n].actionMessage.c_str();
            jsonDoor["button"] = doorNextAction[door[n].state].c_str();
        }
    }
    char gnodeBuffer[622];
    root.printTo(gnodeBuffer,sizeof(gnodeBuffer));
    dStateJSON = String(gnodeBuffer); 
}

One departure from the Arduino documentation. I had to add “.c_str()” to the name of each string variable that I loaded to the JSON object or array. Without this addition, the compiler did not know what type of object that I was loading.

This approach still references quite a few strings, and some of these are duplicated in the StaticJsonBuffer. I ask the experts: “Is this an acceptable solution for IOT devices with limited resources?” and … “Is there a better way to use the SparkJson library for this encoding?”

7

The most popular and resource saving option would be the one already "dismissed": snprintf() and avoid using String in your own code completely.

When you ask for best practice, it doesn't have to be pleasing - it's enough when it is factually true :wink:

2 Likes
8

This is what my code looks like after I converted all of the JSON related variables from data type String to char[]. It turns out that SparkJson expected char[] variables … which is why I had to append “.c_str()” to the String variable names in my previous code. Now, the library appears to function as intended.

My firmware only contains two variables of data type “String” now. These are particle variables. I believe these MUST be data type “String”. Please advise if that is incorrect.

void update_dStateJSON() {
    StaticJsonBuffer<1024> jsonBuffer;
    JsonObject& root = jsonBuffer.createObject();
    JsonObject& gnode = root.createNestedObject("gnode");
    gnode["version"] = "1.0";
    gnode["message"] = gnodeMessage;
    JsonArray& doors = gnode.createNestedArray("doors");
    for (int n = 0; n < NUMDOORS; n++) {
        if (! door[n].state == UNKNOWN) {
            JsonObject& jsonDoor = doors.createNestedObject();
            jsonDoor["id"] = n + 1;
            jsonDoor["state"] = doorStateText[door[n].state];
            jsonDoor["statetime"] = door[n].stateTime;
            jsonDoor["message"] = door[n].actionMessage;
            jsonDoor["button"] = doorNextAction[door[n].state];
        }
    }
    char gnodeBuffer[622];
    root.printTo(gnodeBuffer,sizeof(gnodeBuffer));
    dStateJSON = String(gnodeBuffer); 
}
9

It might be true that your project only features two String objects now, but StaticJsonBuffer<> also makes use of dynamic memory allocation and hence will be contributing to heap fragmentation just the same.

BTW, this statement will also creates a temporary String object

   dStateJSON = String(gnodeBuffer);

And Particle.variable() can just as well expose a char[] string to to cloud, so no String object required for that either.

2 Likes
10

I started programming in ‘c’ eons ago. The fact that some of my early programs included some ‘b’ code might give you a perspective on how long ago it was. As time went on, I programmed primarily in other languages, and ‘c’ evolved to c++. I’ve found writing in c++ to be more difficult than I expected because so much is familiar … while so much has changed.

Last time out, I converted all String variables to char[]. After reading an article about how c++ handles char[] variables, I concluded that I should declare program literals as const char * because that keeps them off of the stack. For example, this is how I am declaring literals that describe a door’s present state (which my firmware stores as an integer (0-5) … to index this array)

const char *DOORSTATETEXT[] = {"Unknown","Closed","Closing","Opening","Open","Stopped"};

I really like the const char * syntax for a variety of reasons. Am I correct in thinking that this is the optimal way to handle program literals.

I continue to declare dynamic variables as char[]. For example, I when a door state changes, I need to capture a time stamp … so I declare a variable and populate it as shown below.

struct Door {
    int state;                      /
char stateTime[21];
} door[NUMDOORS];

strncpy(door[n].stateTime, Time.format(TIME_FORMAT_ISO8601_FULL), sizeof(door[n].stateTime));

Any comments or suggestions?

The Particle.variable “dStateJSON” is now defined as char[], and I update it directly via the SparkJson printTo command. This is working fine and it allowed me to eliminate the temporary string that you mentioned. The reference page for the Particle.variable does not mention that char[] is valid, but I now see where there is an char * example there.

I did find another String in my firmware. It is an input parameter to a Particle.function. The Particle documentation clearly states this must be a String data type. Is that correct?

I have not looked at the StaticJsonBuffer issue yet. I’ll need to try to figure out how this buffer is being used.

This is what the current "update dStateJSON function looks like:

void update_dStateJSON() {
    StaticJsonBuffer<1024> jsonBuffer;
    JsonObject& root = jsonBuffer.createObject();
    JsonObject& gnode = root.createNestedObject("gnode");
    gnode["version"] = "1.0";
    gnode["message"] = gnodeMessage;
    JsonArray& doors = gnode.createNestedArray("doors");
    for (int n = 0; n < NUMDOORS; n++) {
        if (! door[n].state == UNKNOWN) {
            JsonObject& jsonDoor = doors.createNestedObject();
            jsonDoor["id"] = n + 1;
            jsonDoor["state"] = DOORSTATETEXT[door[n].state];
            jsonDoor["statetime"] = door[n].stateTime;
            jsonDoor["message"] = ACTIONMESSAGETEXT[door[n].actionMessage];
            jsonDoor["button"] = DOORNEXTACTION[door[n].state];
        }
    }
     root.printTo(dStateJSON,sizeof(dStateJSON));
}
11

I'd rather store the int which represents the timestamp and create the string on the fly whenever needed - and while we are at byte counting your state could be an int8_t (but due to 32bit architecture not a lot to win there ;-))

Yes, unfortunately IMHO.
As well as other functions take/return String objects to my concern.

But kicking String out of your own code and circumventing functions that use it wherever possible is the best we can do.
e.g. using strftime() instead of Time.format()