ArduinoJson buffer limits?

Troubleshooting
1

The following correctly requests and parses a json string from a local server, but only twice before we get a “parse fail”. If I manually increase the size of the StaticJsonBuffer (I tried up to 10000) it works for more iterations and then fails. Any suggestions? I tried to use DynamicJsonBuffer jsonBuffer(bufferSize); but this would not compile.

#include <SparkJson.h>
#include <ArduinoJson.h>
#include "application.h"
#include "HttpClient.h"

//Sample of string
///const char[] = {"data": [{"time": 0, "headsign": "Middlesex Ave & Second St", "id": "38428705"}, {"time": 25, "headsign": "Wellington", "id": "38428527"}, {"time": 61, "headsign": "Wellington via Veterans Senior Center", "id": "38428504"}]}

const size_t bufferSize = JSON_ARRAY_SIZE(6) + JSON_OBJECT_SIZE(1) + 6*JSON_OBJECT_SIZE(3) + 390;
StaticJsonBuffer<bufferSize> jsonBuffer;
HttpClient http;
http_request_t request;
http_response_t response;

// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers[] = {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    { "Accept" , "*/*"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

void tickTock() {
   //Create path for http address
    request.hostname = "10.1.41.87";
    request.port = 5000;
    request.path = "/";
  
    //Get http response and print status, length and response
    http.get(request, response, headers);
    Serial.println(response.status, DEC);
    Serial.println(response.body.length(), DEC);
    Serial.println(response.body);
    
    //Convert json String to char*
    int j_length = response.body.length()+1;
    char json[j_length];
    response.body.toCharArray(json, j_length);
    
    //Create json object
    JsonObject& root = jsonBuffer.parseObject(json);
    if (!root.success()) 
        Serial.println("Parse fail");
    else
        Serial.println("Parse success");
    
    const char* direction = root["data"][0]["headsign"];
    int arrival = root["data"][0]["time"];
    Serial.println(direction);
    Serial.println(arrival, DEC);
}

Timer timer(5000, tickTock);
 
void setup() {
    timer.start();
    Serial.begin(9600);
}

void loop() {
}
2

I found ArduinoJson to be unreliable for large JSON data. It keeps too many copies of the data and uses too many heap-based options for use in a limited memory environment, in my opinion.

I wrote JsonParserGeneratorRK to solve this issue. The API is quite different from ArduinoJson/SparkJson, but it’s really memory-efficient.

3 Likes
3

Cool. Will it support more than 256 bytes of data? I could probably work around this, but just wanted to check.

4

Yes, it can even handle the multiple chunk response from a webhook, including super-sized 622 byte chunks in 0.8.0.

1 Like
5

Does it need to have the JSON in a specific format? The JSON (shown below) that parsed with ArduinoJson will not parse with RK. Below is the code that worked with ArduinoJson now modified for RK. It did work with the test2 string.

{"data": [{"headsign": "Wellington", "id": "38428537", "time": 9}, {"headsign": "Wellington", "id": "38428498", "time": 22}, {"headsign": "Wellington", "id": "38428539", "time": 45}, {"headsign": "Wellington", "id": "38428510", "time": 82}]}

// This #include statement was automatically added by the Particle IDE.
#include <HttpClient.h>

// This #include statement was automatically added by the Particle IDE.
#include <JsonParserGeneratorRK.h>

//#include <ArduinoJson.h>
#include "JsonParserGeneratorRK.h"
#include "application.h"
#include "HttpClient.h"

//const char * const test2 = "{\"t1\":\"abc\",\"t2\":1234,\"t3\":1234.5,\"t4\":true,\"t5\":false,\"t6\":null, \"t7\" : \"\\\"quoted\\\"\" } ";

// Global parser that supports up to 256 bytes of data and 20 tokens
JsonParserStatic<1000, 20> parser1;

//StaticJsonBuffer<bufferSize> jsonBuffer;

HttpClient http;
http_request_t request;
http_response_t response;

// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers[] = {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    { "Accept" , "*/*"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

void tickTock() {
   //Create path for http address
    request.hostname = "10.1.41.87";
    request.port = 5000;
    request.path = "/";
  
    //Get http response and print status, length and response
    http.get(request, response, headers);
    Serial.println(response.status, DEC);
    Serial.println(response.body.length(), DEC);
    Serial.println(response.body);
    
   // StaticJsonBuffer<bufferSize> jsonBuffer;
    
    //Convert json String to char*
    int j_length = response.body.length()+1;
    char json[j_length];
    response.body.toCharArray(json, j_length);
    
    parser1.clear();
	parser1.addString(json);
//    parser1.addString(test2);
	if (!parser1.parse()) {
		Serial.println("parse fail");
	} 
	else {
	    Serial.println("parse pass");
	}
}

// create a software timer to get new prediction times every minute
Timer timer(5000, tickTock);
 
void setup() {
    timer.start();
    Serial.begin(9600);
}

void loop() {
}
6

You’re not null-terminating the copy of the string in json so there will be garbage characters at the end. You’d need to:

json[j_length - 1] = 0;

However, you should not make a copy of it anyway, because the HttpClient response.body is a String. Just do:

parser1.addString(response.body.c_str());
4 Likes
7

Got it. I assumed that the parser would balk on a const char*.

Maybe I am being dense, but could you provide an example for extracting, say the third [index 2] headsign value from from the string shown below?

When I ran the following with String keyValue = "headsign"; strValue contained the entire Json string. With an index of 1 it returned null.

parser1.getOuterKeyValueByIndex(0, keyValue, strValue);

{"data": [{"headsign": "Wellington", "id": "38428510", "time": 4}, {"headsign": "Wellington", "id": "38428517", "time": 13}, {"headsign": "Wellington", "id": "38428488", "time": 42}]}

Thanks!

8

The best way is to use this: http://rickkas7.github.io/jsonparser/

Paste in the JSON and click on the row you want, it will generate the code for you:

parser.getReference().key("data").index(2).key("headsign").valueString()
4 Likes
9

Great. That works perfectly. Would you mind also provided an example of how to retrieve the number of objects in a Json array? i.e. the data array shown above.

Thanks so much!

10

how to retrieve the number of objects in a Json array? i.e. the data array shown above.

parser.getReference().key("data").size()
1 Like
11

I did something similar a while back. This loads a JSON string and then loops through the JSON elements. I was sending JSON messages to a Photon via UDP multicast. It’s probably not the most efficient method of looping through elements but it’s a working example.

//char msg_cp[255] = "";
JsonParserStatic<2000, 250> jparser; //2000 is oversized to accomodate a minium Network MTU of 1500 with a token every ~8 bytes.

void ParseNew(char *NewCmd) 
{
    //bool error = false;
    jparser.clear();
	jparser.addString(NewCmd);
	if (!jparser.parse()) {
		Log.error("JSON parse failed. Check your format.");
        Log.error("Cmd Txt: %s", NewCmd);
        return;
	}
	
	for (int i=0; i<32; i++) {
    	const JsonParserGeneratorRK::jsmntok_t *targetKeyTok;
    	const JsonParserGeneratorRK::jsmntok_t *targetValueTok;
    	
    	String targetKeyName;
    	size_t targetKeyLen;
    	
	    if (jparser.getKeyValueTokenByIndex(jparser.getOuterObject(), targetKeyTok, targetValueTok, i)) {
	        if (jparser.getTokenValue(targetKeyTok, targetKeyName)) {
	             Log.info("(%i) New cmd targets device/group: %s", i, targetKeyName.c_str());
	            
	            
	        } else {
	            //Log.error("Cannot parse target object %i or end of objects.", i);
	            //error = true;
	            break;
	        }
	    }
    } 
} //ParseNew
12

Great. Thanks!