Long JSON Strings

Is there an easy way to parse long JSON strings? Many of the pages I am using have over 5K characters which are read into the Photon in many chunks from the webhook. Once I have the complete string, it looks like I can easily use SparkJSON to parse the string, but is building a string of over 5K characters feasible, or is there a better method?

Do you actually need all of the response?
The Particle Cloud has the ability to extract the desired fields and only pass on what you need.
Have a look at custom response template

2 Likes

Cool. Thanks! I have used simple response templates like the following.

"responseTemplate": "{{#data}}{{attributes.arrival_time}}{{/data}}"

I assume the following example from the documents is similar, but how do I access “lat” and “lng” in my code?

"responseTemplate": {
  "lat": "{{{results.0.location.lat}}}",
  "lng": "{{{results.0.location.lng}}}"
}

Once you get the reduced JSON response onto your device you can use any JSON parser library you like (like this) or you just parse the string yourself.

You can also use @rickkas7’s great mustache tester to develop your response template. It has saved me numerous hours of testing!

I am okay with using JSON parsers, I just don’t know how to access the specific sub-strings using the variables “lat” and “lng” as suggested in the example from the document. When I subscribe to a webhook, I have only received a single string, but the example (included below) implies that you can get two or more separate sub-strings identified by “lat” and “lng” in the example below. Just don’t know how to reference these sub-strings in code. Sorry if this is dumb question.

Also, thanks for the link to the mustache tester.

@stk, here is an example using openweatherapi that I use for my weather station. A sample call gets a 7-day forecast:
https://samples.openweathermap.org/data/2.5/forecast/daily?id=524901&appid=b1b15e88fa797225412429c1c50c122a1

A webhook based on this API would produce this raw JSON data:

{"cod":"200","message":0,"city":{"geoname_id":524901,"name":"Moscow","lat":55.7522,"lon":37.6156,"country":"RU","iso2":"RU","type":"city","population":0},"cnt":7,"list":[{"dt":1485766800,"temp":{"day":262.65,"min":261.41,"max":262.65,"night":261.41,"eve":262.65,"morn":262.65},"pressure":1024.53,"humidity":76,"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"speed":4.57,"deg":225,"clouds":0,"snow":0.01},{"dt":1485853200,"temp":{"day":262.31,"min":260.98,"max":265.44,"night":265.44,"eve":264.18,"morn":261.46},"pressure":1018.1,"humidity":91,"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13d"}],"speed":4.1,"deg":249,"clouds":88,"snow":1.44},{"dt":1485939600,"temp":{"day":270.27,"min":266.9,"max":270.59,"night":268.06,"eve":269.66,"morn":266.9},"pressure":1010.85,"humidity":92,"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13d"}],"speed":4.53,"deg":298,"clouds":64,"snow":0.92},{"dt":1486026000,"temp":{"day":263.46,"min":255.19,"max":264.02,"night":255.59,"eve":259.68,"morn":263.38},"pressure":1019.32,"humidity":84,"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"speed":3.06,"deg":344,"clouds":0},{"dt":1486112400,"temp":{"day":265.69,"min":256.55,"max":266,"night":256.55,"eve":260.09,"morn":266},"pressure":1012.2,"humidity":0,"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13d"}],"speed":7.35,"deg":24,"clouds":45,"snow":0.21},{"dt":1486198800,"temp":{"day":259.95,"min":254.73,"max":259.95,"night":257.13,"eve":254.73,"morn":257.02},"pressure":1029.5,"humidity":0,"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"speed":2.6,"deg":331,"clouds":29},{"dt":1486285200,"temp":{"day":263.13,"min":259.11,"max":263.13,"night":262.01,"eve":261.32,"morn":259.11},"pressure":1023.21,"humidity":0,"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13d"}],"speed":5.33,"deg":234,"clouds":46,"snow":0.04}]}

For my application, I only need the values for dt, temp.day, temp.min, humidity and the weather icon text. Using @rickkas7’s mustache tester, I developed the following mustache filter for the Response Template:

{{#list}}{{dt}}~{{temp.day}}~{{temp.min}}~{{humidity}}{{#weather}}~{{icon}}{{/weather}}/{{/list}}

Now, using this template, the webhook returns:

1485766800~262.65~261.41~76~01d/1485853200~262.31~260.98~91~13d/1485939600~270.27~266.9~92~13d/1486026000~263.46~255.19~84~01d/1486112400~265.69~256.55~0~13d/1486198800~259.95~254.73~0~01d/1486285200~263.13~259.11~0~13d/

The tildes (~) are used to separate values and the slashes (/) are used to separate days. These were specifically included in the mustache filter. I can now parse this much smaller string using strtok() along with atoi(), atof() and strncpy() to get all the data into a daily data structure which I then use to display on an epaper display.

1 Like

I have definitely used and parsed response templates like those described above. My basic question is, if the following is my response template, what does my firmware look like?

"responseTemplate": {
  "lat": "{{{results.0.location.lat}}}",
  "lng": "{{{results.0.location.lng}}}"
}

The library I linked you to features an example that illustrates the use

Instead of "t1" you should be able use "lat" or "lng"

Thanks to all your help I am very close. The following code correctly extracts arrive and id and publishes both once using the static json string in setup. Yet using similar code in loop, I only get null values for both arrive and id. Importantly, the string published by “MBTA2”

"{ \"arr\": \"2018-11-08T15:10:49-05:00\", \"id\": \"38428527\" }"

is character for character IDENTICAL to the static json string in setup. Any suggestions?

#include <ArduinoJson.h>

void getBusTimes() {
    Particle.publish("MBTA2");
}

Timer timer(5000, getBusTimes);

void setup() {
    timer.start();
    Particle.subscribe("hook-response/MBTA2", condHandler, MY_DEVICES);

    StaticJsonBuffer<200> jsonBuffer;

    char json[] =  "{ \"arr\": \"2018-11-08T15:10:49-05:00\", \"id\": \"38428527\" }";

    JsonObject& root = jsonBuffer.parseObject(json); 

    const char* arrive = root["arr"];
    const char* id = root["id"];

    Particle.publish("arrive", arrive);
    Particle.publish("time", id);
}

void condHandler(const char *event, const char *data) {
    StaticJsonBuffer<200> jsonBuffer;
    char copy[strlen(data)+1]; 
    strcpy(copy, data); 
    JsonObject& root = jsonBuffer.parseObject(copy); 

    const char* arrive = root["arr"];
    const char* id = root["id"];

    Particle.publish("arrive", arrive);
    Particle.publish("time", id);
}
void loop() {
  
}

I have included the response template and URL below.

{ \"arr\": \"{{{data.0.attributes.arrival_time}}}\", \"id\": \"{{{data.0.relationships.trip.data.id}}}\" }

https://api-v3.mbta.com/predictions?filter[route]=134&filter[stop]=9147

Can you Serial.println(data) and post here to see what you actually get as response?

Below is the result of Serial.println(data) and Serial.println(root) which is identical to the string produced by Particle.publish("MBTA2") as mentioned above and identical to the static json string in setup.

{ \"arr\": \"2018-11-09T07:25:49-05:00\", \"id\": \"38442668\" }

Kinda stumped. If json and copy are identical, how can the same code that acts on json and copy produce different results? Not sure what I am missing.

I would have tried your code, but for some reason I can’t import ArduinoJson library in Web IDE, but I’d usually rather go with the previously linked JsonParserGeneratorRK anyway.

Still stuck. Just to simplify, I am only using serial print as suggested (see code below). Farther below, I wrote the same program using JsonPaserGeneratorRK with the same results.

Below is the serial output. The static version correctly produces the parsed json arrival time. The dynamic version does not. Again the static and dynamic json strings are identical.

dynamic
{   \"arr\": \"2018-11-09T15:10:49-05:00\",   \"id\": \"38428527\" }

static
2018-11-08T15:10:49-05:00
#include <ArduinoJson.h>

void getBusTimes() {
    Particle.publish("MBTA2");
}

Timer timer(5000, getBusTimes);

void setup() {
    timer.start();
    Particle.subscribe("hook-response/MBTA2", condHandler, MY_DEVICES);
    Serial.begin(9600);
}

void condHandler(const char *event, const char *data) {
    StaticJsonBuffer<200> jsonBuffer;
    char copy[strlen(data)+1]; 
    strcpy(copy, data); 
    
    StaticJsonBuffer<200> jsonBuffer1;
    char json1[] =  "{ \"arr\": \"2018-11-08T15:10:49-05:00\", \"id\": \"38428527\" }";
  
    JsonObject& root = jsonBuffer.parseObject(copy); 
    const char* arrive = root["arr"];
    const char* id = root["id"];
    Serial.println("dynamic");
    Serial.println(copy);
    Serial.println(arrive);
    
    JsonObject& root1 = jsonBuffer1.parseObject(json1); 
    const char* arrive1 = root1["arr"];
    const char* id1 = root1["id"];
    Serial.println("static");
    Serial.println(arrive1);
}

In this version I am using JsonParserGeneratorRK. This produced the following serial output, and again did not print the parsed Json.

{   \"arr\": \"2018-11-09T16:00:49-05:00\",   \"id\": \"38428529\" }
{   \"arr\": \"2018-11-09T16:00:49-05:00\",   \"id\": \"38428529\" }
#include "Particle.h"
#include "JsonParserGeneratorRK.h"

JsonParserStatic<256, 20> jsonParser;

void getBusTimes() {
    Particle.publish("MBTA2");
}

Timer timer(5000, getBusTimes);

void setup() {
    timer.start();
    Particle.subscribe("hook-response/MBTA2", condHandler, MY_DEVICES);
    Serial.begin(9600);
}

void condHandler(const char *event, const char *data) {
    String strValue;
    int responseIndex = 0;

	const char *slashOffset = strchr(event, '/');
	if (slashOffset) {
		responseIndex = atoi(slashOffset + 1);
	}

	if (responseIndex == 0) {
		jsonParser.clear();
	}
	jsonParser.addString(data);
	
	Serial.println(data);
	
    if (jsonParser.parse()) {
	
		if (jsonParser.getOuterValueByKey("arr", strValue)) {
            Serial.printlnf("arrive", strValue.c_str());
        }
    }	
}

I now managed to test ArduinoJson by buiding via CLI and I can see the expected results.

I just slightly changed the code to incorporate both fields into on publish and cut out the webhook itself.

#include <ArduinoJson.h>

    char txt[64];

void getBusTimes() {
    Particle.publish("hook-response/MBTA2", "{ \"arr\": \"2018-11-09T07:25:49-05:00\", \"id\": \"38442668\" }", PRIVATE);
}

Timer timer(5000, getBusTimes);

void setup() {
    timer.start();
    Particle.subscribe("hook-response/MBTA2", condHandler, MY_DEVICES);

    StaticJsonBuffer<200> jsonBuffer;

    char json[] =  "{ \"arr\": \"2018-11-08T15:10:49-05:00\", \"id\": \"38428527\" }";

    JsonObject& root = jsonBuffer.parseObject(json); 
    const char* arrive = root["arr"];
    const char* id = root["id"];

    snprintf(txt, sizeof(txt), "arrive %s; time %s", arrive, id);
    Particle.publish("reply", txt, PRIVATE);
}

void condHandler(const char *event, const char *data) {
    StaticJsonBuffer<200> jsonBuffer;
    char copy[strlen(data)+1]; 

    strcpy(copy, data); 
    JsonObject& root = jsonBuffer.parseObject(copy); 

    const char* arrive = root["arr"];
    const char* id = root["id"];

    snprintf(txt, sizeof(txt), "arrive %s; time %s", arrive, id);
    Particle.publish("reply", txt, PRIVATE);
}

void loop() {
  
}

That’s the event I get every 5 seconds

arrive 2018-11-09T07:25:49-05:00; time 38442668

With a slightly altered version using JsonParserGeneratorRK (see below) I get this

>{ "arr": "2018-11-09T07:25:49-05:00", "id": "38442668" }<
arrive 2018-11-09T07:25:49-05:00
id 38442668
>{ "arr": "2018-11-09T07:25:49-05:00", "id": "38442668" }<
arrive 2018-11-09T07:25:49-05:00
id 38442668
#include <JsonParserGeneratorRK.h>

JsonParserStatic<256, 20> jsonParser;

void getBusTimes() {
    Particle.publish("hook-response/MBTA2", "{ \"arr\": \"2018-11-09T07:25:49-05:00\", \"id\": \"38442668\" }", PRIVATE);
}

Timer timer(5000, getBusTimes);

void setup() {
    timer.start();
    Particle.subscribe("hook-response/MBTA2", condHandler, MY_DEVICES);
    Serial.begin(9600);
}

void condHandler(const char *event, const char *data) {
    String strValue;
    int  responseIndex = 0;

	jsonParser.clear();
	jsonParser.addString(data);
	
	Serial.printlnf(">%s<", data);
	
    if (jsonParser.parse()) {
		if (jsonParser.getOuterValueByKey("arr", strValue)) {
            Serial.printlnf("arrive %s", (const char*)strValue);
        }
		if (jsonParser.getOuterValueByKey("id", strValue)) {
            Serial.printlnf("id %s", (const char*)strValue);
        }
    }	
}

So the parsing works either way. This would suggest the difference is in the data you try to parse.

1 Like

Okay now I am really confused. I tried your example, but it only returns static results i.e. the results contained in the string you send as part of Particle.publish. I am trying to get and parse dynamic results from the MBTA API.

I have had no problem parsing a static json string identical to the strings I receive from the MBTA API. I have had no problem getting a dynamic results string from the MBTA API and publishing this string from my response handler. The problem I have is parsing this dynamic string in my results handler, even though it is identical to the static string.

That's because I have no webhook that would give me the dynamic result. But that shows it's not the location where the parsing happens (as you anticipated in one of your previous posts), but must be due to differences in the data you get from your webhook compared to the static version.

It may appear to be the same, but since the result is different, it probably isn't.

One suspicion is that you get the escaped string back (aka "actually containing the ") which looks the same as the escaped way of writing a string in C, but the backslash \ actually never is part of the string.

If you look at the Serial.print() of the static string, you won't see the backslashes, but in your post above they are there (but shouldn't).
Your custom JSON template should not inject the backslashes.
If you look at the sample template I linked earlier, you'll see they are not there either.
https://docs.particle.io/reference/device-cloud/webhooks/#responsetemplate

Yes. I got it. Thanks so much! One of the problems I was having is that the response template in the webhook builder should be entered as a non-escaped string and the response template in the custom template should be entered as an escaped string. It works perfectly now.

1 Like