JSON Parsing Help

smd75jr · March 29, 2016, 11:59pm

Hi,

I am trying to parse a JSON string which is returned from a webhook from AWS DynamoDB/Lambda/API Gateway.

I am trying to use the “SparkJson” library but can’t figure out how to make it work.

My JSON data looks like this:

{"Data":{"batV":{"S":"4.00"},"deviceId":{"S":"540056000a51343334363138"},"lat":{"S":"41.2083"},"batSoC":{"S":"88"},"uvI":{"S":"0.0"},"wDir":{"S":"0"},"wSpd":{"S":"0.0"},"wGst":{"S":"0.0"},"hum":{"S":"29.0"},"temp":{"S":"21.3"},"tm":{"N":"1459294350"},"lon":{"S":"-73.3440"},"pres":{"S":"970.7"}}}

And the code I am trying to use to split each element into it’s own variable is:

void parseWeather(const char *name, const char *data) { // Parse returned weather data
    
    StaticJsonBuffer<200> jsonBuffer; // Initialize JSON buffer
    
    char *weatherChar = const_cast<char*>(data); // Type cast returned weather data so the JSON parseing library can handle it
    
    Serial.println("char *weatherChar: " + String(weatherChar));
    
    JsonObject& weather = jsonBuffer.parseObject(weatherChar); // Parse the returned and type cast JSON data
    
    if (!weather.success()) { // Check if parse succeded
        Serial.println("parseObject() failed");
        return;
    }
    
    int weatherTm = weather["Data"]["tm"]["N"]; // Am I doing it right? I don't think so
    
    
} // End of function parseWeather()

I want to be able to have each element end up in its own variable or (preferably) in an object that can be called like:

weather.temp
weather.batV
weather.tm
etc...

Any help would be greatly appreciated.

Thanks,
smd75jr

rickkas7 · March 30, 2016, 1:20am

I was able to get it to work with this test code:

#include "Particle.h"
#include "SparkJson/SparkJson.h"

const char *rawJsonData = "{\"Data\":{\"batV\":{\"S\":\"4.00\"},\"deviceId\":{\"S\":\"540056000a51343334363138\"},\"lat\":{\"S\":\"41.2083\"},"
		"\"batSoC\":{\"S\":\"88\"},\"uvI\":{\"S\":\"0.0\"},\"wDir\":{\"S\":\"0\"},\"wSpd\":{\"S\":\"0.0\"},\"wGst\":{\"S\":\"0.0\"},"
		"\"hum\":{\"S\":\"29.0\"},\"temp\":{\"S\":\"21.3\"},\"tm\":{\"N\":\"1459294350\"},\"lon\":{\"S\":\"-73.3440\"},\"pres\":{\"S\":\"970.7\"}}}";


void parseWeather(char *data); // forward declaration

void setup() {
	Serial.begin(9600);

	delay(5000);
	char *dataCopy = strdup(rawJsonData);
	parseWeather(dataCopy);
	free(dataCopy);

}

void loop() {

}

void parseWeather(char *mutableData) {
	StaticJsonBuffer<1000> jsonBuffer;
	JsonObject& root = jsonBuffer.parseObject(mutableData);

	if (!root.success()) {
		Serial.println("parseObject() failed");
	    return;
	}

	Serial.printlnf("batV=%s", root["Data"]["batV"]["S"].asString());
	Serial.printlnf("deviceId=%s", root["Data"]["deviceId"]["S"].asString());
	Serial.printlnf("lat=%s", root["Data"]["lat"]["S"].asString());
	Serial.printlnf("lon=%s", root["Data"]["lon"]["S"].asString());
	Serial.printlnf("batSoC=%s", root["Data"]["batSoC"]["S"].asString());
	Serial.printlnf("uvI=%s", root["Data"]["uvI"]["S"].asString());
	Serial.printlnf("wDir=%s", root["Data"]["wDir"]["S"].asString());
	// ...
	Serial.printlnf("hum=%s", root["Data"]["hum"]["S"].asString());
	Serial.printlnf("temp=%s", root["Data"]["temp"]["S"].asString());
	Serial.printlnf("pres=%s", root["Data"]["pres"]["S"].asString());

	Serial.printlnf("tm=%s", root["Data"]["tm"]["N"].asString());

	// All of the values in the JSON are strings, so you need to convert them to
	float temp = atof(root["Data"]["temp"]["S"].asString());
	Serial.printlnf("temp=%f (as float)", temp);

}

Results:

batV=4.00
deviceId=540056000a51343334363138
lat=41.2083
lon=-73.3440
batSoC=88
uvI=0.0
wDir=0
hum=29.0
temp=21.3
pres=970.7
tm=1459294350
temp=21.299999 (as float)

smd75jr · March 30, 2016, 1:33am

Thank you very much!! It worked almost perfectly. I only had to recast the the variable that initially holds the data from a const char* to char* since it’s from a Particle.subscribe.

BulldogLowell · March 30, 2016, 3:05am

another method to consider… Mustache Templates

MORA · March 30, 2016, 9:58am

200 bytes is not enough to hold that object in memory, even the json string is more than 200 chars, leading to partial parsing.
The easy way out is using DynamicJsonBuffer it allocates the memory it needs on its own, but it may be a bit slower since it does it in cycles.
Or you can use a larger staticbuffer, like @rickkas7 showed.

rickkas7 · March 30, 2016, 12:08pm

Yes, a mustache template is a great idea. Not only does this make the data significantly smaller, from 287 bytes to 187 bytes), but it also corrects the weird raw data format you have that makes everything a string.

Here’s the mustache template:

{"batV":{{Data.batV.S}},"deviceId":"{{Data.deviceId.S}}","lat":{{Data.lat.S}},"batSoC":{{Data.batSoC.S}},"uvI":{{Data.uvI.S}},"wDir":{{Data.wDir.S}},"wSpd":{{Data.wSpd.S}},"wGst":{{Data.wGst.S}},"hum":{{Data.hum.S}},"temp":{{Data.temp.S}},"tm":{{Data.tm.N}},"lon":{{Data.lon.S}},"pres":{{Data.pres.S}}}

And you can see how much easier to use the reading code is. It takes advantage of the SparkJson library’s automatic support for int/float/long variables.

#include "Particle.h"
#include "SparkJson/SparkJson.h"

const char *rawJsonData =
"{\"batV\":4.00,\"deviceId\":\"540056000a51343334363138\",\"lat\":41.2083,\"batSoC\":88,\"uvI\":0.0,\"wDir\":0,\"wSpd\":0.0,\"wGst\":0.0,\"hum\":29.0,\"temp\":21.3,\"tm\":1459294350,\"lon\":-73.3440,\"pres\":970.7}"; 


void parseWeather(char *data); // forward declaration

void setup() {
	Serial.begin(9600);

	delay(5000);
	char *dataCopy = strdup(rawJsonData);
	parseWeather(dataCopy);
	free(dataCopy);

}

void loop() {

}

void parseWeather(char *mutableData) {
	StaticJsonBuffer<512> jsonBuffer;
	JsonObject& root = jsonBuffer.parseObject(mutableData);

	if (!root.success()) {
		Serial.println("parseObject() failed");
	    return;
	}

	Serial.printlnf("batV=%f", (float) root["batV"]);
	Serial.printlnf("deviceId=%s", root["deviceId"].asString());
	Serial.printlnf("lat=%f", (float) root["lat"]);
	Serial.printlnf("lon=%f", (float) root["lon"]);
	Serial.printlnf("batSoC=%d", (int) root["batSoC"]);
	Serial.printlnf("uvI=%f", (float) root["uvI"]);
	Serial.printlnf("wDir=%f", (float) root["wDir"]);
	Serial.printlnf("wSpd=%f", (float) root["wSpd"]);
	Serial.printlnf("wGst=%f", (float) root["wGst"]);
	Serial.printlnf("hum=%f", (float) root["hum"]);
	Serial.printlnf("temp=%f", (float) root["temp"]);
	Serial.printlnf("pres=%f", (float) root["pres"]);

	Serial.printlnf("tm=%ld", (long) root["tm"]);

}

rickkas7 · March 30, 2016, 5:00pm

Or, if you’re going to use mustache, you can just eliminate the entire JSON library, saving lots of code space and reducing the data down to 92 bytes.

Mustache template:

{{Data.batV.S}},{{Data.batSoC.S}},{{Data.lat.S}},{{Data.lon.S}},{{Data.uvI.S}},{{Data.wDir.S}},{{Data.wSpd.S}},{{Data.wGst.S}},{{Data.hum.S}},{{Data.temp.S}},{{Data.pres.S}},{{Data.tm.N}},{{Data.deviceId.S}}

Code with manual parsing:

#include "Particle.h"


const char *rawData = "4.00,88,41.2083,-73.3440,0.0,0,0.0,0.0,29.0,21.3,970.7,1459294350,540056000a51343334363138";


void parseWeather(const char *data); // forward declaration

void setup() {
	Serial.begin(9600);

	delay(5000);

	parseWeather(rawData);

}

void loop() {

}

typedef struct {
	float batV;
	int batSoC;
	float lat;
	float lon;
	float uvI;
	int wDir;
	float wSpd;
	float wGst;
	float hum;
	float temp;
	float pres;
	long tm;
	String deviceId;

	bool read(const char *data);
} WeatherData;

const char *dataSeparator = ",";

bool WeatherData::read(const char *data) {
	char buf[32];

	// Mustache template:
	// {{Data.batV.S}},{{Data.batSoC.S}},{{Data.lat.S}},{{Data.lon.S}},{{Data.uvI.S}},{{Data.wDir.S}},{{Data.wSpd.S}},{{Data.wGst.S}},{{Data.hum.S}},{{Data.temp.S}},{{Data.pres.S}},{{Data.tm.N}},{{Data.deviceId.S}}

	char *dataCopy = strdup(data);
	char *cp = strtok(dataCopy, dataSeparator);
	if (cp != NULL) {
		batV = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		batSoC = atoi(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		lat = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		lon = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		uvI = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		wDir = atoi(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		wSpd = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		wGst = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		hum = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		temp = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		pres = atof(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		tm = atol(cp);
		cp = strtok(NULL, dataSeparator);
	}
	if (cp != NULL) {
		deviceId = cp;
	}
	free(dataCopy);

	// This would be easier with sscanf, but I can't get it to properly read the floating point values
	//if (sscanf(data, "%f,%f,%d,%f,%d,%f,%f,%f,%f,%f,%f,%ld,%31s", &batV, &lat, &batSoC, &uvI, &wDir, &wSpd, &wGst, &hum, &temp, &lon, &pres, &tm, buf) == 13) {

	return (cp != NULL);
}

void parseWeather(const char *stringData) {
	WeatherData weather;
	if (!weather.read(stringData)) {
		Serial.println("failed to parse");
		return;
	}

	Serial.printlnf("batV=%f", weather.batV);
	Serial.printlnf("batSoC=%d", weather.batSoC);
	Serial.printlnf("lat=%f", weather.lat);
	Serial.printlnf("lon=%f", weather.lon);
	Serial.printlnf("uvI=%f", weather.uvI);
	Serial.printlnf("wDir=%f", weather.wDir);
	Serial.printlnf("wSpd=%f", weather.wSpd);
	Serial.printlnf("wGst=%f", weather.wGst);
	Serial.printlnf("hum=%f", weather.hum);
	Serial.printlnf("temp=%f", weather.temp);
	Serial.printlnf("pres=%f", weather.pres);
	Serial.printlnf("tm=%ld", weather.tm);
	Serial.printlnf("deviceId=%s", weather.deviceId.c_str());

}

hbierau · July 13, 2016, 11:38am

Hey guys, this might be a stupid question, so apologies for that in advance, but I cannot get my head around (I think it is my limited understanding of JSON) and maybe you can give me a hint. I am implementing a project to control a Philips Hue system. This works over a REST web service and I successfully implemented the part of PUT commands to switch lights on and off. Using GET calls (to which the system responds with JSON), information about on / off states etc can be retrieved. I implemented a rudimentary way of parsing the HTTP response by “extracting” the relevant information using findUntil() and readStringUntil(), e.g.

HueHub.findUntil("\"on\":", "\0");
String jsonExtract = HueHub.readStringUntil(',');
Serial.println(Extract);
if (Extract == "true") LighIsOn = true;

This works, but it is not very elegant. Also, I have not tested it on larger responses, e.g. for several lights in one response. So I though to implement a JSON parser based on one of the libraries available. I started off with the “SparkJson” library and the first thing I did was to use the JSONPARSEXAMPLE.INO file (which compiles correctly) and paste in a sample response from my Hue system, just to see what happens. However, the compiler complains with messages like “error: unable to find string literal operator 'operator”" state’". My suspicion is that the format of the JSON is somehow different. For example, the JSONPARSEXAMPLE.INO file parses the following string:

"{\"sensor\":\"gps\",\"time\":1351824120,\"data\":[48.756080,2.302038]}"

while the one I got from the Hue system looks like that:

{"state": {"on":false,"bri":254,"hue":13399,"sat":204,"effect":"none",....

So it does not have all the “” to separate the elements. Are there different JSON formats/specs? Can they be converted? Any suggestions how I can process the responses from my system?

rickkas7 · July 13, 2016, 11:43am

The backslashes are a C thing, not a JSON thing. Because strings in C are surrounded by double quotes, you cannot put a double quote in a C string unless you escape it with a backslash. The backslashes are removed by the compiler, so if you were to print out each character in the string from the C code, you wouldn’t see the backslashes; it would look like normal JSON again.

hbierau · July 13, 2016, 11:58am

Hey rickkas7, thanks for the swift response. Yes, I thought already it is something like that. I was just looking at the github page of the author of the original Arduino libary (from which the Spark one was ported) commenting on the example. There he explains…

Here an example that parse the string {"sensor":"gps","time":1351824120,"data":[48.756080,2.302038]}:

and he puts the following in his Arduino (C-)code:

char json[] = "{\"sensor\":\"gps\",\"time\":1351824120,\"data\":[48.756080,2.302038]}";

What I still don’t get is how to “convert” the original JSON, eg. that I receive as a response from a http (GET or PUT) call, i.e. without the backslashes into a C-compatible string that I can then process with the library? Would I need to read the response into a string and then run it through a function which inserts the backslashes in the right places (e.g. before each ")? How did you solve this? Thx in advance.

rickkas7 · July 13, 2016, 12:03pm

Using SparkJson, you typically do something like:

StaticJsonBuffer<512> jsonBuffer;
JsonObject& root = jsonBuffer.parseObject(dataYouGotFromTheServer);

The examples use a string hardcoded in the code, but you’d just pass the string directly into parseObject. Replace dataYouGotFromTheServer with the variable that contains the data you got from the server. You never have to deal with the backslashes at all.

hbierau · July 13, 2016, 12:38pm

Excuse my stubbornness and probably also "stupidity, your patience is very much appreciated … but I still have difficulties implementing this although it should be straightforward. It’s true, the example uses a hard-coded string, or rather, a char array:

char json[] = "{\"sensor\":\"gps\",\"time\":1351824120,\"data\":[48.756080,2.302038]}";

However, it then passes the char on to the object with exactly the same command that you mention:

JsonObject& root = jsonBuffer.parseObject(json);

So in a sense as if I would do

char json[] = dataYouGotFromTheServer; 
JsonObject& root = jsonBuffer.parseObject(dataYouGotFromTheServer);

What I still don’t get is, the char that is passed on to the object already contains the backslashes ("{“sensor”:“gps”,“time”:1351824120,“data”:[48.756080,2.302038]}") , so in a sense, it is already C-compatible. If I use the original json without the backslashes ({“sensor”:“gps”,“time”:1351824120,“data”:[48.756080,2.302038]}) and pass this on to the object, it wont work.

ScruffR · July 13, 2016, 12:56pm

Nope, it should not.
These backslashes should not be contained in the received string - they are not even in the string literal once compiled, they only exist in your source code.
These are just a way of writing special characters in code. "Escaping" characters is a way "invented" to enable you to code such otherwise "unwritable" things.
But the escape character never ends up in the compiled binary.

An illustration:
You might want to write a screen play where an actor should "blow a raspberry", but how would you actually write that?
Since it is impossible to write the actual sound with letters, someone came up with the "escaped" term "blow a raspberry", but the actor who's going to play that part will never actually say: "I'm feeling blow a raspberry", but he'd just do it

hbierau · July 13, 2016, 1:14pm

Thanks to you both for bearing with me…
OK, coming back to the orgininal questin, how would I go about parsing the response from a HTTP call. Should something like this work?

  if (HueHub.connect(HueHubIP, HueHubPort)) { 
    HueHub.print("GET /api/");
    .... //code for the HTTP GET call (works)
  }
  while (HueHub.available()) {
     char HueResponse[] = client.read();
  }
  JsonObject& root = jsonBuffer.parseObject(HueResponse);   // to pass the json response to the object

ScruffR · July 13, 2016, 1:17pm

This will not work for several reasons.

HueResponse does not exist outside of the while() { } block
you’d overwrite the contents of HueResponse with each iteration of while()
with char HueResponse[] you are not actually reserving any memory for the string to go
you can’t be certain that you actually got all your expected data just because you fell out of while()
client.available() just tells you that there is some data, but not necessarily all of it

hbierau · July 13, 2016, 1:34pm

You are right of course regarding the declaration. This should be a global variable. I included the declaration here in order to show how I would declare it.

The length of the response can be variable. If I declare HueResponse globally with a pointer, let's say like this:

char *HueResponse;

Would that work? and how would I read out the response and store it in HueResponse?
Thx for your patience.

hbierau · July 13, 2016, 1:38pm

Got it, client.available is not a boolean type answer, so I need to do something like client.available() > 0, right?

ScruffR · July 13, 2016, 2:18pm

Nope, that would be the same, you'd not have any memory reserved to put your data. The pointer can point anywhere but doesn't "own" anything.
You need to reserve some space. You could do that dynamically, but that would require some better understanding of the matter, for a "beginner" I'd suggest to go for a fixed size buffer which should be big enough to hold even the longest single set of data you'd expect to deal with.
So at least start off with

char HueResponse[512];

and fill that buffer like so

  HueResponse[index++] = client.read();

But while(client.available() > 0) will still not be enough to know whether you got all your data or not. You'd still only know that you got all the data that was already received but won't know if there's more to come.
This can only be done by either checking the actual data for completenes (e.g. via a terminator byte) or when you know how many bytes you should expect and check if you already have got all of them.

hbierau · July 13, 2016, 2:28pm

Thanks very much for all your help. Will give this a try as soon as I have the occasion and let you know.

jeremywmccarter · November 8, 2016, 7:31pm

Hey guys,
Jumping on an old post here, but I’m really struggling to understand how to get useable data out of my Webhook call.

I have a Google Sheet with two columns of data, like so:

       A                 B  
1 User Name          Password
2 Alfred Albrecht      1234
3 Bob Billings         2234
4 Charlie Chaplin      3234
5 Dave Dink            4234

Using a Webhook, I’m able to retrieve the data from Google Sheets, as shown in the Particle Log:

{"data":"{\n \"range\": \"Sheet1!A2:B7\",\n \"majorDimension\": \"COLUMNS\",\n \"values\": [\n [\n \"Albert Albrecht\",\n \"Bob Billings\",\n \"Charlie Chaplin\",\n \"Dave Dink\"\n ],\n [\n \"1234\",\n \"2234\",\n \"3234\",\n \"4234\"\n ]\n ]\n}\n","ttl":"60","published_at":"2016-11-08T19:28:44.039Z","coreid":"particle-internal","name":"hook-response/getToolNames3/0"}

From there, I can do:

Serial.println(data);

…and get the following:

{
  "range": "Sheet1!A2:B7",
  "majorDimension": "COLUMNS",
  "values": [
    [
      "Albert Albrecht",
      "Bob Billings",
      "Charlie Chaplin",
      "Dave Dink"
    ],
    [
      "1234",
      "2234",
      "3234",
      "4234"
    ]
  ]
}

What I’d like to have is two arrays in this format:

String UserName [4] = {"Albert Albrecht", "Bob Billings", "Charlie Chaplin", "Dave Dink"}; 

int UserPassword [4] = {1234, 2234, 3234, 4234};

@rickkas7, I saw you’d done some great work above, and I’m trying to grab what is relevant to my data, just not getting there. Could you show me how you might parse this data?

Any advice from the community?

Thanks all, in advance,
Jeremy

Topic		Replies	Views
Parsing JSON help Troubleshooting	6	2267	February 5, 2017
Noob help: Parsing json with SparkJson Troubleshooting	21	5340	April 22, 2020
Trouble Parsing JSON from Webhook Troubleshooting	3	1297	September 18, 2017
How to Easily Parse JSON Payload from Webhook Using JsonParserGeneratorRK Firmware	22	8611	May 11, 2018
Unable to parse JSON with Built-in JSON Parser Firmware	5	680	April 22, 2021

JSON Parsing Help

Related topics