Tutorial: Webhooks and Responses with Parsing (JSON, Mustache, Tokens)

[RWB]

I need some help with parsing some returned webhook weather data.

I tried to follow @LukeUSMC parsing method above but it needs some tweaking for my application it looks like.

So by following Luke’s example I created a custom response template to pull in only the webhook data that I’m interested in. Here is the custom response code:

"responseTemplate": "{{#weather}}{{id}}~{{main}}~{{description}}~{{/weather}}{{main.temp_min}}~{{main.temp_max}}~{{main.humidity}}~{{wind.speed}}~{{wind.deg}}~{{sys.sunrise}}~{{sys.sunset}}~{{name}}"

This provides me with the following GET response:

{"data":"\"responseTemplate\": \"800~Clear~clear sky~78.8~87.1~24~3.36~200~1460977185~1461025619~Fishers\",","ttl":"60","published_at":"2016-04-18T22:02:18.622Z","coreid":"1a002d000347343339373536","name":"hook-response/FishersWeather/0"}

So the webhook is working just fine and it was very easy to setup which is all great news.

Now I need to parse this weather data into their own seperate variables so I can use it as needed and to do that I’m trying to use Lukes code below but I need to modify it.

I changed this code some attempting to handle the larger amount of returned data. I bumped the buffer size up to 500 from 125.

Basically I’m expecting this function below to parse the data that is in between

// This function will get called when weather data comes in
void gotWeatherData(const char *name, const char *data) {

    Serial.print("Running gotweatherData function");
    String str = String(data);
    char strBuffer[500] = "";
    str.toCharArray(strBuffer, 500); // example: "\"21~99~75~0~22~98~77~20~23~97~74~10~24~94~72~10~\""

    int weathercode = atoi(strtok(strBuffer, "\"~"));
    int maxtempday1 = atoi(strtok(NULL, "~"));
    int mintempday1 = atoi(strtok(NULL, "~"));
    int maxwindday1 = atoi(strtok(NULL, "~"));
    int forecastday2 = atoi(strtok(NULL, "~"));
    int maxtempday2 = atoi(strtok(NULL, "~"));
    int mintempday2 = atoi(strtok(NULL, "~"));
    int maxwindday2 = atoi(strtok(NULL, "~"));
    int forecastday3 = atoi(strtok(NULL, "~"));
    int maxtempday3 = atoi(strtok(NULL, "~"));
    int mintempday3 = atoi(strtok(NULL, "~"));
    int maxwindday3 = atoi(strtok(NULL, "~"));
    
    Serial.println("Weaher Code");
    Serial.println(weathercode);
   Serial.println(maxtempday1);
   Serial.println(maxtempday1);
   Serial.println(maxwindday1);
   Serial.println(maxtempday2);
   Serial.println(mintempday2);
  
}

The serial output from the function above looks like this. I’m just testing to see if the data between the ~ & ~ signs get put into the variables but only a few get parsed for some unknown reason. I know the weather description and main are text character data so they probably shouldn’t be placed in INT holders. I tried changing them to char data types but that didn’t help.

@ScruffR So I guess my first question is if this is the most efficient way of parsing the webhook response or not. If it is then can you give me any hints about what needs to be changed?

All help is appreciated

[LukeUSMC]

I have an updated “gotWeather” I use but the prob is you need to parse the data in order. Each piece of data between ~data~ needs its own variable to map to. You have 3 things to parse before you get the first temp and that should be a float not an int. Match them to the proper data type being sent.

UPDATED:
Totally untested and likely to need a bit of tuning but try this out. The key here is to make sure you parse things in order, the gotWeatherData is just extracting the things between ~data1~data2~data3~. You have to “map”/parse data1 before you can map data2.

void gotWeatherData(const char *name, const char *data) {

    Serial.print("Running gotweatherData function");
    String str = String(data);
    char strBuffer[500] = "";
    str.toCharArray(strBuffer, 500); // example: \"800~Clear~clear sky~78.8~87.1~24~3.36~200~1460977185~1461025619~Fishers\",

    int weatherID = atoi(strtok(strBuffer, "\"~"));
    String weatherMain = strtok(NULL, "~");
    String weatherDesc = strtok(NULL, "~");
    float mintemp = atof(strtok(NULL, "~"));
    float maxtemp = atof(strtok(NULL, "~"));
    int mainHumidity = atoi(strtok(NULL, "~"));
    int maxwind = atoi(strtok(NULL, "~"));
    float windDeg = atof(strtok(NULL, "~"));
    uint32_t sunrise = atoi(strtok(NULL, "~"));
    uint32_t sunset = atoi(strtok(NULL, "~"));
    String weatherName = strtok(NULL, "~");
    
  Serial.print("Weather Code: ");
  Serial.println(weatherID);
  Serial.print("Max Temp: ");
  Serial.println(maxtemp);
  Serial.print("Max Wind: ");
  Serial.println(maxwind);

  
}

[RWB]

@LukeUSMC Just saw your reply so I’ll follow up after testing the code you provided.

[LukeUSMC]

Here we go…atoi is a function from core c stdlib (no include needed)
http://www.cplusplus.com/reference/cstdlib/atoi/
Look at the first part of the parse statement and you can see it is looking for the beginning of your response which is "~ then 800 which is the first piece of data which is an integer so we use atoi to convert the extracted char to an int. The strtok is another function from cstring lib (no include needed).
http://www.cplusplus.com/reference/cstring/strtok/

Each subsequent line picks up from where the last one left off until we stop giving it things to parse out to vars. Remember all of those variables will go out of scope as soon as that function completes so if you want to keep any of that data you need declare them outside of the function then use them in function. What I actually do is parse to local vars like you have then after some validation process I set my globals to equal the temp values. A bad example of data validation is below…remember “trust no data until validated!”

if (maxtemp != 0){
gMaxTemp = maxtemp;
}

Let me know how the testing goes.

[RWB]

OK, were making progress.

Also thinks for the links to the atoi & strtok info!

I’m using this function code:

void gotWeatherData(const char *name, const char *data) {

    Serial.println("Running gotweatherData function");
    String str = String(data);
    char strBuffer[500] = "";
    str.toCharArray(strBuffer, 500); // example: \"800~Clear~clear sky~78.8~87.1~24~3.36~200~1460977185~1461025619~Fishers\",

    int weatherID = atoi(strtok(strBuffer, "\"~"));
    String weatherMain = strtok(NULL, "~");
    String weatherDesc = strtok(NULL, "~");
    float mintemp = atof(strtok(NULL, "~"));
    float maxtemp = atof(strtok(NULL, "~"));
    int mainHumidity = atoi(strtok(NULL, "~"));
    int maxwind = atoi(strtok(NULL, "~"));
    float windDeg = atof(strtok(NULL, "~"));
    uint32_t sunrise = atoi(strtok(NULL, "~"));
    uint32_t sunset = atoi(strtok(NULL, "~"));
    String weatherName = strtok(NULL, "~");
    
  Serial.print("Weather Code: ");
  Serial.println(weatherID);
  Serial.print("Description: ");
  Serial.println(weatherMain);
   Serial.print("Description 2: ");
  Serial.println(weatherDesc);
  Serial.print("Max Temp: ");
  Serial.println(maxtemp);
  Serial.print("Max Wind: ");
  Serial.println(maxwind);

  
}

On this webhook reply data:

{"data":"\"responseTemplate\": \"800~Clear~clear sky~69.8~78.8~43~4.16~257.001~1461063574~1461112027~Fishers\",","ttl":"60","published_at":"2016-04-19T01:05:38.278Z","coreid":"1a002d000347343339373536","name":"hook-response/FishersWeather/0"}

And now I’m getting to the desired data to show up.

So I see that the first line is returning 0 and the Description line is returning "800.

I’m trying to figure out how to eliminate the first variable that is returning 0 since it’s not needed. This line of code is the first parse and its returning 0. Since this data is not needed can we delete it?

 int weatherID = atoi(strtok(strBuffer, "\"~"));

---

This next line of code is parsing the first bit of data which is the weather status code:

String weatherMain = strtok(NULL, "~");

But the data it parses has the : " in front of the 800. How do I go about parsing the 800 without the beginning : " in front of it?

[LukeUSMC]

I think I see where I went wrong. Take the “~” out of the first var around statement and make it “/”. See if that will get the first piece of data to line up. If you don’t want that you may be able to remove from your response template but if you don’t map it to another variable later it will just go away when the function completes.

[RWB]

@LukeUSMC I did try taking that out while playing around but it didn’t help for some reason. I’m sure if I kept playing around I would figure it out.

What I ended up doing was to use the gotWeatherFunction from the Particle docs since I had a better understanding of how it worked and it made more sense to my newbie brain

So I used this code here: https://docs.particle.io/guide/tools-and-features/webhooks/#the-weather-firmware

I changed the gotWeatherFunction to fit my needs:

  // This function will get called when weather data comes in

void gotWeatherData(const char *name, const char *data) {

    // Important note!  -- Right now the response comes in 512 byte chunks.
    //  This code assumes we're getting the response in large chunks, and this
    //  assumption breaks down if a line happens to be split across response chunks.
    //
    // Sample data:
    //  <location>Minneapolis, Minneapolis-St. Paul International Airport, MN</location>
    //  <weather>Overcast</weather>
    //  <temperature_string>26.0 F (-3.3 C)</temperature_string>
    //  <temp_f>26.0</temp_f>


    String str = String(data);
    String weatheridStr = tryExtractString(str, "<WeatherID>", "</WeatherID>");
    String weatherstatusStr = tryExtractString(str, "<WeatherStatus>", "</WeatherStatus>");
    String weatherdiscriptionStr = tryExtractString(str, "<WeatherDescription>", "</WeatherDescription>");
    String tempStr = tryExtractString(str, "<Temp>", "</Temp>");
    String humidityStr = tryExtractString(str, "<Humidity>", "</Humidity>");
    String windspeedStr = tryExtractString(str, "<WindSpeed>", "</WindSpeed>");
    String winddirectionStr = tryExtractString(str, "<WindDirection>", "</WindDirection>");
    String cloudinessStr = tryExtractString(str, "<Cloudiness>", "</Cloudiness>");
    String sunriseStr = tryExtractString(str, "<Sunrise>", "</Sunrise>");
    String sunsetStr = tryExtractString(str, "<Sunset>", "</Sunset>");
    String cityStr = tryExtractString(str, "<City>", "</City>");
   
    

    if (weatheridStr != NULL) {
        Serial.println("Weather ID: " + weatheridStr);
    }

    if (weatherstatusStr != NULL) {
        Serial.println("Weather Status: " + weatherstatusStr);
    }

    if (weatherdiscriptionStr != NULL) {
        Serial.println("Weather Discription: " + weatherdiscriptionStr);
    }

    if (tempStr != NULL) {
        Serial.println("The Temp is: " + tempStr + String(" *F"));
    }
    
     if (humidityStr != NULL) {
        Serial.println("The Humidity is: " + humidityStr + String(" %"));
    }
    
     if (windspeedStr != NULL) {
        Serial.println("The Wind Speed is: " + windspeedStr + String(" MPH"));
    }
    
     if (winddirectionStr != NULL) {
        Serial.println("The Wind Direction is: " + winddirectionStr + String(" Degrees"));
    }
    
    if (cloudinessStr != NULL) {
        Serial.println("The Cloudcover is: " + cloudinessStr + String(" %"));
    }
    
     if (sunriseStr != NULL) {
        Serial.println("The Sun Will Rise at: " + sunriseStr + String(" UTC"));
    }
    
    if (sunsetStr != NULL) {
        Serial.println("The Sun Will Set at: " + sunsetStr + String(" UTC"));
    }
    
     if (cityStr != NULL) {
        Serial.println("Weather Data for: " + cityStr );
    }
    
}

// Returns any text found between a start and end string inside 'str'
// example: startfooend  -> returns foo
String tryExtractString(String str, const char* start, const char* end) {
    if (str == NULL) {
        return NULL;
    }

    int idx = str.indexOf(start);
    if (idx < 0) {
        return NULL;
    }

    int endIdx = str.indexOf(end);
    if (endIdx < 0) {
        return NULL;
    }

    return str.substring(idx + strlen(start), endIdx);
}

I changed the Custom Response code to this:

"responseTemplate": "{{#weather}}<WeatherID>{{id}}</WeatherID><WeatherStatus>{{main}}</WeatherStatus><WeatherDescription>{{description}}</WeatherDescription>{{/weather}}<Temp>{{main.temp}}</Temp><Humidity>{{main.humidity}}</Humidity><WindSpeed>{{wind.speed}}</WindSpeed><WindDirection>{{wind.deg}}</WindDirection><Cloudiness>{{clouds.all}}</Cloudiness><Sunrise>{{sys.sunrise}}</Sunrise><Sunset>{{sys.sunset}}</Sunset><City>{{name}}</City>",

And now the data I get via Serial Print is exactly what I wanted

Now I just need to save these variables to Global variables so they are available when I need them.

And as Luke suggested I should validate the data before I update the global variable.

“trust no data until validated!”

if (maxtemp != 0){
gMaxTemp = maxtemp;
}

Thanks @LukeUSMC

[ScruffR]

@RWB, glad you get your data, but in answer to this question of yours ...

@RWB

So I guess my first question is if this is the most efficient way of parsing the webhook response

... I'd say: "The way you got it now isn't"

It uses too many String objects for my liking (personal opinion), uses a much longer response string, is slower and "clumsier".
@RWB's strtok() is definetly more my own taste too, although I'd see the extra String str = String(data); superfluous, as you could just as well directly write

  char szBuffer[strlen(data)+1]; // hungarian notation for zero-terminated string (aka C string)
  strcpy(szBuffer, data);

And it's always worth to wrap your brain round these C functions, they make dealing with dynamic data much more flexible.

An alternative aproach would be sscanf()

[RWB]

@ScruffR I agree with you 100% especially since some of this will be done using the Electron where data usage needs to be kept to a minimum.

I’m glad I got to learn about these 2 ways to accomplish the same thing.

I would say that the way I did it for sure is not the most efficient but it is for me the most readable and I do see ways to cut it down so the webhook response is not so long an therefore does not consume so much data when used on a Electron.

The C functions do look more flexible but honestly most of the time it just confuses me because it’s not presented in a well documented format like all the Arduino Youtube video tutorials which is how I usually learn the quickest.

Since on the Photon the Wifi data is free I’m cool with the way I have it working now even though it consumes more data to work.

@ScruffR Thanks for your feedback as always Your brain contains a wealth of knowledge

[ScruffR]

Would these help?
strtok: https://www.youtube.com/watch?v=fpbOh3k4z04
parse: https://www.youtube.com/watch?v=OYK5iOXIj9M

Or any of the others
https://www.google.at/?ion=1&espv=2#q=strtok%20video

[RWB]

I checked out the videos and they are not very good at explaining it in plain English. Both videos basically have no audio coaching it’s just them typing the code out and your expected to understand everything that is going on. Don’t get me wrong, if I go over it and watch enough videos I’m sure I will eventually get it.

My experience is that once you jump from the simplified Arduino coaching over to the C++ side of the world things get really confusing really fast. That’s just my take on it as a beginner who got started with Arduino a few years ago with zero programming background experience. The C++ just blows my mind when I look at it because it so foreign to me.

Without the community help, guidance, and support I would get stuck to the point of not being able to move forward when certain things pop up.

I’m guessing when I start using this on the Electron I’m going to be back here digging deeper into this more efficient way of getting the data transferred.

For now I’m feeling pretty proud that I created my first working Webhook without needing to deal with installing the CLI

[LukeUSMC]

It does take some getting used to when you shift from Wiring/Particle to the more base C/C++ but it is a powerful thing when you do. I’m still a beginner myself. One thing to note is that you can’t use any of those string objects to do any math without using atoi or whatever is appropriate. I chose to exclude any string data from my response template because I use the data to calculate a number of things and it made it easier. Unless you are going to retransmit those strings it won’t affect your data usage (ota flashing aside) but it will affect the size of your program.

Either way, I’m glad you have a working solution either way…keep at it and looking forward to seeing your completed project!

[RWB]

@LukeUSMC Thanks for the extra info because the difference between the way you did it and the way I used it which is basically following the Particle Doc example have differences in how the data output can be handled.

Based on what your saying it looks as if your way is better than the Particle example for the simple fact that you can do math on the returned data which is important in some situations like where I’m trying to take the UTC time and convert it to normal 12 hour time.

Using the atoi & strtok functions actually were not that bad and it worked fine, the only issue I had was figuring out how to get rid of the unwanted data that was first parsed out. So now what I see clearer the difference between the 2 way of doing the parsing I now know which method is best for the particular application at hand which is valuable.

Thanks for the help and info on this subject guys

This will be used in 2 separate projects and I’ll be sure to share both when the time comes

[lmnichols1]

@LukeUSMC First off thank you for making this tutorial it has been very helpful. That being said I am trying to edit your code in order to get the sunrise and sunset times from wunderground and I'm not having much luck. Whenever I run the webhook it returns undefined data and I'm not sure what I'm doing wrong. Any help would be appreciated.

http://api.wunderground.com/api/a40f611ad11e4632/astronomy/q/IA/Iowa_City.json is my hook ID, I'm pretty sure I'm calling the correct spot/url to access the astronomy info. Sun_phase isn't mentioned in the API directory on their website for whatever reason.

{
"event": "GetSunTimes_hook",
"url": "http://api.wunderground.com/api/key/astronomy/q/IA/Iowa_City.json",
"requestType": "POST",
"headers": null,
"query": null,
"responseTemplate": "{{#astronomy}}{{#sun_phase}}{{sunrise.minute}}~{{sunrise.hour}}~{{sunset.hour}}{{sunset.minute}}~{{/sun_phase}}{{/astronomy}}",
"json": null,
"auth": null,
"mydevices": true
}

That is the code I have edited.

Thanks again.

[BulldogLowell]

@lmnichols1

here is a webhook that works:

{
	"event": "sun_time",
	"url": "http://api.wunderground.com/api/getYourOwnApiKey/astronomy/q/{{my-state}}/{{my-city}}.json",
	"requestType": "POST",
	"headers": null,
	"query": null,
	"responseTemplate": "{{#sun_phase}}{{sunrise.hour}}~{{sunrise.minute}}~{{sunset.hour}}~{{sunset.minute}}~{{#moon_phase}}{{current_time.hour}}~{{current_time.minute}}{{/moon_phase}}~{{/sun_phase}}",
	"responseTopic": "{{SPARK_CORE_ID}}_sun_time",
	"json": null,
	"auth": null,
	"coreid": null,
	"deviceid": null,
	"mydevices": true
}

Also may be of interest:

Using responseTopic in webhook for custom response to get DST offset

OK, so now that we have this new tool, I wanted to provide an example of how to use a single webhook on multiple devices returning location data. I am using my handy webhook that returns sunrise and sunset times for a city and state in the US: { "event": "sun_time", "url": "http://api.wunderground.com/api/getYourOwnApiKey/astronomy/q/{{my-state}}/{{my-city}}.json", "requestType": "POST", "headers": null, "query": null, "responseTemplate": "{{#sun_phase}}{{sunrise.hour}}~{{sunrise.minute}…

[lmnichols1]

@BulldogLowell Thanks for the help, it is appreciated. Unfortunately when I tried this it still returned undefined data on my particle dashboard. I’ll try tinkering with it, but thought I’d let you know.

This is what the response looks like on the dashboard. I’m not sure why it is defaulting the device to cloud, in comparison to @LukeUSMC code where the device is the internal particle, I am also not getting a hook sent message on the dashboard either.

[ScruffR]

Your event name in dashboard is GetSun_hook but your json looks for GetSunTimes_hook

[lmnichols1]

Ahh, that makes sense. Thanks for catching the error.

[lmnichols1]

Changing the event names made me get a response, however my returned data is still undefined. I’m going to continue tinkering with the API in the mean time because I’m assuming this is where I’ve gone wrong.

I’m also calling too many times it would appear as well.

[BulldogLowell]

Could you try this with the webhook I posted:

const char* cityLocation = "Iowa_City";  //City for my Photon
const char* stateLocation = "IA";     // State for my Photon

char publishString[125] = "";
void setup() 
{
  strcpy(publishString, "{\"my-city\": \"");
  strcat(publishString, cityLocation);
  strcat(publishString, "\", \"my-state\": \"");
  strcat(publishString, stateLocation);
  strcat(publishString, "\" }");
  
  String responseTopic = System.deviceID();
  Particle.subscribe(responseTopic, webhookHandler, MY_DEVICES);
}

void loop() 
{
  Particle.publish("sun_time", publishString, 60, PRIVATE);
  delay(60000);
}

void webhookHandler(const char *event, const char *data)
{
  if (strstr(event, "sun_time"))
  {
    gotSunTime(event, data);
  }
}

void gotSunTime(const char * event, const char * data)
{
  char sunriseBuffer[256] = "";
  strcpy(sunriseBuffer, data);
  int riseHour = atoi(strtok(sunriseBuffer, "\"~"));
  int riseMinute = atoi(strtok(NULL, "~"));
  int setHour = atoi(strtok(NULL, "~"));
  int setMinute = atoi(strtok(NULL, "~"));
  int currentHour = atoi(strtok(NULL, "~"));
  int currentMinute = atoi(strtok(NULL, "~"));
  Time.zone(0);
  Time.zone(utcOffset(Time.hour(), currentHour));
  sprintf(sunriseBuffer, "%s, %s Sunrise: %02d:%02d, Sunset: %02d:%02d, Current Time: %02d:%02d", cityLocation, stateLocation, riseHour, riseMinute, setHour, setMinute, currentHour, currentMinute);
  Particle.publish("pushover", sunriseBuffer, 60, PRIVATE);
}

int utcOffset(int utcHour, int localHour)  // sorry Baker Island, this won't work for you (UTC-12)
{
  if (utcHour == localHour)
  {
    return 0;
  }
  else if (utcHour > localHour)
  {
    if (utcHour - localHour >= 12)
    {
      return 24 - utcHour + localHour;
    }
    else
    {
      return localHour - utcHour;
    }
  }
  else
  {
    if (localHour - utcHour > 12)
    {
      return  localHour - 24 - utcHour;
    }
    else
    {
      return localHour - utcHour;
    }
  }
}