Tutorial: Webhooks and Responses with Parsing (JSON, Mustache, Tokens)


#1

First of all many thanks to @peekay123 @BulldogLowell and @Awake for their help in getting my particular use case to work but hoping that I can distill this down so others might find the answers they need easier than I did. I want to be clear that almost none of this is my own original work but an amalgamation of the previously mentioned people’s work.

What I wanted to accomplish was to query the WeatherUnderground API for Forecast data to include the following; Day of the Month, Max Temp, Min Temp and Max Windspeed. Here is how I accomplished that and have the response values displayed on a 128x64 OLED (I won’t be covering the OLED implementation for this tutorial).

Step 1: Identify the API you wish to call, the response you will get and what values you want from that response. Here is a JSON weatherunderground response for Dallas, TX that has been formatted using a Chrome Extension called JSONView JSONView Chrome Extension

I have collapsed the response and txt_forecast sections to be able to show the data I want to work with.
Once you know what data you want, the “sections” where it is contained you will need to create a file called whatever.JSON that will be used to create your webhook on the Particle.IO cloud. Remembering the data I wanted was the Day of the Month, High and Low Temp and Windspeed…here is my webhook that I named weatherU.json

{
  "event": "weatherU_hook",
  "url": "http://api.wunderground.com/api/YOURAPIKEYGOESHERE/forecast/q/TX/Allen.json",
  "requestType": "POST",
  "headers": null,
  "query": null,
  "responseTemplate": "{{#forecast}}{{#simpleforecast}}{{#forecastday}}{{date.day}}~{{high.fahrenheit}}~{{low.fahrenheit}}~{{maxwind.mph}}~{{/forecastday}}{{/simpleforecast}}{{/forecast}}",
  "json": null,
  "auth": null,
  "mydevices": true
}

Now let’s go through this a bit and explain what is there and why. The event entry is what will be shown on Particle.IO dashboard when this webhook is invoked by Spark.publish or via the CLI for testing (more on this latter). Url is not surprisingly the url of the API you wish to obtain data from. RequestType for this API is POST, some will be GET…read the documentation of the API you are working with to determine which is appropriate. I didn’t need a header or query but again, this is API dependant so my weatherU.json definitely wouldn’t work with OpenWeatherMap example in the Particle Docs which requires query entries. Now the good stuff, the reason we are doing this in the first place…the DATA! If you look at the screenclip with formatted data then the webhook you’ll notice that any section that can be expanded is opened with a # and closed with a / (Example: {{#simpleforecast}}Bunch of other tags{{/simpleforecast}}. I was able to dive into the JSON message and then select only the data I want using the response filter and Mustache. Check out trymustache.com for more on that. Notice the ~ between entries, that is important for later when we need to parse out the response into usable data. In my example I selected the Forecast Section, then Simpleforecast then Forecastday. Refer back to the screenclip again and you’ll see that once in the forecastday section of the response I then selected the exact data I wanted by subsection.item entries in my Response Filter and closed up the sections by working backwards, ‘close’ the last thing you ‘opened’ first and progress like so until you are back to the main (I did not close the main section #forecast but it didn’t affect my results as there was no higher entry to get to). So to get the Day of the Month I entered {{date.day}} and so on for the other data points I needed for my application. The remaining entry that is Particle specific is the "mydevices": true which tells the Particle.io cloud that this webhook is only for the devices I have claimed. Don’t put webhooks out there for public that have your personal API key and don’t post your API keys for anything!

Step 2. Now take your whatever.json file and save it someplace you will remember and open the particle cli. Login to the particle.io cloud by typing “particle login”, enter your email and password then navigate to the folder where you saved whatever.json. Now you can create your webhook by running “particle webhook create whatever.json”.

You will receive a success or fail message along with your webhook ID (which as far as I can tell you only need if you want to delete it later but you can find that anytime by running “particle webhook list”. Now that you have a webhook it is time to test if you got your {{}} and ~ where they return something you can parse and use. To test your newly minted webhook, open dashboard.particle.io in your browser of choice, run “particle publish your_event_name”, here is my weatherU_hook event being tested from the cli.

Great, we posted an event but where did it go? If you go back to the Particle.io Dashboard you opened earlier you should see two new events, like this.


NOTE:If you don’t have a response that means your webhook was bad and did not return any data to the particle.io cloud, check the API documentation again and adjust accordingly.

Step 3. Now we have a webhook that is returning the data we want with ~ between each entry so we can parse it. In my example the weatherunderground forecast returns current +3 more days worth of forecast data. So when you look at the data in the response you will see dayofmonth~maxtempthatday~mintempthatday~maxwindspeedthatday~ and that is repeated 3 more times. In my example you can see it is the 21st of August so the first piece of data is 21. In our particle firmware there are few things to do so we can “trigger” (proper term is Publish) this webhook, catch the response and parse it.

Step 4, Starting with how to trigger a webhook call. First we define the hook Response and Hook Publish statements (this makes it modular for later, instead of changing lines in code we can just edit the definitions in the beginning of our firmware).

#define HOOK_RESP	"hook-response/weatherU_hook"
#define HOOK_PUB	"weatherU_hook"

Remember we named our hook event "weatherU_hook and if you look at the data in dashboard you will see how that correlates to the HOOK_RESP entry.

Working our way down our firmware code we need to put an entry in setup to subscribe to our hook response. This tells our device what to do when the particle.io cloud posts a response. My entry looks like so:

Spark.subscribe(HOOK_RESP, gotweatherData, MY_DEVICES);

This will call the function gotweatherData when the particle.io cloud posts a hook response that matches my HOOK_RESP in my definitions. Starting to tie together?

Now lets trigger (Spark.publish) our webhook in our code so the particle.io cloud will reach out to the API we designated and post a response. In my instance the call looks like this:

Spark.publish(HOOK_PUB);

This actually isn’t sufficient though. I have learned that you need to give the cloud some time to respond or depending on how you are calling this your loop will post your Spark.publish over and over. Here is my getWeather function (which is a slightly modified version of @peekay123 getWeather function in his RGB Pong Clock project Peekay123 RGBPong Clock Github).

//Updates Weather Forecast Data
void getWeather() {
  Serial.print("in getWeather");
  Serial.println();
  weatherGood = false;
  // publish the event that will trigger our webhook
  Spark.publish(HOOK_PUB);

  unsigned long wait = millis();
  //wait for subscribe to kick in or 5 secs
  while (!weatherGood && (millis() < wait + 5000UL))
    //Tells the core to check for incoming messages from particle cloud
    Spark.process();  
  if (!weatherGood) {
    Serial.print("Weather update failed");
    Serial.println();
    badWeatherCall++;
    if (badWeatherCall > 2) {
      //If 3 webhook call fail in a row, Print fail
      Serial.print("Webhook Weathercall failed!");
    }   
  }
  else
    badWeatherCall = 0;
}//End of getWeather function

Step 5. Now we’ve built a webhook, subscribed to that event and have called for particle.io to give us the response which will in turn call our gotWeather function. The gotweatherData function will parse the response from a bunch of numbers with ~ between them to the data I actually want. Here is my gotWeather function which uses the ~ to tokenize the data and maps it to int variables for use elsewhere.


void gotweatherData(const char *name, const char *data) {
    Serial.print("Running gotweatherData function");
    String str = String(data);
    char strBuffer[125] = "";
    str.toCharArray(strBuffer, 125); // example: "\"21~99~75~0~22~98~77~20~23~97~74~10~24~94~72~10~\""
    int forecastday1 = atoi(strtok(strBuffer, "\"~"));
    int maxtempday1 = atoi(strtok(NULL, "~"));
    int mintempday1 = atoi(strtok(NULL, "~"));
    int maxwindday1 = atoi(strtok(NULL, "~"));
    int forecastday2 = atoi(strtok(NULL, "~"));
    int maxtempday2 = atoi(strtok(NULL, "~"));
    int mintempday2 = atoi(strtok(NULL, "~"));
    int maxwindday2 = atoi(strtok(NULL, "~"));
    int forecastday3 = atoi(strtok(NULL, "~"));
    int maxtempday3 = atoi(strtok(NULL, "~"));
    int mintempday3 = atoi(strtok(NULL, "~"));
    int maxwindday3 = atoi(strtok(NULL, "~"));
    if (forecastday1 == Time.day()) {
      currentFcastday = forecastday1;
      currentFcastmax = maxtempday1;
      currentFcastmin = mintempday1;
      currentFcastwind = maxwindday1;
      Serial.print("Selected forecastday1");
    }
    if (forecastday2 == Time.day()){
        currentFcastday =forecastday2;
        currentFcastmax = maxtempday2;
        currentFcastmin = mintempday2;
        currentFcastwind = maxwindday2;
        Serial.print("Selected forecastday2");
      }
    bool weatherGood = true;
    updateweatherhour = Time.hour();
}

In my example since I received multiple days worth of data but only want the current day so I am making sure that the day of the month matches the current system day of the month then maps them to different int vars I am using elsewhere in my code.

Just to round this out my application only requires that I update the forecast once an hour so I am tracking if the forecast has been updated in the current hour and only calling for my getWeather function when the current hour doesn’t match the updateweatherhour var. Here is what that looks like in my loop.

if (Time.hour() != updateweatherhour){
  getWeather();
}

@peekay123’s RGBPongClock has a much more elegant way of doing all of this but I simplified it down for my needs. If you are parsing data that isn’t an int value you can find examples of float values and other things in this thread: Community Thread: Execute function once a day OR after reset/reboot

Thanks everyone for a great community and all the help, hopefully this will help others in the future.


Newbie - retrieving data from web services
Struggling with webhook response
Object building from JSON List
Photon publish to IFTTT to PHP to IFTTT and back to the Photon
Best way to get JSON from web and parse with Photon?
Tutorial idea for 'reverse' of Internet of Things
JSON Parsing Help
Calling/processing webhook causes connection to drop using the weather example code
Webhook response filtering
Project Help - Parsing MTA Bustime data from a json request
Any JSON parsing library for Electron devices?
Parsing weather api
Parsing web pages/data to control servos/leds?
Help with subscribe - reading .publish from other photon
[SOLVED] How to Parse Numbers from JSON into analogWrite Values?
Next thing to learn
Photon publish to IFTTT to PHP to IFTTT and back to the Photon
Photon Thermostat Based on Spark Nest - Added Web Control, Anticipation, and Scheduling
Getting just temperature from any weather service?
Is this the best way to handle simple Ubidots Webhook Responses?
Global Twitter Mood Light
Artik Cloud with Particle HW
Xxx does not name a type error / .ino vs .cpp / a first project
Webhook responseTemplate not working correctly?
Connect to Wunderground Website
#2

Fantastic tutorial! Thank you!
I will be applying you weather forecast code to my current project.


#3

I haven’t adjusted for time zone or daylight savings so be aware of that but other than that it’s working for me now!


#4

This is what I use.
The Photon syncs its time to a time server when it connects to the network, and from then on uses the built in RTC to keep track of time.
This will force a time sync for the RTC and adjust for time zone.

  Time.zone (-5);
  while(Time.year() <= 1970){
    Spark.syncTime();
    Spark.process();
    delay (100);
    }

from then on you can use the built in Time functions such as:
Time.now()
Time.minute()
and so on… they are listed in the docs.

Adjusting for DST is a problem because it is determined by state legislatures, and can vary from year to year.


#5

Hmmm…
Now that we know how to request and parse JSON… take a look at this… it tells you the timezone offset for your IP address… (fails with proxy or onion router but otherwise)… should handle DST!
http://www.telize.com/geoip


#6

I’ll check that out. The other part I’d like to figure out is hour to make the webhook dynamic. So the device queries based on its physical location instead of the hard coded value I set.


#7

How would you modify your .json file when you don’t have leading headings for every group of data, such as those from this .json api response?

"daily": {
 "data": [
  {
    "apparentTemperatureMax": 72.6, 
    "apparentTemperatureMaxTime": 1444510800, 
    "apparentTemperatureMin": 40.69, 
    "apparentTemperatureMinTime": 1444474800, 
  }, 
  {
    "apparentTemperatureMax": 82.86, 
    "apparentTemperatureMaxTime": 1444597200, 
    "apparentTemperatureMin": 56.06, 
    "apparentTemperatureMinTime": 1444564800, 
  }, 

In other words, how do you specify the second value of “apparentTemperatureMin”?


#8

can you post the entire JSON response?

Try to develop a template that will return every “apparentTemperatureMin” and then tokenize and parse the return in your function, keeping only the specific data (day?) you want to retain.


#9

The rest of the JSON response is not really relevant to what I’m trying to parse. It just contains more objects similar to the ones posted. If I was to pull every “apparentTemperatureMin” value, how would this line of the .json file change?

  "responseTemplate": "{{#daily}}{{data.apparentTemperatureMin}}{{/daily}}",

I’d think that this would do what I want, but it does not return any values:

  "responseTemplate": "{{#daily}}{{data[0].apparentTemperatureMin}}{{/daily}}",

#10

It may be relevant to someone trying to help :wink:

you could mess around with the response to your template here at trymustache.com


#11

Now that is cool! That site helped tremendously! I was able to figure out what I needed to do within about 5 minutes of playing around. To anyone looking to parse JSON data, try the trymustache.com link above!


#12

@Alligator - I’d love to hear how you solved it. I’m trying to do something similar with the below and not sure how to proceed. Note, I’m pretty much a n00b so take it easy on me :smile:

Effectively there could be 0 to n results (number of busses arriving to a given station in the next 30 minutes), and I want to pull out the results, loop through them and translate them to LEDs to light up. E.g., if # arriving in 0-1 min >1, HIGH, etc.

@LukeUSMC tutorial is awesome! I went with {{#0.varible}} to try to solve Alligator’s issue after play with mustache, but am getting an undefined result back in the particle.io dashboard - so not sure where the root cause for that lies. It may be with me setting up the .json file incorrectly to return a result.

Full details below - any help would be greatly appreciated!

{
  "event": "tfl274_hook",
  "url": "https://api.tfl.gov.uk/StopPoint/490007244N/Arrivals?app_id=<yourID>&app_key=<your_app_key>",
  "requestType": "GET",  //note I tried as POST as well
  "headers": null,
  "query": null,
  "responseTemplate": "{{#0}}{{0.timeToStation}}~{{0.lineId}}{{/0}}",
  "json": null,
  "auth": null,
  "mydevices": true
}

The data is coming back in the dashboard as ‘undefined’

The entire JSON returned from hitting the URL directly is:

{
     $type: "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
     id: "-1515226023",
     operationType: 1,
     vehicleId: "LK09EOA",
     naptanId: "490007244S",
     stationName: "Gloucester Avenue",
     lineId: "274",
     lineName: "274",
     platformName: "null",
     direction: "inbound",
     destinationNaptanId: "",
     destinationName: "Islington Angel",
     timestamp: "2015-12-06T20:21:17.874Z",
     timeToStation: 480,
     currentLocation: "",
     towards: "Camden Town",
     expectedArrival: "2015-12-06T20:29:18Z",
     timeToLive: "2015-12-06T20:29:48Z",
     modeName: "bus"
},
{
     $type: "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
     id: "1964355453",
     operationType: 1,
     vehicleId: "LK11CWV",
     naptanId: "490007244S",
     stationName: "Gloucester Avenue",
     lineId: "274",
     lineName: "274",
     platformName: "null",
     direction: "inbound",
     destinationNaptanId: "",
     destinationName: "Islington Angel",
     timestamp: "2015-12-06T20:21:37.953Z",
     timeToStation: 1078,
     currentLocation: "",
     towards: "Camden Town",
     expectedArrival: "2015-12-06T20:39:36Z",
     timeToLive: "2015-12-06T20:40:06Z",
     modeName: "bus"
}

note that to get the results in mustache, I had to add quotes to each of the items versus the JSON that came back from the tfl API, in case that matters…:

Thanks :smile:

Steve


#13

After looking at a few other things, by changing the webhook file to use:

  "responseTemplate": "{{#0}}{{timeToStation}}~{{lineId}}{{/0}}",

instead of

  "responseTemplate": "{{#0}}{{0.timeToStation}}~{{0.lineId}}{{/0}}",

I was able to get a response (yes!)

{"data":"294~274","ttl":"60","published_at":"2015-12-06T22:12:25.207Z","coreid":"particle-internal","name":"hook-response/tfl274_hook_1/0"}

Now the issue is that the data returned the result for the first bus, but the actual API returned 3 results. I’m pretty sure the #0 is the issue here, but haven’t yet figured out what to use instead.

Also, a side-note, I’m getting quite a few more items returning in the dashboard per call then @LukeUSMC shows - which may (or may not be part of my problem…)


#14

For what it’s worth, here is the code I settled on after using Mustache and just playing around until I got it to display the results I wanted.

My json file contained this:

{
  "event": "get_weather",
  "url": "https://api.forecast.io/forecast/xxxxxxkeygoesherexxxxx/42.348,-95.673",
  "requestType": "GET",
  "headers": null,
  "query": null,
  "responseTemplate": "{{#currently}}{{temperature}}{{/currently}}~{{#daily}}{{#data}}{{temperatureMax}}~{{icon}}~{{/data}}{{/daily}}",
  "json": null,
  "auth": null,
  "mydevices": true
}

And my Core code used this to process the response:

void gotWeatherData(const char *name, const char *data) {
    String str = String(data);
    char strBuffer[500] = "";
    str.toCharArray(strBuffer, 500);
    int currenttemp = atoi(strtok(strBuffer, "\"~"));
    int tempmaxtoday = atoi(strtok(NULL, "~"));
    String forecasttoday = (strtok(NULL, "~"));
    int tempmaxtomorrow = atoi(strtok(NULL, "~"));
    String forecasttomorrow = (strtok(NULL, "~"));

#15

Thanks. I still can’t figure out how to get the response to get more than one result.

If I change the json file to have:

  "responseTemplate": "{{#0}}{{timeToStation}}~{{lineId}}{{/0}}~{{#1}}{{timeToStation}}~{{lineId}}{{/1}}",

I’ll get 2 responses, but given you guys are getting multiple results without having to have multiple copies of the variables, seems like I"m doing something wrong.


#16

You can see me slowly working my way through this :smile:

The below works in http://trymustache.com/ as it returns 4 sets of data as expected, just not from the webhook… ideas??

{{#.}}~{{timeToStation}}~{{lineId}}{{/.}}

#17

@stu187 Apparently there was some kind of issue with the compiler. Try updating your particle CLI and see if that works for you. :smile:

@LukeUSMC I am stuck a bit with the code. Are you able to post your full code as an example? I think it may just be something arduino specific that I’m missing… I’d like to see how your code flows in its working form to see what I’m doing wrong.

Cheers.


#18

What are you trying to do? Can you post what you have so far? Mine is massive at this point because of all the other stuff I’m doing and wouldn’t be of much use…


#19

Actually, I figured it out. Just some deprecated names (Spark instead of Particle) and some undeclared variables. It’s been while.

Now I’m feeding my data to an LCD, trying to figure out the best way to refresh it.


#20

@LukeUSMC Hey I just wanted to say thanks for this tutorial :smiley:

You provided me with the info I needed to properly parse and format the data returned from my first local OpenWeatherMap webhook that I created using the new web based Webhook builder.

After looking over your explanation and referencing the screen shots I finally figured out how to get only the data I wanted returned from the webhook :+1:

The other code you posted is going to be useful also.

Just wanted to say thanks since this allowed me to go to bed tonight having accomplished successfully creating and parsing my own custom webhook for the first timeout. Without your help I’m pretty sure this would have taken much longer to figure out.