Cloud JSON Parsing

@ruben, if you give me the link to the site you are querying, I can get info on their JSON format. I can’t help if I don’t know the full JSON format.

Cheers peekay,

the info is here:
http://www.wunderground.com/weather/api/d/docs?d=data/hourly

@ruben, can you give your query minus your key? I will look ate the data and see if I can build a filter for the data you want.

Thanks PK ( @peekay123)

I’ve been using the following for JSON:
http://api.wunderground.com/api/tHiSiSmYkEy/hourly/q/OH/Cincinnati.json
I have not used this but it is also available for JSON
GET /api/tHiSiSmYkEy/hourly/q/OH/Cincinnati.json HTTP/1.1

XML:
http://api.wunderground.com/api/MyKEYisTHIs/hourly/q/OH/Cincinnati.xml

Also sending you a quick PM

@ruben, here is the webhook json file with the JSON filter you need to extract the data you specificed:

{
"event": "test1",
"url": "http://api.wunderground.com/api/yourkey/hourly/q/OH/Cincinnati.json",
"requestType": "POST",
"headers": null,
"query": null,
"responseTemplate": "{{#response}}{{#hourly_forecast}}{{FCTTIME.hour}}~{{qpf.english}}~{{pop}}~{{/hourly_forecast}}{{/response}}",
"json": null,
"auth": null,
"mydevices": true
}

The filer works this way:

  • the root object is response which leads to {{#response}} to “open” it
  • the object under the root where all your data is hourly_forecast which leads to {{#hourly_forecast}} to open it
  • the first item you want is hour under FCTTIME which leads to {{FCTTIME.hour}}
  • the ~ is included in the filtered string and acts as a separator
  • the second item you want is english under qpf which leads to {{qpf.english}}
  • another ~ separator
  • the final item you want is pop which leads to {{pop}}
  • a final ~ as separator
  • The {{/hourly_forecast}}{{/response}} are used to “close” the two open objects

You are left with 36 hours of data or one response “payload” sent by the cloud. Of this data, you keep only the first 28 bytes being the next 3 hours of data. You then parse that data with the ~ separating the hourly data sets. :smiley:

5 Likes

Outstanding explanation!

In order to keep only the first 28 bytes would you recommend looking at the packet number a la the example
http://docs.spark.io/webhooks/#handling-web-responses

or should I stream the response and drop off anything after the first three using something like strtok like described here?

or is there an easier way of just counting bytes as they pass through the buffer and copying the first 28 to a char array or string? Or am I misunderstanding.

To those who don’t know, you take this text including the curly braces and copy it into a blank xxx.json file in any directory on your computer. You open your terminal (or powershell in my case), nav to the folder w the file and issue the command spark webhook create xxx.json
An example is here http://docs.spark.io/webhooks/#lets-make-some-graphs-an-example-librato-webhook

@ruben, the response will be a String payload and I realized my comment on the first 28 bytes was too specific since the values in the string may change in size.

The best approach is to parse using the strtok() approach, looking for the first 3 sets of numbers. You can use a different delimiter for the “end” item in your webhook if that helps. Note that using a comma seemed to cause problems on the webhook parsing side so avoid those. Another thing to note is that the entire return payload will start and end with double quotes ("). :grinning:

I have a problem… not sure why this would happen. Just a minute ago, wunderground said that I made 11 API calls in less than 1 minute.
My device isn’t even plugged in.
Not sure how this is possible.
Can the webhook get called when my device isn’t online? Is it somehow public?
I only shared the key with @peekay123 privately so I’m not sure how this is happening (barring testing by peekay).

Any ideas?

I think I know the reason. I created a spark webhook with the title “get_weather” like the example documentation but I changed what the webhook did. Is it possible that as other people were trying the example they invoked this? I since then changed the title but that webhook has not been called :wink:
Still, other people shouldn’t have been able to call my webhook, even if it has the same name.

@Zachary Is this a possible bug? I killed the webhook so I’ll see if this flat-lines now.

Thanks for the report @ruben, possibly so. @Dave is the creator of the webhook functionality and should know for certain.

1 Like

Hi @ruben,

Good question! When you setup your webhook, make sure you set "mydevices": true. This will tell the webhook to only react to events from your private event stream, and not other people’s events. Webhooks is still somewhat new, so it’s possible there is a bug there, can you try creating your hook with that and see if its running as you expect?

Thanks,
David

Ah. mydevices:true. I don’t recall doing that. Is that in the docs example?

if you used the cli spark webhooks GET, then I’m guessing it didn’t put the parameter in there by default, but it’s in the docs here: http://docs.spark.io/webhooks/#webhook-options-mydevices

I agree this is a bit confusing, I think maybe I’ll switch it to be on by default instead of off by default.

Thanks!
David

1 Like

:+1:

1 Like

Hello folks,

I’m trying to parse the response I get from the weather server, so that I only get the temperature back.

@peekay123, I tried to follow the response structure you describe above but it isn’t working for me:

{
"eventName": "get_weather",
"url": "http://api.openweathermap.org/data/2.5/weather",
"requestType": "GET",	
"headers": null,
"query": {
	"q": "Chicago,US",
	"units": "imperial",
	"mode": "json",
},
"responseTemplate": "{{#main}}{{temp}}~{{/main}}",
"json": null,
"auth": null,
"mydevices": true
}

Using this webhook I still get the full response from the server:

{"name":"hook-response/get_weather/0","data":"\"{\\\"coord\\\":{\\\"lon\\\"
:-87.65,\\\"lat\\\":41.85},\\\"sys\\\":{\\\"message\\\":1.6062,\\\"country\\\":\\\"US\\\",\\\"sunris
e\\\":1433499371,\\\"sunset\\\":1433553730},\\\"weather\\\":[{\\\"id\\\":802,\\\"main\\\":\\\"Clouds
\\\",\\\"description\\\":\\\"scattered clouds\\\",\\\"icon\\\":\\\"03d\\\"}],\\\"base\\\":\\\"statio
ns\\\",\\\"main\\\":{\\\"temp\\\":51.62,\\\"temp_min\\\":51.62,\\\"temp_max\\\":51.62,\\\"pressure\\
\":1008.57,\\\"sea_level\\\":1030.85,\\\"grnd_level\\\":1008.57,\\\"humidity\\\":95},\\\"wind\\\":{\
\\"speed\\\":9.85,\\\"deg\\\":10},\\\"clouds\\\":{\\\"all\\\":36},\\\"dt\\\":1433508135,\\\"id\\\":4
887398,\\\"name\\\":\\\"Chicago\\\",\\\"cod\\\":200}\\n\"","ttl":"60","published_at":"2015-06-05T13:
05:44.517Z","coreid":"null"}

Could anybody tell me what I’m doing wrong?

Thanks!

@sazp96, the requestType should be POST, not GET. Also, remove the comma at the end of the “mode”: “json” line. The response template looks good so try those change and see what happens. Please make sure you delete your old webhook and your are using the latest version of CLI! :smiley:

The switch from GET to POST worked. Thanks @peekay123!!

Why is POST the right type to use? I though GET is the type to use for retrieving information.

@sazp96, you would think but that’s now quite how RESTFUL API works. I will defer the explanation to @bko or @Moors7 who are a lot smarter than me on this topic. :smiley:

I know this isn’t a real “Particle” question but there seems to be some real knowledge of parsing html data here on the forum. Using the “moustache” format @peekay showed us, for example.

I am looking to capture sports scores. I’ve searched and searched and have found other methods like this.

Ideally, I’d like to set up a webhook that would help me bring in a specific team’s scores from ESPN, however I have no skills for that.

Anyone ever tried to do this?

I’m looking to recreate a typical MLB scoreboard… ‘live’ on a Particle-enabled display.

                R  H  E
Red Sox         5  4  0
Yankees         7  6  0
Bottom of the 7th