Parsing Webhook Response

Cloud
1

Hi all,

I am finally looking into some advanced cloud functionality. One thing I need to do is have my controller reach out and fetch data from an external API. I’ve heard of people doing this with weather stations.

What I need is to parse the returning JSON and update some variables on the particle controller. I know and have tested how to use a published event to trigger the webhook… what I can’t figure out how to do, can’t find a decent beginer tutorial for, is how to have the particle cloud parse the JSON and update cloud vairables.

Additionally, it would be cool if I could set up the cloud to run the webhook at an interval and have the data pushed down to the device, rather than having the device have to handle the timing.

If anyone knows where to find a good tutorial, please let me know. This seems to be one of those things where the info is only good if you already know how to use it!

Thanks!

2

Cloud variables are not updated by the webhook/cloud but by the device that registered them.
The webhook can filter your JSON response and forward a reduced result event to which your device would subscribe to in order to further process it (e.g. update your local variables).

Since webhooks are mainly meant for the device to request specific data and only relay these requests that is not a feature which is likely to be implemented.
Furthermore, since the webhook is only useful when the device is online (which is in many real-life use-cases not always the case) it would be difficult to sync.
All that is perfectly solved with the way webhooks work now.

If you provide some specifics about your needs we might be able to assist.

3

Hey, thanks for the amazingly quick response. Also, I’ve been seeing you answer a bunch of stuff I’ve looked up in the last couple days with some libraries, so thank you for all your work.

OK, I don’t mind triggering from the device. I am already doing that to send updates to my smarthome.

The API I want to hit is to grab a bus prediction. Sadly, the structure on these can change for no good reason, but I’ll deal with that when I get there. I did have some code running to parse the entire JSON down on the device, but that wasn’t efficient and it was hard to update.

Here’s an example of the JSON:

URL

http://webservices.nextbus.com/service/publicJSONFeed?command=predictionsForMultiStops&a=sf-muni&stops=N_OWL|6695&stops=N_OWL|7447

RESPONSE

{
   "predictions":[
      {
         "message":[
            {
               "text":"Muni for essential trips only. Masks or face coverings are required. Practice physical distancing. Visit sfmta.com/COVID19 for more info.",
               "priority":"Low"
            },
            {
               "text":"Predictions may be unavailable. Buses every 30 minutes",
               "priority":"Normal"
            },
            {
               "text":"Text COVID19SF to 888-777 for official updates. Visit sfdph.org",
               "priority":"Low"
            }
         ],
         "agencyTitle":"San Francisco Muni",
         "routeTag":"N_OWL",
         "routeTitle":"N-Owl",
         "direction":{
            "title":"Outbound to Ocean Beach",
            "prediction":[
               {
                  "isDeparture":"false",
                  "minutes":"13",
                  "seconds":"791",
                  "tripTag":"9341488",
                  "vehicle":"8660",
                  "affectedByLayover":"true",
                  "block":"9797",
                  "dirTag":"N____O_N00",
                  "epochTime":"1588674469947"
               },
               {
                  "isDeparture":"false",
                  "minutes":"43",
                  "seconds":"2591",
                  "tripTag":"9341490",
                  "vehicle":"8873",
                  "affectedByLayover":"true",
                  "block":"9799",
                  "dirTag":"N____O_N00",
                  "epochTime":"1588676269947"
               },
               {
                  "isDeparture":"false",
                  "minutes":"73",
                  "seconds":"4391",
                  "tripTag":"9341489",
                  "vehicle":"8887",
                  "affectedByLayover":"true",
                  "block":"9798",
                  "dirTag":"N____O_N00",
                  "epochTime":"1588678069947"
               }
            ]
         },
         "stopTitle":"The Embarcadero & Brannan St",
         "stopTag":"7447"
      },
      {
         "message":[
            {
               "text":"Muni for essential trips only. Masks or face coverings are required. Practice physical distancing. Visit sfmta.com/COVID19 for more info.",
               "priority":"Low"
            },
            {
               "text":"Predictions may be unavailable. Buses every 30 minutes",
               "priority":"Normal"
            },
            {
               "text":"Text COVID19SF to 888-777 for official updates. Visit sfdph.org",
               "priority":"Low"
            }
         ],
         "agencyTitle":"San Francisco Muni",
         "routeTag":"N_OWL",
         "routeTitle":"N-Owl",
         "direction":{
            "title":"Outbound to Ocean Beach",
            "prediction":[
               {
                  "isDeparture":"false",
                  "minutes":"9",
                  "seconds":"589",
                  "tripTag":"9341488",
                  "vehicle":"8660",
                  "affectedByLayover":"true",
                  "block":"9797",
                  "dirTag":"N____O_N00",
                  "epochTime":"1588674267679"
               },
               {
                  "isDeparture":"false",
                  "minutes":"39",
                  "seconds":"2389",
                  "tripTag":"9341490",
                  "vehicle":"8873",
                  "affectedByLayover":"true",
                  "block":"9799",
                  "dirTag":"N____O_N00",
                  "epochTime":"1588676067679"
               },
               {
                  "isDeparture":"false",
                  "minutes":"69",
                  "seconds":"4189",
                  "tripTag":"9341489",
                  "vehicle":"8887",
                  "affectedByLayover":"true",
                  "block":"9798",
                  "dirTag":"N____O_N00",
                  "epochTime":"1588677867679"
               }
            ]
         },
         "stopTitle":"Townsend St & 4th St",
         "stopTag":"6695"
      }
   ],
   "copyright":"All data copyright San Francisco Muni 2020."
}

What I want to get from this is going to be:

predictions[0].direction[0].prediction[0].seconds
and
predictions[1].direction[0].prediction[0].seconds

I am a little worried because I am not 100% sure that the seconds will be associated with the correct bus… that is, it might be that these come in different orders. Which means I might have to match one of the tags (routeTag=N_OWL) or direction. However, I believe that I can change the URL parameters so that I get a simplified version of this with just the route I want. But it would be nice to only have to set up one webhook to parse all the data… unless it’s easier to set up multiple hooks and handle them separately.

Lemme know if this is helpful. (XML is also available for this)

4

direction doesn't appear to be an array tho'

Would something like this do?

predictions[*].direction.prediction[0].seconds

which should render this

[
    "429",
    "227"
]

The downside of this is that you don't know which prediction rendered which seconds value, but do you need to?
Although I'm not sure the Particle JSON parser understands wildcards for the array.

However, this would be another possible approach

[
  {"dir": predictions[0].direction.title, "sec": predictions[0].direction.prediction[0].seconds},
  {"dir": predictions[1].direction.title, "sec": predictions[1].direction.prediction[0].seconds},
]

which should render something like this

[
    {
        "direction": "Outbound to Ocean Beach",
        "time": "429"
    },
    {
        "direction": "Outbound to Ocean Beach",
        "time": "227"
    }
]
5

I’m not sure I understand… are you talking about parsing the data on the device?

Oh, and the directions. This one kills me. The API sucks in that if there is no prediction for a second direction, it’s not an array. BUT if you happen to send it a URL for a stop that has multiple directions, it’s an array. I have to have an exception handler in the JSON parse code for it. Ugly and took me forever to figure out.

I just found the “mustache” guide. I think that’s what others have mentioned, parsing it in the cloud with those {{{}}} codes.

What about {{{predictions.0.direction.prediction.0.seconds}}}? If I put those into a response topic, would that not work? Does the response template send the reformatted JSON down for me?

But I am having a hard time even understanding what is sent down to the device in what format with those.

I think I need to play with it.

6

Providing the response template is valid, then yes.

It will just send a string that is composed according to your response template where any of the {{{...}}} fields will be substituted with the actual contents.
That string can be a JSON string or a CSV or whatever else you prefer for parsing on the device.

7

OK, maybe I am wrapping my head around it then.

Last question… I see that I can test the webhook, but can I test the parse? Or do I have to use a device to see the parsed and formatted response?

8

OK, I was able to test the mustache parsing with an online parser, and it looks like I might be in good shape. I will play around with it. Am I right in assuming I don’t have to format my template as JSON? I could just grab the value and send that down?

Thank you for all your help.

1 Like
9

Yup, that's what I meant with this

10

Right! When I read that, I was under the impression that formatting it as

template: { var1: "{{{predictions.0.direction.prediction.0.seconds}}}" }

Would update a cloud variable ‘var1’. Now I get it.

Like I said, the documentation makes perfect sense… if you already understand the general workflow.

Again, thank you for all this help.

1 Like
11

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.