Parsing web pages/data to control servos/leds?

Hi - After years of playing with Arduinos - ive just ordered a photon dev kit. Ive made a few Arduino projects in the past that scrape data from an API and as a result turn some servos or light some LEDs.

After lots of searching - My question is how best to achieve this with the photon.

Example:

1: Connect to an API
2: Parse out some data
3. Turn Servo/Light LED

Any pointers greatly appreciated

Thanks
Al

@surfershort, I suggest you look at the following doc items:

TCPClient
Webhooks

Also, do a search in the Community for the httpclient library which is available on the web IDE. RTFM is always a good start :wink:

2 Likes

Another suggestion: If the web page is large or the code to parse it is complicated string parsing, you can put the scraping and parsing on a simple online script hosting platform like hook.io and connect to your hook.io script with the Photon. For that last part you can use webhooks or the HTTP client as mentioned above.

Thanks - the API is here http://magicseaweed.com/developer/forecast-api I only need the ‘combined: {
height: 1.1,
period: 14,’

This would then turn a servo depending on the value ie: if ‘height’ is 1.2 turn servo 0 degrees, if ‘height’ is 10.2 turn servo 140 degrees etc…and refresh every 3 hours

Will the board be able to handle this or will i need to use an external hook.io,temboo etc?

I am RTFM :smile: and hoping ill figure it out…but any advice happily received

Cheers

This sounds remarkably similar to a project I’m finishing up right now :smile:

I would do the JSON munching in the cloud because you’ll be able to use convenient JavaScript syntax to get values from the hash instead of ugly C++ string code. Also, it will be a lot easier to test and update your code if it’s split in 2 parts.

Thats amazing! And yes looks like we are trying to achieve the same thing. Ill start a project now on hackster.io to show you

Hi,

Having spent the weekend learning from some great resources i seemed to have hit a wall…

Using this tutorial here i have created the webhook with the set of data i need from here

Using this json file

{
  "event": "surf_hook",
  "url": "http://magicseaweed.com/api/252522525252525252525252525/forecast/?spot_id=1253",
  "requestType": "POST",
  "headers": null,
  "query": null,
  "responseTemplate": "{{#swell}}{{#components}}{{combined.period}}~{{combined.height}}{{/components}}{{/swell}}",
  "json": null,
  "auth": null,
  "mydevices": true
}

When i publish this webhook from the cli it doesnt return any data

Where as if i use the tutorial using webhooks here its fine…ive tried comparing the json files and the strucure of the data but cant see why one returns data and the other one doesnt?? I have a lot to learn so im hoping its a simple oversight…but i cant see it

What im hoping to achieve is to take what i created Arduino WAVEbuouy here but using webhooks and the core.

Any pointers would be greatly appreciated

  Serial.begin(9600);
  pinMode(red   ,OUTPUT);
  pinMode(green ,OUTPUT);
  pinMode(blue  ,OUTPUT);
  if (Ethernet.begin(mac) == 0)
  { // start ethernet using mac & IP address
    while(true) ;
  }
  delay(2000); // give the Ethernet shield 2 second to initialize
}
void loop()
{
  if (client.connect(serverName, 80)>0) {
    client.println("GET http://www.ndbc.noaa.gov/data/latest_obs/62081.rss");
  }
  if (client.connected()) {
    if(client.find("Significant Wave Height:"))
    {//look for the wave height and pass the value to variable
      waveheight = client.parseFloat();
  }
  else{
    digitalWrite(13,HIGH);
  }
  //The code below lights up RGB module light depending on the wave height
   if ((waveheight>=0)&&(waveheight<3))
   {
     analogWrite(red   ,0  );
     analogWrite(green ,0  );
     analogWrite(blue  ,255);
   }
   if ((waveheight>=3)&&(waveheight<8))
   {
     analogWrite(red   ,0);
     analogWrite(green ,255 );
     analogWrite(blue  ,0);
   };
   if ((waveheight>=8)&&(waveheight<100))
   {
     analogWrite(red   ,255 );
     analogWrite(green ,0);
     analogWrite(blue  ,0);
   }

<img src="//discourse-cloud-file-uploads.s3.dualstack.us-west-2.amazonaws.com/business7/uploads/particle/original/2X/b/bda905de3d87ffc50e27282e9b517b03726abcc0.jpg" width="690" height="307">

@surfershort, did you notice that the API you are hitting allows you filter only the fields you need:

If you don’t need all the information available in your response you can specify only specific fields or groups of fields like this:

http://magicseaweed.com/api/YOURAPIKEY/forecast/?spot_id=10&fields=timestamp,wind.*,condition.temperature

It would seem to me you can do something like:

"url": "http://magicseaweed.com/api/252522525252525252525252525/forecast/?spot_id=1253&swell.components.*"

This should return the entire swell.components object data. Then you can part that for just period and height.

You can still parse the way you are doing nonetheless. However, I have found that I needed to play with the responseTemplate syntax to get everything working. You may want to start with just the swell data to see if you get a response, then add components, etc. This way you can identify which filter is not working. :wink:

Webhook responses seem to have a large delay (1 to 2 minutes) right now. @Dave can you comment?

This simple test with a simple echo HTTP service has a 1 minute 25 seconds delay in the response.

particle webhook create echo https://hook.io/Marak/echo
particle publish echo

@surfershort make sure the magicseaweed endpoint returns the data you want (maybe the / after forecast is extra)
curl http://magicseaweed.com/api/XXX/forecast/?spot_id=1253

You can try a simpler webhook first: particle webhook create surf_hook http://magicseaweed.com/api/XXX/forecast/?spot_id=1253

Thanks guys

@jvanier the /?spot_id=1253 is the location for which i need the data so i guess i need to keep that in there - i tried using your simpler web-hook but it still doesn’t pull out the ‘height’ or ‘period’

@peekay123 your correct yes you can filter the data from the API which is really handy but i still cant extract that ‘height’ and ‘period’ So im using

http://magicseaweed.com/api/APIkey/forecast/?spot_id=1253&fields=timestamp,swell.,components.height,swell.,components.period

with this json

{
  "event": "surf_hook",
  "url": "http://magicseaweed.com/api/APIkey/forecast/?spot_id=1253&fields=timestamp,*swell.,components.height,swell.*,components.period",
  "requestType": "POST",
  "headers": null,
  "query": null,
  "responseTemplate": "{{#swell}}{{#components}}{{combined.period}}~{{combined.height}}{{/components}}{{/swell}}",
  "json": null,
  "auth": null,
  "mydevices": true
}

And i get this…

I guess its something to do with the template but im lost for the minute…

@surfershort, start with the high level object to see if you get that and one by one, add more granularity with extra filtering. Your approach is all or nothing right now and impossible to debug. :wink:

I meant maybe you need http://magicseaweed.com/api/APIkey/forecast?spot_id=1253

Ah, the JSON is wrapped in an array!

I don’t know how to extract the first element using responseTemplate syntax, but I’m sure somebody else does.

Does anyone have an idea on how to extract the first element…spent hours searching… :frowning:

Try
{{#0}}{{#swell}}{{#components}}{{combined.period}}~{{combined.height}}{{/components}}{{/swell}}{{/0}}

1 Like

Thanks @jvanier but still no joy :confused: I tried removing the 0 too

There is a secret way to get this:

IF the response from the server is a JSON array, then the template exposes a top-level property “body”, so you can write something like:

{{#body}}
* {{.}}
{{/body}}

I hope that helps! :slight_smile:

Thanks,
David

2 Likes

As @peekay123 mentionned, the best way to debug this is take it step by step.

I put this in a webhook JSON template surf.json (except that I used a real API key):

{
  "event": "surf_hook",
  "url": "http://magicseaweed.com/api/apikey/forecast?spot_id=1253",
  "requestType": "POST",
  "headers": null,
  "query": null,
  "json": null,
  "auth": null,
  "mydevices": true
}

Created a webhook with particle webhook create surf.json while monitoring with particle subscribe mine in one terminal I published the event with particle publish surf_hook in another terminal.

I got this in the particle subscribe mine

{"name":"surf_hook","data":"undefined","ttl":"60","published_at":"2015-12-09T03:09:28.264Z","coreid":"001"}
{"name":"hook-sent/surf_hook","data":"undefined","ttl":"60","published_at":"2015-12-09T03:09:28.266Z","coreid":"particle-internal"}
{"name":"hook-response/surf_hook/0","data":"{\"error_response\":{\"code\":119,\"error_msg\":\"Path or method restricted for your API key.\"}}","ttl":"60","published_at":"2015-12-09T03:09:28.372Z","coreid":"particle-internal"}

I then noticed that the webhook JSON you posted said to POST the request instead of doing a GET. POST is usually for submitting a form or creating a resource on a web server. GET is for retrieving a page.

I switched surf.json to:

{
  "event": "surf_hook",
  "url": "http://magicseaweed.com/api/apikey/forecast?spot_id=1253",
  "requestType": "GET",
  "headers": null,
  "query": null,
  "json": null,
  "auth": null,
  "mydevices": true
}

Deleted the 1st webhook and tried again. I got a huge amount of output

{"name":"surf_hook","data":"undefined","ttl":"60","published_at":"2015-12-09T03:10:17.851Z","coreid":"001"}
{"name":"hook-sent/surf_hook","data":"undefined","ttl":"60","published_at":"2015-12-09T03:10:17.854Z","coreid":"particle-internal"}
{"name":"hook-response/surf_hook/0","data":"[{\"timestamp\":1449619200,\"localTimestamp\":1449619200,\"issueTimestamp\":1449597600,\"fadedRating\":4,\"solidRating\":1,\"swell\":{\"minBreakingHeight\":8,\"absMinBreakingHeight\":8.2,\"maxBreakingHeight\":13,\"absMaxBreakingHeight\":12.81,\"unit\":\"ft\",\"components\":{\"combined\":{\"height\":11.5,\"period\":12,\"direction\":67.52,\"compassDirection\":\"WSW\"},\"primary\":{\"height\":11.5,\"period\":12,\"direction\":71.17,\"compassDirection\":\"WSW\"}}},\"wind\":{\"speed\":19,\"direction\":66,\"compassDirection\":\"WSW\",\"chill\":39,\"gusts\":23,\"unit\":\"mph\"},\"co","ttl":"60","published_at":"2015-12-09T03:10:17.986Z","coreid":"particle-internal"}
{"name":"hook-response/surf_hook/1","data":"ndition\":{\"pressure\":1032,\"temperature\":53,\"weather\":12,\"unitPressure\":\"mb\",\"unit\":\"f\"},\"charts\":{\"swell\":\"http:\\/\\/hist-1.msw.ms\\/wave\\/750\\/1-1449619200-1.gif\",\"period\":\"http:\\/\\/hist-1.msw.ms\\/wave\\/750\\/1-1449619200-2.gif\",\"wind\":\"http:\\/\\/hist-1.msw.ms\\/gfs\\/750\\/1-1449619200-4.gif\",\"pressure\":\"http:\\/\\/hist-1.msw.ms\\/gfs\\/750\\/1-1449619200-3.gif\",\"sst\":\"http:\\/\\/hist-1.msw.ms\\/sst\\/750\\/1-1449619200-10.gif\"}},{\"timestamp\":1449630000,\"localTimestamp\":1449630000,\"issueTimestamp\":1449597600,\"fadedRating\"","ttl":"60","published_at":"2015-12-09T03:10:18.239Z","coreid":"particle-internal"}
...

I then put the responseTemplate I mentioned above:

{
  "event": "surf_hook",
  "url": "http://magicseaweed.com/api/apikey/forecast?spot_id=1253",
  "requestType": "GET",
  "headers": null,
  "query": null,
  "responseTemplate": "{{#0}}{{#swell}}{{#components}}{{#combined}}{{period}}~{{height}}{{/combined}}{{/components}}{{/swell}}{{/0}}",
  "json": null,
  "auth": null,
  "mydevices": true
}

Tried again and got:

{"name":"surf_hook","data":"undefined","ttl":"60","published_at":"2015-12-09T03:10:57.815Z","coreid":"001"}
{"name":"hook-sent/surf_hook","data":"undefined","ttl":"60","published_at":"2015-12-09T03:10:57.816Z","coreid":"particle-internal"}
{"name":"hook-response/surf_hook/0","data":"12~11.5","ttl":"60","published_at":"2015-12-09T03:10:57.940Z","coreid":"particle-internal"}

That should do it for you @surfershort

2 Likes

@jvanier @Dave @peekay123

Thanks so much guys i really learned a lot there - especially the debugging process, i wasnt creating a new webhook everytime i altered the json file and the GET/POST thing was staring at me…

Now just to turn the servos/light LEDs based on those values…pointers appreciated but i know i have much to learn first

Thanks again and hopefully this will help others :smile:

2 Likes