Webhook posting as form-data

Troubleshooting
1

I’ve created a custom webhook to post data to a database ingest agent, and it seems to be acting in unexpected ways. Specifically, it is sending data as content-type: application/x-www-form-urlencoded. Really what I need is for it to just send the data I post to the event as straight binary data.

So the CURL command I use is
curl -i -XPOST ‘http://my-server/write?db=iotdata’ --data-binary 'volume_reading value=155’
So I need to send that key/value pair but not as JSON or as url encoded data.

I don’t see an obvious way to do that. In the webhooks creation form on my Console I see that I can fill out Query Parameters but there’s no indication of what those should be.

My on-device code:
String data = String::format(“volume_reading value=%d”, range);
Particle.publish(“Log”, data, PRIVATE);
Certainly calls the webhook. and that webhook certainly posts to the server in question, it just doesn’t post the data.

2

So I never was able to solve the web hooks issue, so I solved it another way. If anyone’s interested in how I integrated a Particle Photon with InfluxDB, feel free to read about it here: https://davidgs.com/2017/running-influxdb-on-an-artik-520/

Yeah, I built InfluxDB on an ARTIK-520.

2 Likes
3

As a better solution, I wrote a Particle.io integration to InfluxData’s Telegraf data ingestion agent. It’s now an officially recognized integration from the Particle Console. Problem solved.

2 Likes
4

Thanks for letting us know!

5

Has anyone been able to follow the Tutorial on creating a Webhook to publish from a Photon to Particle Cloud to InfluxDB in AWS via a TICK stack in the AWS? There is also a webinar of the tutorial at the InfluxDB website under resources.
My test app webhook to Thingsspeak works fine, but all I get from the a Webhook implementation following the tutorial is an ETIMEDOUT error. I suspect my JSON data construction is flawed but cannot even get the Webhook TEST connection to pass.

6

I don’t have a solution, but I do emphasise with your dilemma. Without wanting to hijack your specific issue (which may or may not be the same as mine) I have a working TICK stack running on DO (I say working because I’m sending data to it successfully via non-Particle devices). I’ve worked through both the webinar and tutorial here: https://docs.particle.io/tutorials/integrations/influxdata/photon/#storing-data-in-an-influxdata-database.

There seems to be some inconsistency in whether the telegraf port (1619) is required in the endpoint URL in the webhook definition. The above links suggest http://<my_URL>/particle is sufficient in the webhook definition whereas https://github.com/influxdata/telegraf/tree/master/plugins/inputs/webhooks/particle suggests it needs to be http://<my_URL>:1619/particle. In the telegraf.conf I’ve uncommented both service_address = ":1619" (per the webinar) and [inputs.webhooks.particle] and path = "/particle". I get Error 404 using just /particle and ECONNREFUSED if I try with the port when using test in the webhook integrations.

My suspicion isn’t the JSON but rather something in telegraf. But I base that suspicion on little knowledge aside from the observation that, unsurprising, going to http://<my_URL>/particle returns error 404.

7

Thanks for sharing your experience with sending Particle cloud data to InfluxDB.
One point, per David G of Influxdata, is that the port address is absolutely required in the end point URL in the Particle Webhook. He also suggests confirming The latest TICK components are being used.
More troubleshooting ahead.

8

I’ve been digging into this for a while now, and it appears as though something has changed somewhere. The problem appears to be in the fact that Particle Web sends the data portion (which is sent from the device as a JSON as a string. It also appears as though no amount of jiggering with the Webhooks format actually changes anything in what is sent.

For instance, according to https://docs.particle.io/reference/device-cloud/webhooks/#body I should be able to define a custom body, but when I do that, it still persists in sending what looks like the default JSON body.

{
  "name": "{{{n}}}",
  "data": "{{{v}}}",
  "ttl": 60,
  "published_at": "{{PARTICLE_PUBLISHED_AT}}",
  "measurement": "iotdata"
}

should avoid html escaping the value, but it appears not to. In addition, the value sent ends up being different.

The value sent from the sensor is:

{
    "tags": {
        "id":"xxxxxxxx000000000xxxxxxxx"
        "location":"Traveler"
    }
    "values":{
        "RSSI":-76
        "temp_c":26.74
        "temp_f":80.132
        "humidity":27.709961
        "pressure":1006.729065
        "altitude":54.433502
        "broadband":10
        "infrared":1
        "lux":8
    }
}

The message sent to the Webhooks is

{
  "name": "sensor",
  "data": "{\"tags\": {\"id\": \"58004a000351353530373132\", \"location\": \"Traveler\"}, \"values\": {\"RSSI\": -74, \"temp_c\": 27.209999, \"temp_f\": 80.977998, \"humidity\": 26.309570, \"pressure\": 1006.834045, \"altitude\": 53.554543, \"broadband\": 10, \"infrared\": 1, \"lux\": 8}}",
  "ttl": 60,
  "published_at": "2019-03-08T21:01:24.243Z",
  "coreid": "xxxxxxxx000000000xxxxxxxx",
  "userid": "5617ee125ac8e5ec701a060d",
  "version": 12,
  "public": false,
  "productID": 5343
}

So it appears to ignore the body definition I have given it, as well as ignoring the desire to not html-escape the incoming JSON. If I could find a way to have that data portion not in quotes, I believe that would solve it.

Ideas?

1 Like
9

I’m a bit confused about what you actually are after.
Can provide this information to clarify?

  • the Particle.publish() code
  • a sample event string (printed via Serial.print() and not copied from console.particle.io)
  • the actual custom body definition
  • a sample (mock-up) of the output you want to have sent to the target server

BTW; html-escaping is not the same thing as escaping the nested double quotes.
The \" you are seeing in your data value is only a representation used by console.particle.io but the backslashes are not actually part of the sent body.

When I want to see what the raw request looks like I direct the request to a simple NodeJS server running on my dev machine.

For example
event publishing command

  char payload[] = "{ \"id\": 123, \"loc\": \"<here>\", \"cap\": 5}";
  Particle.publish("pd", payload, PRIVATE);

actual, outgoing event string

{ "id": 123, "loc": "<here>", "cap": 5}

custom body definition

# XML
<root>
  <tags>
    <id>{{id}}</id>
    <location>{{loc}}</location>
    <unescaped>{{{loc}}}</unexcaped>
  </tags>,
  <values>
    <count>{{cap}}</count>
  </values>
</root>
# JSON
{
  "tags": {
    "id": {{id}},
    "location": "{{loc}}"
    "unescaped": "{{{loc}}}"
  },
  "values": {
    "count": {{cap}}
  }
}

actual output on server

# XML
<root>
  <tags>
    <id>123</id>
    <location>&lt;here&gt;</location>
    <unescaped><here></unexcaped>
  </tags>,
  <values>
    <count>5</count>
  </values>
</root>
# JSON
{
  "tags": {
    "id": 123,
    "location": "&lt;here&gt;"
    "unescaped": "<here>"
  },
  "values": {
    "count": 5
  }
}

for reference, the console representation of the request

image
image984×448 14.7 KB

1 Like