Sending coreid using webhooks (without default data)

Hi all

I’m trying to send a webhook to AWS using

sprintf(dataString, "{\"VGE\": %lu, \"PWR\": %lu}",voltage,power);

and:

Particle.publish("data",dataString,PRIVATE);

The webhook gets through allright, however the template mapping I’m using in API Gateway gets confused with the backslash escape characters.

I’ve seen some others using the following technique when creating the webhook like so:

{
    "eventName": "particleEvent",
    "url": "api_endpoint.com",
    "requestType": "POST",
    "json": {
        "voltage": "{{VGE}}",
        "power": "{{PWR}}",
    },
    "noDefaults": true,
    "mydevices": true
} 

This works, however I’d like to still send my coreid (Device ID) through. I realise I could do this by setting noDefaults to false, however this seems a bit inefficient as then the webhook is sent with two versions of the voltage and power data. Is there any way I can include the coreid without sending the default data? @ScruffR?

Do you really want the event name to be dataString?
Would you not rather use

Particle.publish("particleEvent", dataString, PRIVATE);

And I’ve also read, that for one member adding a blank after the opening and before the closing curly braces made a difference.
The next thing was the “need” to have your values wrapped in double quotes in your outgoing JSON.

But just recently there was a possible workaround mentioned by @jvanier to use body instead of json to allow for non-double-quoted values.

I’m not sure if you want to do it that way, but you can always add the device ID to the JSON.

Sorry, I’ve edited my publish code, I was indeed using the same method as you’ve described.

I don’t quite understand your comment regarding the need to have your values wrapped in double quotes?

I should have mentioned also, I’m trying to avoid sending the deviceID as part of the dataString to save data (using an electron) thought that since the deviceID was available as part of the webhook response I shouldn’t have to.

My bad, I misunderstood your actual question, hence what I said doesn’t apply to your use case, since you said (and I missed at first), that this setup already works as is.
But other users had problems with their target server expecting a JSON template that provides numeric values in the key-value-pair without the double-quotes (which seems no issue for your target).
And hence a workaround like this would be a solution

But for the device ID, I’d have to refer you to @Dave
Also about this, since I can’t really see the reason for that

Where would that extra version come from?

Looking at my API logs, if I include the default data in the webhook I get:

{
"event": "particleEvent",
"data": "{\"TIME\": 1485845337, \"VGE\": 239717, \"PWR\": 0}",
"published_at": "2017-01-31T06:49:01.449Z",
"coreid": "2e00XXXXXXXXXXXXXXX3635",
"userid": "5558XXXXXXXXXXXX53001fdc",
"fw_version": "1",
"time": "1485845337",
"power": "0",
"voltage": "239717",
"from": "particle"
}

So we’re essentially including the time/power/voltage data twice?

Ah, I missed the documentation sorry I just needed to use {{PARTICLE_DEVICE_ID}} as in:

{
"eventName": "particleEvent",
"url": "api_endpoint.com",
"requestType": "POST",
"json": {
    "voltage": "{{VGE}}",
    "power": "{{PWR}}",
    "deviceID: "{{PARTICLE_DEVICE_ID}}"
},
"noDefaults": true,
"mydevices": true
}
4 Likes

@ScruffR I realised I actually do have the problem that others were having now in regards to needing an integer instead of a string :frowning:

I couldn’t get your solution to work:

{
  "event": "Data",
  "url": "https://example-endpoint.com",
  "requestType": "Post",
  "headers": {
    "Content-Type": "application/json"
  },

 "body": 
"{ \"charge_voltage\": {{VGE}},
    \"charge_power\": {{PWR}},
    \"time\": {{TIME}},
    \"deviceID\": {{PARTICLE_DEVICE_ID}}
}",
"mydevices": true
}  

Is this what you meant?

In another post, @Dave mentioned a possible fix,although I’m not sure if it’s been implemented? Webhooks with JSON arrays or nested objects escaping quotes

Since this was an “unreflected” quote from a post by @jvanier, I’d have to leave it to him (or @Dave) to jump in.
Julien seemed to be successful with that workaround.

What do you get with that template?
It shouldn’t matter, but how about changing Post to POST and Data to data?

1 Like

The device Id is not an integer. It’s a string.

As for the other values, which may need to be integers, why can’t you handle that on AWS’s side?

2 Likes

I’m reworking the webhook docs to add more advanced examples.

The short version is that right now using the “json” key in the webhook config only allows string values. There’s a workaround to send integers or booleans using the “body” key that I’m documenting. I should post those docs by the end of today.

1 Like

Thanks @harrisonhjones, I realise the device ID is a string, that’s ok. Since I’m using API Gateway and a simple mapping template to wrap the dynamoDB API there isn’t any smarts in the middle to do the conversion. Perhaps I could store the data as a string and deal with it later but I thought if I could find a solution to this it would be good

For your specific use case you can do this:

{
    "eventName": "demo",
    "url": "https://example.com/hook",
    "requestType": "POST",
    "headers": {
    	"Content-Type": "application/json"
    },
    "body": " { \"voltage\": {{VGE}}, \"power\": {{PWR}}, \"deviceID\": \"{{PARTICLE_DEVICE_ID}}\", \"time\": \"{{PARTICLE_PUBLISHED_AT}}\" }",
    "mydevices": true
}

In firmware:

    String data = String::format("{\"VGE\": %lu, \"PWR\": %lu}", voltage, power); // or sprintf if you prefer
    Particle.publish("demo", data, PRIVATE);

This is the webhook called:

POST /hook HTTP/1.1
Content-Type: application/json
User-Agent: ParticleBot/1.1 (https://docs.particle.io/webhooks)
host: example.com
content-length: 111

 { "voltage": 1234, "power": 6789, "deviceID": "3f002b00________________", "time": "2017-01-31T19:31:19.562Z" }

On way to debug this stuff is using the netcat utility nc and create an HTTP tunnel to your machine with ngrok.

# In one terminal
$ ngrok http 3000
# In another terminal
$ nc -l -p 3000

Create the webhook with the host displayed by ngrok, publish the event from firmware and see in netcat if the format is as expected

Edit: @rickkas7 mentioned that it’s massively simpler to just use RequestBin

3 Likes

Interestingly when I tried using your exact example, requestbin showed the rawbody as:

 event=demo&data=%7B%22TIME%22%3A%201485895385%2C%20%22VGE%22%3A%20239326%2C%20%22PWR%22%3A%200%7D&published_at=2017-01-31T20%3A43%3A12.063Z&coreid=2e004XXXXXXXXXX3635&userid=55580d1bXXXXXXXX3001fdc&fw_version=1

Content type is still shown as application/x-www-form-urlencoded in the received header which seems unusual.

Here are the steps I used:

{
    "eventName": "testevent",
    "url": "http://requestb.in/1jb2s2h1",
    "requestType": "POST",
    "headers": {
    	"Content-Type": "application/json"
    },
    "body": " { \"voltage\": {{VGE}}, \"power\": {{PWR}}, \"deviceID\": \"{{PARTICLE_DEVICE_ID}}\", \"time\": \"{{PARTICLE_PUBLISHED_AT}}\" }",
    "mydevices": true
}
  • Create the webhook with the CLI
    particle webhook create hook.json

  • Publish a test event from the command line (this is to make it easy to reproduce. Normally you’d publish from your firmware)
    particle publish testevent '{"VGE":10,"PWR":20}' --private

  • See the request in JSON format on RequestBin

I’m creating the webhook through the console as I’m testing on a product (which I think also means particle publish testevent… won’t work for me either)
Copying your code in however (and just changing eventName and url) doesn’t seem to send the correct Content-Type:

I wonder if this is an anomaly from product webhooks being in ‘beta’
Perhaps I should just use a lambda function instead of template mapping and parse the data on input…

1 Like

Ok so it’s not a problem with product webhooks but a difference in behavior when creating the webhook via CLI vs console. I tried creating a non-product webhook via console and still had the same problem. I then followed your process @jvanier by creating it manually via CLI and observed the desired output as you did. Can anyone else replicate this?

1 Like

Thanks for clarifying that this is for a product webhook created on the console. I see the issue. At the moment it’s not possible to create a webhook with a custom “body” parameter through the console. I’ll bring this up internally.

3 Likes

As a workaround for the CLI not creating product webhooks at the moment, I created this simple script. I haven’t tested it with the body workaround, but since it just passes up the JSON unmodified, I think it should work.

2 Likes