{{SPARK_EVENT_VALUE}} gets HTML encoded

When creating a custom webhook endpoint, the value {{SPARK_EVENT_VALUE}} gets HTML encoded so the endpoint receives " instead of proper quotation marks, making JSON decoding fail.

I created a webhook with this format:

{
    "eventName": "hm_",
    "url": "http://myserver.com/device_events",
    "requestType": "POST",
    "json": {
      "name":  "{{SPARK_EVENT_NAME}}",
      "value": "{{SPARK_EVENT_VALUE}}",
      "occured_at": "{{SPARK_PUBLISHED_AT}}",
      "device_id": "{{SPARK_CORE_ID}}"
    },
    "mydevices": true
}

When I publish an event from the Spark with the data formatted as JSON, the value field posted to my endpoint is HTML encoded.

This is a raw HTTP request that was posted to my endpoint. Notice how the original data field is correctly quoted, but the value field is not.

POST /device_events HTTP/1.1
User-Agent: SparkBot/1.0 (http://docs.spark.io/webhooks#bot)
host: redacted
accept: application/json
content-type: application/json
content-length: 312
Connection: keep-alive

{"event":"hm_data","data":"{\"battery\":4.896004,\"hours\":0.733333}","published_at":"2015-06-03T18:23:24.786Z","coreid":"redacted","name":"hm_data","value":"{"battery":4.896004,"hours":0.733333}","occured_at":"2015-06-03T18:23:24.786Z","device_id":"redacted"}

The firmware code that published this event is:

String data = String("{"
    "\"battery\":" + String(battery) + ","
    "\"hours\":" + String(hours) +
    "}");
Spark.publish("hm_data", data, 60, PRIVATE);

Is this a bug in the server code?

Also, I expected that when I provide a custom “json” field in the webhook then the default fields would be omitted. What if my endpoint throws an error when receiving unknown fields? Adding an option to the webhook config file “omitDefaultFields” would allow to more completely customize the webhook request.

1 Like

Try removing the quotes around the value of “value” in the webhook:

{
    "eventName": "hm_",
    "url": "http://myserver.com/device_events",
    "requestType": "POST",
    "json": {
      "name":  "{{SPARK_EVENT_NAME}}",
      "value": {{SPARK_EVENT_VALUE}},
      "occured_at": "{{SPARK_PUBLISHED_AT}}",
      "device_id": "{{SPARK_CORE_ID}}"
    },
    "mydevices": true
}

And your code that defines data looks a little off to me. Try:

String data = String("{\"battery\": " + String(battery) + 
    ", \"hours\": " + String(hours) + "}");

Hum. This isn’t valid JSON and the particle CLI refuses it with Parse error [SyntaxError: Unexpected token {]

For the code, you just put all the string on one line but it’s really equivalent. I used string literal concatenation to break the string to multiple lines. For example: String(“a” “b”) is the same as String(“ab”). The compiler will combine both strings.

I am experiencing the same “Parser error” when trying to use the {{SPARK_EVENT_VALUE}} in a webhook without enclosing the {{SPARK_EVENT_VALUE}} in double quotes. It makes sense that the JSON is invalid because you are trying to put a string in the JSON without the double quotes. However in my case the SPARK_EVENT_VALUE is a number and I don’t want it to be a string in the JSON. Has anyone found a way to do this without wrapping the token with double quotes?

Thanks

That's a valid use case but I don't think there's a way to do that right now.

@dave, could you be of assistance here?

This is something that’s on the roadmap, but there isn’t a way to set that yet. I’m trying to find a way where the syntax isn’t awkward :\ The most obvious solution to me is to add another property, that lets you set the types of other properties, or a meta property. If you were going to add the ability to set a property that should be treated as a number (no quotes when making the request), how would you want to specify that?

Thanks,
David

How about allowing the use of Javascript conversion methods like Number() and Boolean() in the template and skipping the quotes in the generated JSON if the result is a number or a boolean.

Example:

{
    "eventName": "hm_",
    "url": "http://myserver.com/device_events",
    "requestType": "POST",
    "json": {
      "name":  "{{PARTICLE_EVENT_NAME}}",
      "value": "{{Number(PARTICLE_EVENT_VALUE)}}",
      "occured_at": "{{PARTICLE_PUBLISHED_AT}}",
      "device_id": "{{PARTICLE_CORE_ID}}"
    },
    "mydevices": true
}
1 Like

I have the same problem posting sensor data to SAP Hana Cloud Platform. The IoT Services in the free trial account do in the latest release enforce the data type for incomign sensor values. Due to this limitation I cannot post sensor values at all anymore into this server (other then if I want to deal with them as strings in the database which is not a good idea).

My suggestion is to put the whole payload into a string (with " as " inside) that can be parsed when webhook is created as such and only do the template-replacing when the the webhook is used. Errors in the syntax of the payload will materialize later this way but I assume this is something you test at least once and then forget once you got it right.

[edit] checking again i found that now quotes are only there if the incoming json had quotes in it - so issues seems to be resolved by now with some automagic.

I have the same issue. I tried publishing a raw JSON string containing numbers from my photon to the cloud. Then I used a web hook to publish to keen.io. The web hook requires quotes, which makes keen.io treat it as a string, instead of a number.

Any update on adding the ability to publish a number or pass a number through the web hook JSON?

Sorry, there’s no update at this point.

What I’ve been doing is creating a webhook to a script running on a 3rd party server like https://hook.io or Amazon Lambda if you have larger event volumes. You can massage the data as needed on the script.