Webhook custom templates not working as expected

tiagonmas · January 4, 2016, 7:13pm

Hi,
I am successfully connecting a particle device to Azure but the data that is reaching Azure Event hub is not in the format I want.
As described in https://docs.particle.io/guide/tools-and-features/webhooks/ I created a custom template providing the Json that I want to be forwarded to Azure, but basically the webhook ignores the template. The data sent to azure is the same as I had not included a json format in the webhook.

The webhook definition I am using

{
  "event": "tasJson",
  "url": "https://Myservice.servicebus.windows.net/particle/messages",
  "requestType": "POST",
  "json": {
	"payloadtype":"{{payloadtype}}",  
	"pin":"{{pin}}",  
    "value": "{{value}}",  
    "timestamp": "{{SPARK_PUBLISHED_AT}}",
	"guid":  "{{SPARK_CORE_ID}}"
   },
  "azure_sas_token": {
     "key_name": "Send",
     "key": "MYKEY"
  },
 "mydevices": true
}

What is sent is:

"data":"{\"payloadtype\": \"pinvalue\",\"pin\": \"D0\",\"value\": \"4\"}","ttl":"60","published_at":"2016-01-04T19:06:26.569Z","coreid":"55ff6f066678505530381667","name":"tasJson"}

this is not the format I requested.
If the webhook is created without the Json part, the result is exactly the same.

{
  "event": "tasJson",
  "url": "https://Myservice.servicebus.windows.net/particle/messages",
  "requestType": "POST",
  "azure_sas_token": {
     "key_name": "Send",
     "key": "MYKEY"
  },
 "mydevices": true
}

data that is sent using this webhook

{"data":"{\"payloadtype\": \"pinvalue\",\"pin\": \"D0\",\"value\": \"4\"}","ttl":"60","published_at":"2016-01-04T19:08:58.593Z","coreid":"55ff6f066678505530381667","name":"wsa1"}

Code that is being used in the firmware:

        // publish the event that will trigger our Webhook
        data=String("{\"payloadtype\": \"pinvalue\",\"pin\": \"D0\",\"value\": \"" + String(count[D0]) + "\"}");
        success = Particle.publish("tasJson", data, 60, PRIVATE);

Am I doing something wrong or custom templates are not working ?
here is what is shown on the dashboard. Why the “undefined” data ???

bennettj1087 · January 5, 2016, 2:38am

There was an update to the particle cli in the last couple weeks. Have you updated/installed the cli recently? If not, update by running npm update -g particle-cli. Then delete and recreate your webhook and try it again. Let me know how you make out.

tiagonmas · January 5, 2016, 9:12am

Thanks for your reply.
After updating the CLI, deleting and recreating the webhook with a custom template definition detailed above, I still see the following message on the dashboard (not following the template defined):

{"data":"{\"payloadtype\": \"pinvalue\",\"pin\": \"D0\",\"value\": \"1\"}","ttl":"60","published_at":"2016-01-05T09:10:00.117Z","coreid":"55ff6f066678505530381667","name":"tasJson"}

tiagonmas · January 5, 2016, 2:01pm

After further testing I noticed that the custom template does in fact change the data as it is sent to Azure. But it does not change what is shown in the dashboard, so it creates confusion.

Are is my test:

The webhook:

{
  "event": "simultemp2",
  "url": "https://myservice.servicebus.windows.net/myhub/messages",
  "requestType": "POST",
  "json": {
	"deviceid":"{{deviceid}}",  
	"temperature":"{{temperature}}",  
    "timestamp": "{{SPARK_PUBLISHED_AT}}",
	"guid":  "{{SPARK_CORE_ID}}"
   },
  "azure_sas_token": {
     "key_name": "<mykey_name>",
     "key": "<mykey>"
  },
 "mydevices": true
}

The firmware code:

// publish the event that will trigger our Webhook
snprintf(payload, sizeof(payload), "{ \"deviceid\":\"%o\", \"temperature\":\"%f\"}", 1,temperatures[1]);
Serial.println(payload);

Data from event as shown in particle dashboard:

 {"data":"{ \"deviceid\":\"1\", \"temperature\":\"-0.285078\"}","ttl":"60","published_at":"2016-01-05T13:54:20.237Z","coreid":"54ff71066678574917500867","name":"simultemp2"}

Data that arrives in Event Hub:

[{"event":"simultemp2","data":"{ \"deviceid\":\"1\", \"temperature\":\"-6.007269\"}","published_at":"2016-01-05T13:09:33.8610000Z","coreid":"54ff71066678574917500867",
"deviceid":"1","temperature":"-6.007269","timestamp":"2016-01-05T13:09:33.8610000Z","guid":"54ff71066678574917500867",
"EventProcessedUtcTime":"2016-01-05T13:19:43.1145430Z","PartitionId":3,"EventEnqueuedUtcTime":"2016-01-05T13:09:34.1120000Z"}

The dashboard should show the data exactly as it was sent to Azure and it does not!!!

bennettj1087 · January 5, 2016, 2:06pm

I was just going to ask that exact question. I wondered whether the dashboard was showing the correct information.

In your first post you asked why the dashboard shows “Undefined” for the data in the “hook-sent” entry. I don’t know the answer to that question, but I do know that my webhooks behave the same way. The “hook-sent” entry always says “Undefined” under data. It seems likely that the “hook-sent” event is probably a separate thing you can subscribe to if you would like to monitor it separately.

Glad the end result is working for you. Perhaps someone from the dev team can shed some light on why the dashboard doesn’t display all of the data that is being sent. @Dave, thoughts?

tiagonmas · January 5, 2016, 2:11pm

Despite the fact that the custom template does in fact change the data that is sent to Azure, it is strange to see that data is sent duplicate! All custom fields are still included in the data field … so a lot of spared space by sending duplicate info…

ex from case above, from what arrives to event hub:
“data”:"{ “deviceid”:“1”, “temperature”:"-6.007269"} … “deviceid”:“1”,“temperature”:"-6.007269"

tiagonmas · January 5, 2016, 2:19pm

It would be great to have a way to directly connect from the device to Azure as discussed here and these problems would not exist …

bennettj1087 · January 5, 2016, 2:26pm

I was going to suggest using the http/https client libraries but I see that was suggested/tried in the thread you linked to without much success. The webhook framework is great for retrieving data from web APIs (having an intermediary that can parse out just the data you care about makes the code on the Photon far simpler) but I can see your point that it provides little value for sending data to a service like Azure.

I wish I had another suggestion but I think you’ve successfully found the limits of my experience with these devices.

Dave · January 14, 2016, 5:01pm

Hi @bennettj1087,

Sorry about the slow response! At the moment the hook-sent is more just an event letting you know the hook was triggered and a request was sent, but it doesn’t also publish what data was sent to the target url.

@tiagonmas good question! You can remove that duplicate default info with the parameter "noDefaults": true.

There will always be a little bit of work when connecting between services, I think connecting cloud-to-cloud is easier imho, and allows things to be more secure.

I hope that helps!

Thanks,
David

endlessFancy · March 20, 2019, 5:57am

Hi,
I have the same issue to populate a json post to google docs. When sending the webhook to Requestbin instead I see that the json is not populated by the information from the device publish() function json string of identical format. Any recomendations?

ScruffR · March 20, 2019, 7:18am

Since this isn't your thread, we don't know anything about your setup.

how does your webhook definition look?
how does your Particle.publish() command look?
what does the particle cloud receive from your publish command?
...

Provide as much context to your question as possible to save others from guessing or the need to inquire.

endlessFancy · March 20, 2019, 7:27am

Hi! Thanks for the fast reply!

I did read the comment in this thread : WebHook with more then one value

And managed to solve my issue by following Rickkas7 example.

What solved my issue was of the following:

Migrating from usind the web console to using the Particle CLI (fantastic tool btw) for seting up my webhook.
My data buffer in the snprintf call was a bit short so I made it longer.

Now it works perfect. I dont know exactly what made the difference.

Some code for other users to see:
On the particle Electron:

            char data[1000];
            snprintf(data, sizeof(data), 
                 "{\"airPressure\":\"%f\", \"PhonePressure\":\"%f\",\"StartUpTime\":\"%f\",\"LocalTimestamp\":\"%s\"}",
                 p_airSupplyPressure_F32,p_phonePressure_F32,t_startUpTime_F32,Time.timeStr().c_str()); 
                 
            /* Send to cloud */
            s_sendIsSuccess_B = Particle.publish("measurementDone",data, PRIVATE);

My weebhook with the webadress removed:

{
    "event": "measurementDone",
    "url": "https://script.google.com/macros/REMOVED/exec",
    "requestType": "POST",
    "json": {
    	  "airPressure": "{{airPressure}}",
		  "PhonePressure": "{{PhonePressure}}",
          "StartUpTime": "{{StartUpTime}}",
          "LocalTimestamp": "{{LocalTimestamp}}"
    },
    "mydevices": true,
    "noDefaults":true
}

Topic		Replies	Views
Custom JSON with Azure Integration? Cloud	2	1356	August 17, 2017
Particle->Azure Event Hubs Webhook with Publishers URL format Troubleshooting	1	1497	April 7, 2016
Webhook is ignoring my template Troubleshooting	7	4278	December 23, 2015
Particle - Azure Webhook [Answered] Cloud	2	1383	July 17, 2015
Sending device diagnostics with a webhook Cloud	2	543	July 1, 2019

Webhook custom templates not working as expected

Related Topics