Webhook "hook-sent" message very slow

Troubleshooting
1

We are using Webhooks to communicate between our devices and servers, the webhooks are currently sending very slowly.

Here is a screenshot from the Particle logs showing the “hook-sent” message at 12.55pm and the response at 1.01pm

and our server logs showing the “hook-sent” message being received at 1.01pm.

Dec 22 13:01:17 app/web.1:  [info] application - e:reg5 - coreid=xxx - response time: 51ms 
Dec 22 13:01:52 app/web.1:  [info] application - s:data1 - coreid=xxx - response time: 63ms

So it looks like messages being sent via Webhooks are taking ~6 mins to get to our server but the response is pretty instant.

Any ideas what is causing this, how to fix it, how to stop it from happening in future?

It’s really important for us that we receive a response from the server within 20 secs of the Webhook being called to ensure that the response is matched with the message sent. If this is not reliable we will have to include a message id in both the sent and received messages so they can be matched but I don’t feel this should be necessary.

Thanks

1 Like
2

Hi @joe4465,

Sorry about that, ideally the hook delivery time should be much quicker. This is on my radar and something I will fix, but I might not be able to get to it until early January. It’s a byproduct of hooks being very popular, and us continuing to add capacity as demand grows. I haven’t been able to work on this recently since we’ve been quite busy with work on the Electron, but that’s one of my top priorities right after. Sorry about that!

Thanks,
David

3

Thanks for the response.

We are launching to our customers in the first week of January so we really need this solved before then.

Is there any way this is going to be possible? (extra servers on your end or workarounds on ours)

Cheers

4

@joe4465, when did you create the hooks? I had issues with a webhook and deleting and recreating it with the latest version of CLI seemed to fix the problem. It may be worth a try. :smile:

5

They were created ~2 months ago and have been getting delayed every so often but never by as much as this time. I’ll try to remove and recreate.

1 Like
6

I’m trying to wrap up some things for the electron in the next day or so, but I’ll look into it if I can. Would you send me an email at david (at) particle.io to chat more about your use case / launch timeline / expected load / etc.

Thanks!
David

7

will do.

Thanks

1 Like
8

Hey All,

Sorry about the delay here, finally got unblocked and spent some time improving webhooks. We’re also now tracking average hook delivery time (the time between when the server gets your published event, and when the hook sends a request to your server), so we can monitor this closely down the road.

Thanks!
David

9

Just in case you’re not doing this - consider tracking standard deviation and max/min in last 1/6/24 hours as well. The simple mean will not serve you well as a quality indicator.

2 Likes