Webhooks SLOW! Why?

Troubleshooting
1

I have a webhook that send published data to my server. But the data getting to my server is five minutes after it was published. Why?

2

I’ve been having the same issue since I started using webhooks a few weeks ago. Responses will often take a minute or two to arrive and it sometimes takes up to four or five minutes. At first I thought it might the the server I was requesting the hook from but I’ve since tested with other servers and the delay is still happening. Any comms from the particle team about the performance of the webhook server would be greatly appreciated!

3

Hey guys, I have been seeing the same slow response behavior.

A few things:

  • This wasn’t always the case, the issue started about a month or so ago.
  • Webhooks seem to respond fast after the Photon is flashed (or restarted, I haven’t tested this). After running for some time, then they “slow down”.

@Dave have you seen an increase in the response time of webhooks on your side?

Thanks!

4

Hi @sazp96,

Hmm, if restarting your photon makes a difference, then I would suspect you’re hitting some event rate limiting in your firmware? The cloud webhooks service has been getting lots of traffic lately, but the status events back from the service are meant to help determine where any slowness might be. (event being sent / heard, target web server, etc). How often are you triggering your hooks, how fast are you expecting a response, how big is the response being sent?

It could just be one of the servers is being loaded down and it’s time to scale out horizontally again, I’ll have more time to look into this once the Electron is shipped. Sorry about the delays! :slight_smile:

Thanks,
David

1 Like
5

Thanks for the quick response @Dave

I believe each Photon is whiting its limits:

  • Publishes every 5 min to each webhook (there are 3 of them)
  • It expects a response withing 20 seconds
  • If it doesn’t get a response, it publishes again every = 20 seconds * (Attempt Number) ^2. The first re-try is exactly 20 seconds after the first publish.
  • The response size is small because I’m using response template. So it only sends to the Photon a string of about 40 characters. Having sad that, the response from the server to Particles servers is way bigger (i.e. weather response from openweather map, and directions from Google Maps).

One more thing, the slow downs are cyclical. At some points the webhook responds within the 20 seconds and at some other times of the day it doesn’t. Here is a graph with 5 days of data.

trend.JPG1073Ă—588 42.2 KB

The requests doesn’t seem to get lost though. Once the webhook responds, I get a response for all the requests I did.

What do you think?

3 Likes
6

That is super helpful! I’m guessing it’s that particular worker on that box being under heavy load, sorry about that! Sounds like it’s on me to tweak some things cloud-side.

Thanks!
David

2 Likes
7

I’m glad I could help!

1 Like
8

Thanks everybody. Such a great community.

2 Likes
9

Thanks to all. To clarify: I was testing at one publish a minute, and the data was a ~30 character string. I was receiving the webhook about once a minute, but the data was five minutes old.

1 Like
10

Hey All,

Sorry about the delay here, I’ve fixed some issues with webhooks and I’ve added a stat to help track the average hook delivery time. I’m testing that metric now and when I’m confident it’s accurate I’ll add it to our status page to help diagnose and catch this if it happens again.

Thanks!
David

2 Likes