Webhooks are timing out when attempting to create

I am trying to register a webhook through the spark-cli and it is providing a timeout message. I have resorted to trying the provided web hook definition as in the documentation. Still the same error.

Exact output from spark webhook create “Something went wrong, server said: Timed out”

Kind of urgent. Thank you

@voodootikigod they are looking in to a fix. I brought this issue up a few hours ago in the webhook discussion

Hi @voodootikigod,

I’m sorry I didn’t see this earlier! Looking into this again now, please ping me directly if you need a workaround in the meantime.

edit: Hey guys, I think this was a heartbeating issue between two services, I bounced those services and I’m fixing the heartbeat issue. Webhook creation should be working now.

Thanks,
David

Thanks @Dave it all seems to be working now

1 Like

I got one create in, then timeout again… Any help?

@Dave halp!!!

@dave Just started hitting this issue today on 3/22 as well. Webhook creation requests are timing out.

Hey All!

Sorry about the webhook creation timing out! I think that was related to us hitting some limits on the box it was hosted on. We upgraded it to a much larger box earlier this week, and we’ll continue to scale it horizontally as demand increases. I’m also adding more monitoring during this and the next sprint hopefully, so we’ll catch this type of issue faster if it comes up again. Should be happier now in the meantime. :slight_smile:

Thanks!
David

@Dave I’m seeing this again today.

I was having trouble receiving responses from a webhook (taking around a minute for response) so I tried deleting/creating the hook again. I was sending the request every 10 seconds I had the TTL set at 5 seconds on these requests (API says this doesn’t work right now) so I’m assuming that all my requests queued up and started causing issues on your server.

Deleting/re-creating the webhook fixed my problem in the past but today I am getting a timeout when I try to create.

1 Like

Also having the issue

Cool. I wasn’t sure if it was just an issue on my end. Hopefully they can reboot the service or get this fixed today…

I did some pretty good testing. Usually it something on my end :slight_smile: I was attempting to add another hook that sent a publish event to our new server. It slowed my previous hooks way down. So I deleted the new hook and now none are coming through. I have tried two different devices, the publishes are visible when subscribing to my devices on the CLI and I sent a POST via a different REST client to my server and received the appropriate response

Yeah… so I had a bunch of the ‘create’ requests fail with a timeout and now I did particle webhook list and it showed me 11 hooks so I guess they still created…

Anyways, it looks like everything is working again. I just need to delete all these extra hooks.

Hi All,

Sorry about this, I was traveling today, and the alerting system didn’t detect the webhook issue. One of the boxes became unhealthy, and no alert was generated so the on-duty team didn’t notice it right away. I’ve restarted the unhealthy node, and added extra alerts and auto-recovery steps in the meantime until we identify the root cause. Sorry about that! Should be back to normal now.

Thanks,
David

I still see this problem today (particle webhook create blah.json => server timeout)

Same here - server times out:

$ particle webhook GET place_weather https://api.forecast.io/forecast/key/lat,long
Sending webhook request { uri: ‘https://api.particle.io/v1/webhooks’,
method: ‘POST’,
json:
{ event: ‘place_weather’,
url: ‘https://api.forecast.io/forecast/key/lat,long’,
deviceid: undefined,
requestType: ‘GET’,
mydevices: true },
headers: { Authorization: ‘Bearer num’ } }
Something went wrong, server said: Timed out