I am trying to register a webhook through the spark-cli and it is providing a timeout message. I have resorted to trying the provided web hook definition as in the documentation. Still the same error.
Exact output from spark webhook create “Something went wrong, server said: Timed out”
I’m sorry I didn’t see this earlier! Looking into this again now, please ping me directly if you need a workaround in the meantime.
edit: Hey guys, I think this was a heartbeating issue between two services, I bounced those services and I’m fixing the heartbeat issue. Webhook creation should be working now.
Sorry about the webhook creation timing out! I think that was related to us hitting some limits on the box it was hosted on. We upgraded it to a much larger box earlier this week, and we’ll continue to scale it horizontally as demand increases. I’m also adding more monitoring during this and the next sprint hopefully, so we’ll catch this type of issue faster if it comes up again. Should be happier now in the meantime.
I was having trouble receiving responses from a webhook (taking around a minute for response) so I tried deleting/creating the hook again. I was sending the request every 10 seconds I had the TTL set at 5 seconds on these requests (API says this doesn’t work right now) so I’m assuming that all my requests queued up and started causing issues on your server.
Deleting/re-creating the webhook fixed my problem in the past but today I am getting a timeout when I try to create.
I did some pretty good testing. Usually it something on my end I was attempting to add another hook that sent a publish event to our new server. It slowed my previous hooks way down. So I deleted the new hook and now none are coming through. I have tried two different devices, the publishes are visible when subscribing to my devices on the CLI and I sent a POST via a different REST client to my server and received the appropriate response
Yeah… so I had a bunch of the ‘create’ requests fail with a timeout and now I did particle webhook list and it showed me 11 hooks so I guess they still created…
Anyways, it looks like everything is working again. I just need to delete all these extra hooks.
Sorry about this, I was traveling today, and the alerting system didn’t detect the webhook issue. One of the boxes became unhealthy, and no alert was generated so the on-duty team didn’t notice it right away. I’ve restarted the unhealthy node, and added extra alerts and auto-recovery steps in the meantime until we identify the root cause. Sorry about that! Should be back to normal now.