Can't Create or Delete Webhooks (Timeout)

Alright. Looks like create and delete are working most of the time. But 9 out of 10 times the response shows an error. Reloading the Integration tab show the Webhooks increasing or decreasing. So, little bug there with the timeout message.

Hey there @JethroNull – looping in @jeiden here who is responsible for webhooks on our end

Hi @will and @jeiden. I’ve been really pummeling the Webhook Builder for the last few hours. I can tell you some things for certain:

Creating and Deleting frequently fail with the “Bummer…” error popup. BUT, in most cases the create or delete actually did work. Going back to the Integrations tab and refreshing shows them there (or not). That confused the heck out of me for a while.

Also, it seems that there is a slight gap (maybe a minute) between webhooks getting created or deleted and them being actioned on (log shows the effect of deleted webhooks once or twice and new webhooks take a little while to be actioned. A small delay but that confused me too since edits in my webhook design that should have fixed my bugs did not seem to work so I kept changing the webhook when, if I’d left it a while it would have worked.

Hope this is useful info.

I really wish i had checked here last night I was having that bummer all night last night. However i noticed if i opened webhooks in a new browser it was actually deleting them. I got bummer when creating them clicked create a few times and got bummer and a few webhooks that where actually created.

Seams to take a while for the system to detect a new webhook as well.

I would like to add disable enable to webhooks to the list as well.

1 Like

I just noticed that my webhooks are not firing when called. In the past, I just deleted the existing ones and recreated them. I believe I was successful at deleting the existing ones–they don’t show up under particle webhook list.

I then tried to recreate them, but got the following message when I issues the create command: Error Webhook creation failed: subscribe error - Error: Too many open files

When I do a particle webhook list, the new ones are listed, but still not firing when called.

What’s up???

1 Like

A bit more detail, it appears webhooks are not firing in multiple accounts of mine.

@ctmorrison, my webhooks are firing this morning. How many webhooks do you have?

There is currently a known problem creating new web hooks ("Error Webhook creation failed: subscribe error - Error: Too many open files”). It’s being investigated.

2 Likes

I have 5 webhooks. I’ve made no changes to webhooks for a month or two.

When I do make webhooks, I use the CLI and have never had an issue in the past.

@ctmorrison, odd. I’ll defer to @rickkas7 on this one :wink:

@ctmorrison I am also experiencing problems - my webhooks stopped working. I am also getting the "(“Error Webhook creation failed: subscribe error - Error: Too many open files”)” when I deleted them and recreated them.

I have 2 photons and 1 core that all hit webhooks and all of them stopped reporting in this morning too.

FWIW, (although I’ve lost track of time a bit this morning) when I deleted and recreated the webhooks, they seemed to work for about 2 minutes. I only know this due to my devices publishing to the cloud every minute and I saw two updates about 7:27AM, which is about when I deleted and recreated the webhooks (I think!).

OK…webhooks are starting to fire, again. Woohoo!!!

Yup, all my offline alerts just cleared.

Hi all –

Thanks for voicing this issue, and sorry for the trouble! We are working hard to improve the stability of our webhooks service. We have been dealing with some sporadic downtime of this particular service over the last few days. Our engineering team is working hard to improve the reliability of webhooks over the next sprint.

Thanks!

Jeff

2 Likes

Hi @tyler785, how do you check if your webhooks are offline?

Hi @sazp96, I push all the data I collect to Librato. Librato can send alerts if it hasn’t received any data in X amount of time and it will keep sending alerts every X minutes until cleared.

1 Like

I’m pushing data to Grovestreams every minute and have defined latency events that alert us if data has not been received within a defined period of time. We then take a look to see if the site is having an issue (lost cell service is the most recent cause). Otherwise, I look at the Dashboard to see if the devices are online and publishing as expected every minute. It’s rarely, but sometimes, webhooks not firing.

1 Like

I am having same time out issue. In my case I can delete and recreate webhook. . But my webhook doesn’t foreward my event dats to end URL. Everything was working fine 10 days ago… @peekay123 @jeiden @jonlogan @will