I have a webhook that has been working for weeks but has been failing all weekend. Do you guys need to reset the server again? cc: @Dave
I’ve seen this issue in the past where the server runs out of memory or something and you guys just did a reset…
This couldn’t possibly be an issue on the side you’re interacting with? What shows up in the Particle dashboard?
I don’t think it could be something on my end. I haven’t changed the webhook in a few months.
I updated the firmware on my device (with no changes to how I was calling the webhook) and now it isn’t working. I can see the webhook being called when I run particle subscribe mine but it doesn’t get a response.
I don’t have the organization dashboard so I don’t think I can see any info on the webhooks… Is that what you’re talking about?
I am personally always very suspicious when I notice something like this. "It worked. I changed something. It doesn't work." Unless that so happens to exactly coincide with something going bad on the server end, I'd double check to make sure it really isn't anything on your end. Seems unlikely the servers faulted at the very moment you changed your code.
Do you by any change still have the old code, so you could test it with that? I know you said you didn't change anything about the webhooks, but it doesn't hurt to try? Have you updated your device to the latest version? Again, shouldn't make a difference, but won't hurt to check either.
I was referring to the 'normal' dashboard, since it'll show you a pretty interface for your SSE and webhook responses. The CLI should work equally fine, albeit a bit less pretty.
Any way you could test the endpoint manually, using postman for example, to make sure that hasn't gone awry?
Ok, I’ll try updating the photon and dig into it more on my end.
I was just thinking it was something to do with the server because I have spent days debugging in the past only to find out that there was an out of memory error or something on the webhook server. There is a thread on that somewhere here when that happened…
What’s the URL to the dashboard you’re talking about? I only know of dashboard.particle.io and I don’t see any webhook stuff there.
In any case, we’ve already pinged Dave, so he’ll most likely look into it when he gets a chance.
I’m not saying there’s nothing wrong, but if there were, more people would probably be affected. The fact that it stopped working after you changed something is another contributing factor that leads me to believe it might be something on your end.
dashboard.particle.io, that’s the one. It’s ‘just’ a pretty interface for SSEs, so you can see those, and thus the webhook responses, easily. There’s no hidden webhook option I’m aware of, yet.
Hey all!
I haven’t received any alerts on hooks, and I just checked the services again and they appear to be operating normally. I also checked your hooks @jlkalberer but I don’t see any rate limiting or other error responses. Can you post a sample of the events you’re getting back, or your firmware or anything and maybe we can help debug?
Thanks,
David
2 Likes
Yeah, can do.
I checked the dashboard and the webhooks seem to be working.
My server seems to be working fine as I can hit the endpoint with a get request.
I’m not getting any response back but you can see the source here. The webhook source is there as well as the call to it Particle.publish("tappt_initialize", (const char *)0, 10, PRIVATE); in KegeratorState.cpp
I’ll investigate more tomorrow.
Hi @jlkalberer,
Sorry hooks were giving you trouble, I’ve kicked one of the services which I hope will help things run smoothly again. Despite most hooks being delivered normally, I think that something must have been jammed up for some people. I’ll be digging into that and working on reproducing the error and building a fix as soon as I get a chance.
Thanks!
David
Hey @Dave,
I woke up this morning and it was working.
Is there any way that you could add a setting to toggle the output in the dashboard to show more debug info? I know it would take more resources but it would be really nice to see the complete HTTP request and server response as well whatever other information the webhook has so I could debug this easier.
Anyways, thanks for looking into this. I know finding the root cause to things like this when you are scaling out services is a pain.
Hi @jlkalberer,
Sure thing, glad it’s working for you again! That’s an interesting idea, and something that I think would really help debugging and setting up a hook. I’ll think about how we could make something like that happen. 
Thanks,
David