System issue? Webhooks failing but status.particle.io says everything is green

Troubleshooting
21

Hi @Dave ,
How did you go, was an issue found?

I’d also like to confirm if the two wildcard hosts above are now whitelisted? They’re the common AWS http gateways, the full fqdn is brittle as its new each time you publish.

Thanks!

1 Like
22

Hi @mterrill,

We haven’t pursued wildcard domain whitelisting yet, but there hasn’t been widespread need for it yet. I’ll create an issue to track this feature suggestion though, since I agree that it will be useful long-term. :slight_smile:

Thanks!
David

23

So in the meanwhile I should send my explicit urls?

I don’t really see it as a feature but as a design bug that doesn’t make sense. Your system is rate limiting innocent targets, the issue I’m pointing to is folk behind iaas providers.

24

Ps, my main question was in regards to webhook failing and then only working after being deleted. Ie critical issue that apparently I’ll be paying for on a per device basis plus platform subscription.

25

Hi @Dave,

I experienced the issue again. This time I didn’t delete the unresponsive webhook, I created a new one.

I also sent a note to support as you may be away:

Persistent issue with webhooks that Dave was looking into (System issue? Webhooks failing but status.particle.io says everything is green) but has not resolved.
Again had the issue where a photon issuing a webhook (id 56fe736a3eb1d4781cd3d156) that would not return a response (observed via subscibe mine). Manually going to webhook destination worked.
Instead of deleting and re-adding (which in the past fixed the issue), I added a new one (573e71d5d1dd39e4066b3bc1) and changed the device firmware code. It worked. It also means the old one can be observed/tested.

I’d appreciate if this is fixed, its been months and could easily mean my customers receiving their product next month would have their devices fail silently due to the particle webhook bug.

26

even stranger, sending to the v3 hook is translated as a v2 response!

{"name":"gettargetsv3","data":"null","ttl":"120","published_at":"2016-05-20T02:39:16.242Z","coreid":"3f0035001347343432313031"}
{"name":"hook-sent/gettargetsv3","data":"undefined","ttl":"60","published_at":"2016-05-20T02:39:16.285Z","coreid":"particle-internal"}
{"name":"hook-response/gettargetsv2_3f0035001347343432313031/0","data":"true,299,244,287,130,195,240","ttl":"60","published_at":"2016-05-20T02:39:16.415Z","coreid":"particle-internal"}

it must be figuring out webook name async on the reply url…

27

Hi @mterrill,

The hooks event filter is a prefix filter, so anything “starting with” the event name you specify will match. In this case you have a hook set to “gettargets”, and another set to “gettargetsv2”, and “gettargetsv3”. So the two latter events will also trigger your initial hook.

Are you seeing hook-sent events every time, and just a missing hook-response event?

Thanks,
David

28

Hi Dave, I believe it’s just the hook-sent per Apr 2 logs, i.e. I didn’t notice anything different. No hook-response or error or logs at the destination.

I wasn’t aware of the more generic hook subscribe in my code, can check tonight.

29

Hi @mterrill,

If you’re seeing “hook-sent” events, then that means your webhook is valid and working correctly. If you’re not getting hook-response events back, it’s because firebase, or whatever you’re hitting isn’t putting anything in the http response body.

If that’s the case, you could try to work around the issue by: setting your responseTemplate to always include some value before the body, or you could try hitting your service with a similar request via curl and see under what circumstances it stops returning a value?

I hope that helps!

Thanks,
David

30

Hi,

There are no AWS HTTP gateways logs when the particle cloud hangs. Running the particle webhook destination from my cli works. Deleting the particle webhook or making a new one results in successful (and naturally logged) requests.

If it was only firebase and I couldn’t validate that no request had apparently egressed particle cloud then I’d not be pursuing this for weeks. Pretty experienced at this kind of thing, 99% likelihood that it’s in the particle cloud, and if it’s not then I’ll buy you a beer next time I’m in SF.

1 Like
31

Hmm, okay, interesting!

Lets get @bryce and @jeiden involved to help track down this issue, and maybe it’ll be us buying you a beer. :slight_smile:

Thanks,
David

32

Hi @Dave, how did you go?

I haven’t had any issues for the past 2 weeks, aside from a weird glitch the other day where it froze for about 8 webhook calls and then came good, and an issue where someone had changed the timeout value somewhere in the webhook chain to 1000ms.

BTW, I sent a support request via the webform for ‘https://r6g4i5ujf9.execute-api.us-west-2.amazonaws.com’ to be whitelisted. I still think its a great idea to whitelist .execute=api..amazonaws.com to avoid support effort :wink:

1 Like
33

Hi @mterrill,

Glad to hear that things have been mostly stable for you. Feel free to ping @bryce if you notice issues, he’s been working on the service recently and will be able to answer questions about it for now.

Thanks!
David