Cellular Devices Having Trouble Sending Webhooks Today

Troubleshooting
1

This morning, I noticed that my Cellular devices are not able to send Webhooks to Ubidots today. I am trying to get a handle on where the problem is.

Here is a snippet from the console:

44%20PM
44%20PM1143×288 41 KB

Usually, these reporting events go like clockwork but, this morning is is broken for about half of my devices. Specifically, I have never seen the error message “Sleeping, too many errors, please wait and try again”

I know that one source of error is when there is more than a 5 second delay between the web hook and the response from Ubidots. It does not look like this is an issue based on the Ubidots Status page for today.

21%20PM
21%20PM930×312 21.4 KB

Here is another snippet from the Particle console which shows that some web hooks are being responded to (code 201) and others are timing out.

30%20PM
30%20PM691×374 12.3 KB

So, to summarize:

  • No change in the code
  • This process happens every hour and, before this morning, is very reliable
  • Both Ubidots and Particle show “green” system status
  • Some devices are able to complete a web hook and some are not

What do you suggest for next steps in troubleshooting?

Thanks, Chip

2

@chipmc,
I don't have any Ubidots webhooks, but I've seen that error when the ThingSpeak API is overloaded or having problems. Today, I've had 10k successful webhooks (zero errors) to ThingSpeak from SouthEast USA......if that helps any.

1 Like
3

OK, so it turned out that this was a service disruption that prevented Ubidots from responding within Particle’s 5 second (?!) response window. They were able to resolve the issue relatively quickly.

Here are my lessons learned:

  1. When using Webhooks, you need to look at both the sender (Particle) and the receiver (in this case Ubidots).
  2. The Status pages for the two services are an imperfect indicator in troubleshooting. Both Particle and Ubidots never showed any issues.
  3. If the service disruption is critical, reach out to support and check the forums to see if anyone is seeing the same problem.
  4. If all of the following is true it may not be better not to react when these events occur:
  • Issues like this are infrequent and of short duration
  • Your vendors monitor their service and are not relying on you to bring these issues to their attention.
  • Your software is written so you don’t lose any data if it misses a connection attempt
  • You are confident that it is not your fault (no software updates or changes to the hardware)

I spent a fair bit of time on this issue today and, in the end, I don’t think my efforts made any difference. Perhaps the best I can do is try to have this be the case every time there is a disruption - and there will always be more disruptions.

Thanks, Chip

3 Likes