I am trying to get my head around a weird issue which started yesterday. Here are the symptoms:
About 1/4 of my devices stopped sending their hourly web hook. 3/4 running the same code, continued as usual
When I launched the Particle Console, I could see that these devices did wake and connect as they sent a message stating that they were going into an error state and resetting.
When they awoke, I could connect to them using the device console and instruct them to send a web hook, which they did.
So, why would a subset of my devices, wake, connect but not be able to send a web hook unless instructed by me to do so manually?
The devices cover a large area - at least twenty miles as the crow flies
I don’t know what carrier they are connected to - I use the Particle SIMs. Is there a way to tell?
I will try to get more information. From what I can see in the console, they are connecting (or so they think) to Particle, send the web hook and time out waiting for a response. They then go to error mode where they reset after 30 seconds.
They weird thing is that it is not all the devices, they we all working correctly up to yesterday and there are two code bases (one for car counting and one for people counting) so I don’t think it is in my software.
Again, any advice or - as you are doing - any information I can gather to narrow down the issue is helpful.
Update: I think there was a web hook response timeout error on Particle or Ubidots’ side. On the last hour 14/15 devices have reported in and the one that did not reported a response timeout error.
So, I checked with Ubidots and checked the Particle status page - neither reported issues. The next time this happens, I think I will try to open a support ticket with both Particle and Ubidots to see if someone can identify the issue.
I will also look at ways to more gracefully recover when Webhooks or the connection process times out.
I still have one device not connecting so any advice is appreciated.
