This is a bit of a shot in the dark because we have not fully root-caused, but we are seeing ~15 Electron devices spread out across the Northeast US that all nearly simultaneously ‘dropped offline’ at ~ 10/05/2019 21:00 Eastern. These devices are still intermittently sending messages, but while they should be always connected (no sleep mode) and sending messages about once per minute, they started to show signs of connectivity issues (not pingable, messages coming in every ~5-10 minutes) starting 10/5.
The issues are spread across multiple states in the Northeast US. We are on an older version of Electron firmware (0.6.4), so we do not have the Device Vitals that I believe would show which cellular provider is in use. I believe the providers we are using are AT&T, TMobile, and Sprint (I don’t believe we are using Verizon).
We know that Device Integrations show that the overall # of messages forwarded through our main data ingestion integration (webhook) significantly dropped off starting at the above time 10/05/2019 ~21:00 Eastern, and also that all 15 geographically distributed devices started having an issue at nearly the same time (within minutes of each other). These have all also been running the same version of our user firmware for some time without issues. Other devices running the same firmware in other parts of the US do not seem to be affected.
I think all of the above helps us likely narrow it down to either a widespread cellular issue or a Particle Cloud issue.
While we continue to investigate, I wanted to take to community’s temperature to see if anyone else was experiencing issues over the weekend continuing to now.