This is a bit of a shot in the dark because we have not fully root-caused, but we are seeing ~15 Electron devices spread out across the Northeast US that all nearly simultaneously ‘dropped offline’ at ~ 10/05/2019 21:00 Eastern. These devices are still intermittently sending messages, but while they should be always connected (no sleep mode) and sending messages about once per minute, they started to show signs of connectivity issues (not pingable, messages coming in every ~5-10 minutes) starting 10/5.
The issues are spread across multiple states in the Northeast US. We are on an older version of Electron firmware (0.6.4), so we do not have the Device Vitals that I believe would show which cellular provider is in use. I believe the providers we are using are AT&T, TMobile, and Sprint (I don’t believe we are using Verizon).
We know that Device Integrations show that the overall # of messages forwarded through our main data ingestion integration (webhook) significantly dropped off starting at the above time 10/05/2019 ~21:00 Eastern, and also that all 15 geographically distributed devices started having an issue at nearly the same time (within minutes of each other). These have all also been running the same version of our user firmware for some time without issues. Other devices running the same firmware in other parts of the US do not seem to be affected.
I think all of the above helps us likely narrow it down to either a widespread cellular issue or a Particle Cloud issue.
While we continue to investigate, I wanted to take to community’s temperature to see if anyone else was experiencing issues over the weekend continuing to now.
Sorry to hear you’re having these issues. Since there’s little the community can do in this case, do send an email to support, and mention the deviceIDs so they can have a closer look.
Thanks! Definitely. It’s been useful a couple of times in the past to see community posts highlighting issues spread out across users, but it doesn’t look like it’s a widespread problem for others this time.
@marekparticle / @d.chauhan the issue resolved itself (without intervention by us) yesterday Oct 9 at ~3:55 EST. We have an open Support ticket but the support team did not communicate that there was a known issue. All devices that were affected and had poor connectivity all started having problems at nearly the same time, and their problems resolved at nearly the same time, so it seems that a cellular issue like today’s could be a culprit.
Marek - it’d be awesome to hear if an investigation reveals this was perhaps a similar issue similar to the one showing up on status today, or a Particle Cloud issue. We’ve PM’d the device IDs to support but I will forward them to you as well so you can look at them in tandem with these other ones.
Similar to @d.chauhan we are on an older version of firmware, 0.6.4. Support team seemed to be confused about why we are not on the latest firmware version. We’d love to be on it and are hard at work on getting there but it’s not a trivial lift and testing needs to be very thorough (we have hundreds of devices running critical firmware across the US).