Hi everyone, My name is Matthew, and I am a Customer Support Engineer here with Particle. I understand that there have been some concerns around some unexpected behavior this past weekend, and I wanted to give you all an update on this. Late last week, one of our caching systems unexpectedly crash…

Hey folks – because this thread is a high priority issue for our engineering team right now, we’re going to move discussions of unrelated issues into parallel threads to prevent confusion. Thanks!

Update from Particle, 5/7 at 3:30pm PDT: The engineering team at Particle has spent the majority of the day investigating this issue to identify a root cause for devices that are still unable to reconnect to the Cloud. Although the caching system that crashed has been successfully restored, there…

Update from Particle, 5/8 at 12:00pm PDT Our engineering team believes it has identified a Cloud-side resolution strategy that will successfully restore Cloud connectivity for many devices that are inaccessible for physical power reset. The team has successfully tested this solution in our stagin…

Update from Particle, 5/8 at 1:30pm PDT Our Cloud-side solution has proven effective in staging and production environments, and we have deployed the fix to our production servers. Customers with devices that are affected by this endless session restore loop should see devices come back online on…

Thanks everyone for your patience as we worked through developing and deploying a fix. Our next step is to follow up with affected customers directly to make sure that the issue has been resolved and to issue refunds for any runaway data consumption that occurred as a result of this bug. We will als…

Update from Particle, 5/8 at 3:00pm PDT We have multiple confirmed reports from customers citing successful resolution of the reconnect loop issue as a result of the fix that was deployed. We will continue to monitor the situation and work with customers who reported connectivity issues to ensure…

Update from Particle, 5/10 at 6:30pm PDT We've been working today on a detailed postmortem that we plan to publish to both the status page and the blog. We're currently targeting publishing it early next week.

Update from Particle, 5/14 at 11:00pm PDT Sorry for the delay folks. The postmortem is tentatively written, but there is an even smaller subset of devices still having issues, and we're continuing to debug. I don't want to publish until we know the full story. We're currently targeting publishing…

Updates for the Status Incident of May 3rd

Particle Updates

Topic		Replies	Views
Device Cloud incident postmortem: sleepy devices stuck using expired sessions Particle Updates	3	1244	May 17, 2019
Over 20 Devices just went down Troubleshooting electron	66	7345	May 9, 2019
[Resolved] Outage! Particle cloud down Cloud	24	1648	December 12, 2020
Todays cloud outage Cloud	20	1108	January 21, 2021
Particle Cloud Outage - 8th Oct 2018 Troubleshooting	4	699	October 9, 2018

Updates for the Status Incident of May 3rd

Related topics