Simultaneous Failure Of Multiple Electrons

Troubleshooting
1

I have 6 electrons currently in the field and two in the lab that have been working flawlessly for over a year. Over night all 8 electrons dropped off the network on 11-7-2017 between 06:42 and 09:56 UTC.

My bill last month went through with no problem and the credit card associated with the account is active and current.

The devices in the lab are both blinking green.

Over the weekend one device stopped functioning properly, so yesterday I put it in DFU mode and ran the Particle Keys Doctor utility from the CLI. This fixed the device and got it back online.

I just ran the device doctor and keys doctor on one of the devices in the lab and can not achieve a connection.

These 8 devices are on at least three different cell towers.

I’m at a total loss here, and need to restore connectivity to these devices ASAP. Any help is greatly appreciated. Thanks!

1 Like
2

Will report this now.

Likely to be a connectivity issue of the cellular provider.

1 Like
3

Seconded. We have close to 100 Electrons offline, all starting near the Nov 7 2017 7:00 AM UTC time.

4

Sorry about the issue! The team is on it:

[Particle status] Investigating: Cellular devices not able to get online Particle Status - Cellular devices not able to get online

5

Thanks for the report! The status page shows a service outage shortly after.

6

Hi, @ppatino and @ScaleFree.

Can you private message me with some details, please?

If you could send me a list of ICCIDs of affected devices (or device IDs and I can look up the ICCIDs) and where they are located approximately (country and state/province is good enough) that will help us troubleshoot the issue.

Also, are the electrons still currently powered up and trying to connect?

Thanks!

JT

1 Like
7

Coming right up. Geographically distributed outage, but I’ll get you details in a private message.

1 Like
8

Hi,

I also have a number of electrons (~30) offline that I would like back online ASAP. Geographically in Oregon and California.

Do I need to send you their details?

9

That’d be great, @hwestbrook.

I need locations and ICCIDs of affected devices.

10

@jtzemp - PM sent. Devices are in West Virginia

11

@jtzemp,

Are we sure this is a minor system outage. For those like me, it does not seem minor.

Chip

12

Outage in Oklahoma and Texas too. Sending info now. Seems like its an entire carrier outage.

13

It’s definitely not an entire carrier. Many, many devices are online still without issues.

Thanks to those who have sent over data. I’m compiling that and sharing it with the carrier to help their investigation. I think I’ve got enough for now.

We’ll post on the status page as the situation changes. I know this is frustrating. Thanks, all for your patience! We’re working on it. :slight_smile:

Thanks!

14

I’m still checking, but I think all of my Electrons just came back online.

15

Just came back online as well.

16

My units are back online as well.

17

@ScaleFree, @jtzemp,

Can we mark this as “Not Resolved”? I would think that an issue like this in a production system is not resolved until we get at least a root cause report?

18

@hwestbrook - Good call. I’ve removed the “resolved” tag.

19

Still down. I am in the state of Maine.

20

Curious what your application is for the 100 Electrons. That’s a lot of units.