All non-beta SIM card equipped Electrons offline today

I have four Electrons with the standard SIM and one with the new Beta Sim. All the standard SIM Electrons are stuck flashing green today. There was a telephony partner update yesterday night. Yet the Particle status page shows all systems operational. What is going on?

Help ticket opened as well.

Chip

Seen this few hours earlier as well.

Will bring it up to the team.

Sorry about the issue! The team is on it:

[Particle status] Investigating: Cellular devices not able to get online http://stspg.io/4707b0f2a

@kennethlimcp,

Any chance I could get a few of those beta SIMs overnighted to me just in case this drags out. :wink:

Doesn’t hurt to ask…kind of dead in the water now.

Chip

Interesting that the Beta Sim’s are still online. Must be a completely different provider.

@kmmonk Who is the beta provider if you can share.

@RWB,

When I started this string there had not yet been any posts on the Forum and the System Status page was green. I was pointing out that the Beta SIM worked as it proved to me that there must be an issue with the telecom connection since of all the devices, that was the one difference.

Since not all SIMs on the network were affected however, we still don’t know if the Beta SIMs are on a different carrier.

Still, glad the outage seems to be over.

Chip

Mine are still offline!

My guess is different particle server

@frlobo,

Wow, thought is was over. I hope you are back on-line soon.

Chip

This reminds me on why it is not a good idea to rely on third parties for final products.

It has been more than 8-10 hours down!

I am glad I have no commercial products deployed.

Hey, @frlobo.

We’re about to resolve the status page issue and start to write up the post mortem. If you’re still down, it sounds like everything isn’t up and working again. Can you PM me some details? Can you give me a list of SIM ICCIDs that aren’t connecting?

The short version is one of the carriers was performing some maintenance and it didn’t go as planned. They had to roll the changes back, and as they did severed the connection to the database that holds SIMs for us and others. Devices trying to get online during this period could connect to the tower, but not get a data connection. Devices already online would stay connected. The kicker is the rollback also booted several (hundred) devices offline, which then couldn’t connect.

So in essence, we’re seeing near normal connection rates from Electrons and yours appear to not be connecting gracefully. The carrier said it might still take time for all devices to re-establish a connection, but I’m thinking that under normal circumstances, you’d be online already.

If you can PM me with some details, we can look into the issue further.

Thanks!

JT

We feel that with a few exceptions, the cellular outage is resolved (see the status page message) and has been for the most part for several hours.

If your devices are still having trouble connecting, we want to look into it, so please let us know by pinging us on the support page: https://docs.particle.io/support/support-and-fulfillment/menu-base/. The most important detail is the ICCID (not the device ID, but we can work with that if you don’t know the SIM’s ICCID).

Thanks!

1 Like

PM sent. I am on a mobile device so difficult to fill the support page.

Well… still no service with my electron.

Re flashed default app with no luck. So I a, sure is not my firmware,
. Tried again with another antenna… no joy…

Any ideas?

It keeps resetting trying to reconnect…

@frlobo, can you flash this debug firmware: https://github.com/rickkas7/electron-clouddebug

And PM me or @rickkas7 the logs?

1 Like

You probably did this, but a full power cycle is required in certain scenarios to get the modem reset. You’d need to unplug the lipo and any other power source.

1 Like

Logs sent to both via PM.

I had a total of 6 units go out yesterday. They were just flashing green. They wouldn’t come back on without a physical power reset, no matter how long I waited.

Is there any known software type reset or code that we can implement? I’m so glad these devices aren’t in the field yet. I’m about 2 weeks away from deploying 12 of them.

@fishmastaflex did it connect immediately after a power reset?

@kennethlimcp Yes, all 6 of them did.