[External Issue, Resolved] Particle Cloud down - any updates?


#1

There seems to be a problem with Particle services regarding cloud connectivity. I see nothing on the status page. Can anyone provide insight? I have ~100 devices down all at once, and the only problem is that they can’t reconnect to the cloud (my mqtt server is fine and connected).


#2

Update - appears to only be Electron devices. Issues present at minimum verified on system fw v0.6.4 and v0.7.0


#3

Devices seem to be performing enough of the handshake to update the last handshake value in particle console, but connection is not persistent


#4

Update: could be related to an issue with a 3rd party sim provider, Hologram. From their status page:

Some issues with packet loss.

https://status.hologram.io/incidents/98gk56hd4ccv

Not sure why this would affect Particle Cloud connectivity when it doesn’t seem to affect any other part of our connectivity.


#5

Particle SIM seemed to work ok, so looks like an issue with our third party provider that somehow is affecting only the Particle Cloud data on our connection.


#6

Since the cloud needs to know a return path to your device, which is maintained via a regular UDP hole punching mechanism which is controlled via the Particle.keepAlive() setting you have to set along with the 3rd party cellular credentials.

As it seems your other communication is originating from your device (either active sending or regularly requesting data).


#7

Yeah, that’s true. I do have a lot of two-way traffic on my MQTT connection, but that’s all TCP. The weird thing was that I was generally able to connect and maintain an MQTT connection (with some intermittent issues, so clearly affected), but the Particle Cloud connection would fully connect, and then drop within 5 seconds (so significantly less than the keepAlive ping I had set at 30 seconds).

I’m sure there is some kind of explanation that will come out in their post-mortem, but I would have expected it to at least connect sometimes, and stay connected for at least a single keep alive interval. It seemed to always successfully handshake enough to update the last handshake time, yet would always immediately disconnect.

Clearly not a Particle root cause, though I do wish I better understood why that connection would behave in this way to better understand how to prepare to better handle other similar external issues when they arise.