Particle API Down?

Both my Electron and Photon are blinking blue this morning. Seems like the latency has also gone through the roof on status.particle.io. Is this a side effect of the database maintenance or another issue possibly?

Regards,

Drew Westrick

My devices also cannot connect, and the product console seems to be down as well. Hopefully this is a short term issue caused by the move to the new pricing model and console.

Same problem here, devices are stuck in rapid blinking blue mode. Last successful connection was an hour ago.

They were in maintenance, which has ended BTW.

Aug 2, 2016
Planned database maintenance on the billing system
Completed - We have completed the updates to the system! All services should now be working normally

Aug 2, 06:37 CDT
Update - The increase in API traffic was due to several overzealous users abusing the API that just happened to coincide with the deploy. We’ve temporarily black listed them for the time being. It was a bit of a red herring, but we’re back on track now and should be finished soon. :slight_smile:

Aug 2, 05:44 CDT
Update - We’re seeing a slight increase in api.particle.io latency and elevated error rates for device handshakes related to recent code changes, we are currently investigating.

Aug 2, 03:03 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 2, 00:01 CDT
Scheduled - We are continuing the previous maintenance window.

I’ve got a connection now on my Electron but still blue flashing on my Photon. Sounds like maybe more than just the billing system was possibly being updated. Must have something to do with the line “related to recent code changes”.

There does appear to be an issue. Many devices that were previously online have stayed online, but Photons are having trouble getting past fast flashing cyan when making a new cloud connection. Investigations are underway.

Yeah no luck with the photons or the product console, also just clicking around the support link is broken and one of the store links points to the sales contact form, could be a long day :wink:

Same is true of all of the Electrons on my desk (9 of them). Rapid flashing cyan, logging shows “device has come on-line” but no publishes ever succeed. Sometimes it switches to red rapid flashing instead, but usually not, just lots of cyan rapid flashing.

Thanks for the info on the Electron. There were reports of Photon issues and I reproduced the Photon connection problem (rapid flashing cyan, with an occasional red flash), but I hadn’t gotten around to testing an Electron making a new connection yet.

1 Like

http://status.particle.io <-- :particle:

thanks for that, I was looking for that link when I started clicking around, unfortunately their status and my modules status seem to be in disagreement!

I guess the status only shows things that fail, whereas we have things that never succeed. :frowning:

All my Photon’s are currently down as well, rapid blue blinking. I expect this is related to the new console stuff, but I can’t use that either, as it constantly gives me an error about agreeing to the TOS. It doesn’t seem to let me accept the TOS, even though I have.

Anyone from Particle care to share?

Yea I cant seem to get anything either but the update feed says however that all services should be working now…

Except for the “overzealous users”. Maybe that’s us and we’ve all been blacklisted?
And I just managed to flash my new firmware… :cry:

I hope not. I had one active device doing 1 publish a minute, nothing else cloud related. If I got blacklisted, someone is doing this with a really bad script.

The new console dashboard replacement seems to be very slow, and not useful right now. I expect they are having some significant issues with this upgrade. Not critical to me now, I had wanted to debug some new code, but I guess I’ll have to wait.

Can anyone mention some Particle Staff members?. I can’t find any user names

There’s an employee in this topic already, and they’re aware of the issue, so no need to tag them any further :wink:

1 Like

@zachary is the CTO, he must know what is going on :wink: Otherwise poke the chief @zach

Edit;

Update - One of our database providers is having performance issues which is causing increased latency for the device services as well as the API.
Aug 2, 09:22 CDT

Investigating - Many devices Photons/Cores aren’t able to re-connect after disconnecting from the device service. We’re investigating.
Aug 2, 09:06 CDT

Folks, again, they’re aware of the issue and working on it: http://status.particle.io
Poking these folks is going to do little other than distract them from getting to the issue. Please give them some time to look into this, the more they can focus on the issue at hand, rather than responding to tags, the sooner we’ll be up and running again.

1 Like