API outages & teaching

I recently purchased 30 Photon’s for use in my Embedded class. Only 15 students were present, but as soon as we started to use the Publish features, the Log viewer started stuttering. The more students that used the feature, the worse the performance.

Is this “how it is”, or was this just random? From the Status page, I can see that there were some issues, but our problems was MUCH worse than this graph indicates. Only 1 in 10 requests would work…

I have been experiencing slow or no responses to function calls the last few days also.

Its definitively not normal, I would say its the first time in 4months where I have been affected by the cloud being down.
Sadly the incident page didnt show anything, but hopefully automated systems triggered an alert to ops.

1 Like

Just as an additional route to investigate to exclude common pitfals:
How did the test code look? Did it heed the 1 publish per second per device rule?
What kind of connection did you use?
How bad is the radio noise without all these devices and how does it degrade with more and more devices sharing the same channel?

Heya @jenschr,

Sorry about the coincidence! We had just spun up a bank of new servers to add capacity, and we were fiddling with some load balancing settings and dealing with one account that was pushing some limits. During that window yesterday we were needing to restart some instances of our API, and deploy new code as well so there were some brief outages.

For reference, here’s a small snapshot of us adding capacity to that cluster the day before yesterday, and then during the blip on the status page. Even though there were some blips, I believe most of the traffic was being served, but I’m sorry about the timing. Yay metrics! :slight_smile:

Thanks,
David

1 Like

Hi @jenschr,

Keep in mind too that there are rate limits for how often you can publish from your device, it’s possible some devices were publishing too frequently, and were being rate limited in firmware.

Thanks!
David

Hi all,
Good to hear that it was temporary! We were just using the standard Publish-example so I sure hope we didn’t swamp the API using that :wink:

J