OTA Issue - Endless updates consuming data at an alarming rate

Troubleshooting
2

Hi Chip,

Thanks for asking these important questions! As you know, we’re working internally to resolve this for you - and due to the nature of the diagnostic process for this issue (poring through account-specific logs in our backend), I strongly recommend anyone experiencing a similar issue to open up a support ticket.

  1. Why did Particle let the dive go over its monthly limit by 3x before pausing the SIM?

Particle works with several telephony partners to manage its Particle SIMs. Particle has realtime access to your data consumption metrics, hitting a partner-specific endpoint when a threshold is crossed to say “shut that SIM down!” Overwhelmingly, the appropriate gate closes immediately. Sometimes, however, this communication can incur some latency - this is typically not an issue given the rate of most normal, day-to-day data consumption. However, sometimes data can be consumed with such vigor that it makes this latency apparent.

  1. Why does this update work for 80% of the devices and fail spectacularly on the 20%

After taking a look in our backend, it’s clear that some of these devices are resetting in the middle of OTAs, either a) rejecting their incoming firmware or b) disconnecting due to some instruction in their firmware. I’d love to take a more concentrated look at this in the context of a support ticket.

  1. Can I remotely fix this issue and bring these devices back into the normal population?

When you mark the device as a development device, what happens? It stops receiving updates but does it go offline? Is it impossible to flash OTA to the device?

  1. What tests can I do to avoid this in the future?

A great question. I will post the results, as appropriate, of our work together here! In general:

  • be careful about the placement and duration of a device’s sleep cycle so as not to interfere with the OTA process
  • be vigilant against calling Particle.function()s and especially .variable()s with such rapidity that it exceeds a given device’s timeout window
  • be aware of blocking and how to avoid it

but we’ll develop the above as we speak further!

2 Likes