Blocking automatic firmware updates

@mdma – here’s what we do:
At the factory, we load the latest version of particle firmware onto the photon, fresh off of https://github.com/spark/firmware/releases We also load a binary of our user firmware onto the photon.
We’ve seen this issue in two batches of photons: one batch that we programmed with 0.4.3, and one batch that we programmed with 0.4.4 and later upgraded to 0.4.5 before shipping them out. One thing worth mentioning – with both these batches, we were loading a user binary that was compiled against firmware version 0.4.3. (I had compiled this firmware with the particle cloud compiler when 0.4.3 was the latest release, and we kept using that binary for our next couple batches)
With each of these batches, we saw the following behavior in some (but not all) of the photons:
Our product firmware is running under SYSTEM_MODE(SEMI_AUTOMATIC), and by default is disconnected from wifi. It only connects if the user switches it into "connected mode"
The firmware ran correctly under disconnected mode, and was able to connect and run properly when we programmed the photon (i.e. when the version of firmware that we loaded onto the photon was the most recent version)
Once a later release of the firmware came out, we saw a “freezing” behavior when we tried to connect the photons to wifi. We’d switch the photon to ‘online’ mode and connect the photon to our lab’s wifi. The photon properly connected to wifi, and then instantly went into a breathing purple “OTA firmware update” mode.
On a lot of these photons, the update never completed. It would simply stay in breathing purple update indefinitely. I tried restarting – same thing. I tried doing a system reset and re-adding the wifi creds, but the photon always went to breathing purple within seconds of connecting to wifi.
These photons never completed the OTA update. I left three of them running for >12 hours in our NY lab, and they were just perpetually breathing purple. Other photons took >10 minutes to complete, a la this issue.
The way that I can get the photon out of breathing purple mode is to update the system firmware to the latest release over USB. As soon as I do that, the photon functions normally.

This issue is troubling to us because it affects our newest users. As soon as someone connects their photon to wifi, it goes into this breathing purple mode. The user (often someone who is using the photon for the first time) is surprised and frustrated by this, because their product just stops working – after all, they didn’t tell it to breath purple. Why is it doing this?

I want our product to work like this:
The user connects their photon to wifi. Our code keeps running, and the user can interact with the photon through our website. If the user flashes new firmware to the photon, and the new firmware is compiled against a later version of the system firmware than whatever’s on the photon, the system will OTA update the system firmware as well as flash the new user firmware.
I also wish the OTA firmware update wouldn’t hang forever, but the most important priority is for a new user to be able to connect their photon to wifi without a confusing incident.

If there’s a better thing we can do when we flash the photons, let us know. I would also like to know what goes on behind the scenes in the OTA update logic, because this is totally opaque to me right now (what triggers the system to start an OTA update? How could we work with this so we don’t start an unexpected update?)

Thanks,
Alex

1 Like