Howdy particles!
Alex here, from L3D cube. We make a LED cube with a photon for a brain, and we’re seeing a problem with the automatic firmware updates.
When we program the photons at the factory, we load on a binary of our compiled code, but we don’t claim the photon or associate it with a particle account. We put the programmed photon into our product and ship it out to our customer.
The customer opens up the product and turns it on. By default, our code runs in SYSTEM_MODE(SEMI_AUTOMATIC) and doesn’t connect to the internet unless the user flips a switch to tell it to go online. The cube runs ok, and then after a little while, the user wants to load some new code onto the photon, so they tell the system to go online, enter their wifi credentials via the app, and then…
BREATHING MAGENTA! As soon as the photon connects to the cloud, it sees that there’s a new version of firmware available (there’s at least a month delay between when a photon is programmed at our factory and when it gets into a user’s hands, and the Particle engineers, bless their hearts, are releasing new firmware updates at least every month). The photons often get stuck for several minutes or longer in breathing magenta update mode. The user freaks out and resets the photon, but as soon as it reconnects to the cloud, it starts breathing magenta again. At this point, we receive an angry tech support email.
I want to find a way to disable the auto-update feature on the photons. It tends to affect our most novice users, the first time they’re doing something new (connecting the photon to the cloud). Is there a flag I can set in firmware that disables it? How do other people, particularly people who use photons in the core of their products, deal with this issue?
@mdma – this is actually really hard to do accurately, because the systems that are having this issue are ones that we were trying to upgrade the firmware on. We eventually upgraded the firmware over USB, but this wiped out the old version number.
Still, my question is more general – OTA firmware updates are freezing a significant number of our customers’ photons the first time they connect them to the internet. We’re seeing this on firmware versions 0.4.3, 0.4.4 and 0.4.5 Is there a way to prevent automatic OTA updates the first time a photon is connected to wifi?
There should be no automatic updates if system firmware has already been installed. The only time the system will attempt to update is if the user flashes an application that requires a newer version of system firmware than has already been installed.
Of course, that doesn’t mean you’re not seeing updates, just that we don’t expect them to happen from the information given. If you can walk me through the programming process you are using I can try to replicate the scenario.
@mdma – here’s what we do:
At the factory, we load the latest version of particle firmware onto the photon, fresh off of https://github.com/spark/firmware/releases We also load a binary of our user firmware onto the photon.
We’ve seen this issue in two batches of photons: one batch that we programmed with 0.4.3, and one batch that we programmed with 0.4.4 and later upgraded to 0.4.5 before shipping them out. One thing worth mentioning – with both these batches, we were loading a user binary that was compiled against firmware version 0.4.3. (I had compiled this firmware with the particle cloud compiler when 0.4.3 was the latest release, and we kept using that binary for our next couple batches)
With each of these batches, we saw the following behavior in some (but not all) of the photons:
Our product firmware is running under SYSTEM_MODE(SEMI_AUTOMATIC), and by default is disconnected from wifi. It only connects if the user switches it into "connected mode"
The firmware ran correctly under disconnected mode, and was able to connect and run properly when we programmed the photon (i.e. when the version of firmware that we loaded onto the photon was the most recent version)
Once a later release of the firmware came out, we saw a “freezing” behavior when we tried to connect the photons to wifi. We’d switch the photon to ‘online’ mode and connect the photon to our lab’s wifi. The photon properly connected to wifi, and then instantly went into a breathing purple “OTA firmware update” mode.
On a lot of these photons, the update never completed. It would simply stay in breathing purple update indefinitely. I tried restarting – same thing. I tried doing a system reset and re-adding the wifi creds, but the photon always went to breathing purple within seconds of connecting to wifi.
These photons never completed the OTA update. I left three of them running for >12 hours in our NY lab, and they were just perpetually breathing purple. Other photons took >10 minutes to complete, a la this issue.
The way that I can get the photon out of breathing purple mode is to update the system firmware to the latest release over USB. As soon as I do that, the photon functions normally.
This issue is troubling to us because it affects our newest users. As soon as someone connects their photon to wifi, it goes into this breathing purple mode. The user (often someone who is using the photon for the first time) is surprised and frustrated by this, because their product just stops working – after all, they didn’t tell it to breath purple. Why is it doing this?
I want our product to work like this:
The user connects their photon to wifi. Our code keeps running, and the user can interact with the photon through our website. If the user flashes new firmware to the photon, and the new firmware is compiled against a later version of the system firmware than whatever’s on the photon, the system will OTA update the system firmware as well as flash the new user firmware.
I also wish the OTA firmware update wouldn’t hang forever, but the most important priority is for a new user to be able to connect their photon to wifi without a confusing incident.
If there’s a better thing we can do when we flash the photons, let us know. I would also like to know what goes on behind the scenes in the OTA update logic, because this is totally opaque to me right now (what triggers the system to start an OTA update? How could we work with this so we don’t start an unexpected update?)
Could you send me the firmware image that is flashed to your photons so that I might try to replicate the issue? The expected/desired behavior that you described is how the system is intended to work, but the behavior you’re seeing is clearly different.