I have a couple of cores out in the field that I’ve been testing for about a year with no issues. This morning, I saw that they got re-flashed with the default Tinker firmware. I have not initiated this and as far as I know it could not have been due to a coding error on my part. So I am pretty concerned about how this came to be. Has anyone else noticed this? Is it related to the system outage Particle was having this morning? Or did someone break into my account and re-flashed the devices intentionally? Either way - sounds like bad news.
On Cores in the old days, there was the user firmware you loaded and then there was the factory backup, which is Tinker by default. If the user firmware failed to many times in a row, a core would switch the factory firmware and try to be in a safe mode. Photon now has a real safe mode that has obviated the need for this. The idea in all cases is to avoid a deadly loop where you cannot reprogram the device if it fails over and over before getting the cloud.
You can control what is in the factory reset firmware slot with this CLI command:
Thanks @bko! That makes sense, though the cores have been running with my custom firmware for many months with no issues. But I wonder if the cloud outage this morning somehow triggered the fallback you’re describing? When you say “user firmware failed too many times” do you mean the device goes into the SOS mode, or just if there are too many restarts over a certain period of time?
I don’t recall the exact way that this gets triggered but if you search the source you can probably find it. I think it is something like three SOS failures within a certain time period.
For products and hard to reach installed Cores, it really makes sense to have your own factory firmware image installed “at the factory.” That was the original design intent for the Core. You could always program this using dfu-util; the CLI command above just wraps dfu-util is nice package for you to use.
Thanks @bko! I will definitely do this as I don’t ever want to revert to tinker for a production device! I didn’t know about the auto-revert function before, nor how to test it. Will need to figure out a way to simulate that condition to see my custom “factory” firmware get reloaded. And probably need to add some logic let the device “chill” for a while if it detects a flurry of restarts.
FYI, I had the same thing happen yesterday morning. The real bummer is I’ve had the same firmware on there for over a year, and now I need to figure out where I put that build/code
I am not able to get my original firmware to work on my Spark Cores. When I load them (both via OTA or over USB) it causes the “three red flashes” behavior and the device just keeps resetting forever. Interestingly, it NEVER reverts to the tinker firmware, as suggested by @bko. So, I believe the problem stems from some change on the Particle cloud. I opened a ticket with Particle support and they suggested that this may be caused by an update in the firmware library and that I should try going back to an older version. However, I am building my user firmware locally using the older version 0.4.3 of the firmware, so I am not sure that suggestion is helpful.
So, to sum up:
User firmware that was compiled locally and worked for many months is not longer working on any Spark Cores.
When loaded, it goes through normal connect cycle, but after flashing cyan, I get 3 red flashes, then the LED turns off for a bit and the device reboots. This continues indefinitely, without eventually reloading to factory firmware (Tinker)
I can do a factory reset without any problems and load tinker or other simple test user firmware.
I did have a look at the source code and the conditions under which a Core device reverts to the factory backup firmware which is typically Tinker are that the checksums are not correct for the user code area. So if you have flash corruption, you land back in Tinker.
When you say “three red flashes” is that an SOS pattern? It would be short-short-short long-long-long short-short-short followed by a number of flashes that tells you what the problem was.
No, not an SOS pattern. Just three long red flashes after rapid flashing cyan. And after that the LED goes dark and a few seconds and the Core reboots. I recall reading about this condition on one of the threads here and the poster saying it had to something to do with the Core not being able to negotiate a connection with the Particle Cloud. But this is strange to me, because flashing the device with Tinker or other simple user firmware works like a charm. I don’t need to regenerate the keys or anything like that. My custom firmware uses almost all the available memory, but it’s under the max limit as it loads fine. And it used to run with no issues for many months on several Spark Cores. Perhaps the updated Particle Cloud code requires more memory to be reserved for the Cloud connection management and this is tripping me up?
