Electron fails OTA update, loops with "panic, assert_failed"

Several Electrons have been working reliably in the field for years, all running on the then-current device OS 0.4.8, and identical app code. I recently had to make a minor code change and successfully flashed it OTA to all devices, and after that everything worked as it should. I then thought it would be a good idea to bring the ancient device OS up to date (ver. 2.3.0), I step I now regret. I flashed the devices again with an OS target of 2.3.0. All but one location worked, but one location (unfortunately 6 hours away by car) is now in a loop “online” >>> “panic, assert_failed” >>> etc.
Observations on the Partice console (see screenshot):

  1. (orange highlight): The device in question is now on device OS 0.6.4, which I did not explicitly request, but it may be one of the intermediate steps to get to 2.3.0
  2. (yellow highlight): The “Last Handshake” is fairly current, and “Last Heard” occurs consistently every 30 seconds or so, so the network connection can’t be all that bad.

Is there anything I can do (preferably remotely) to bring the device back online?

Judging from the timestamps, the device comes online and one second later the panic occurs, which causes the device to reset. Unfortunately once you get into a rolling reboot situation, it’s hard to get out of it remotely because the device will reboot before it can get a software update OTA.

Because you’re jumping forward so many versions there’s at least one intermediate upgrade required, which is why the device appears to be running 0.6.4 even though you upgraded from 0.4.8 to 2.3.0.

There is a way to prevent this by including rolling reboot detection in your firmware and going into safe mode, but unfortunately once it happens it’s too late to add it remotely.