OTA causing LTE device to stay in Safe Mode

Firmware
1

I have had multiple devices out in the field become stuck in safe mode when the automatic update happened from OS 1.0.1 to OS 1.1.0. The updates failed and are now in a state that cannot be recovered from unless the battery is unplugged. The customers are not able to do this. The other workaround is to manually downgrade the firmware back to an earlier version and keep it at that OS version.

Does anyone know why failed auto updates would cause this to happen?

For particle, I have an ID that can be looked at for history. Below are the particle stream events

|spark/status|auto-update|2019-05-15 20:09:54|
|---|---|---|
|spark/flash/status|failed |2019-05-15 20:10:28|
|spark/device/diagnostics/update|{"service":{"device":{"status":"unreachable"},"coap":{"round_trip":null},"cloud":{"uptime":328,"publ|2019-05-15 20:15:16|
|spark/status|auto-update|2019-05-15 20:17:11|
|spark/flash/status|failed |2019-05-15 20:17:38|
|spark/device/diagnostics/update|{"device":{"power":{"battery":{"charge":100,"state":"charged"},"source":"VIN"},"system":{"uptime":70|2019-05-15 20:17:41|
|spark/status|auto-update|2019-05-15 21:09:23|
|spark/device/diagnostics/update|{"device":{"power":{"battery":{"charge":100,"state":"charged"},"source":"VIN"},"system":{"uptime":73|2019-05-15 21:09:25|
|spark/flash/status|started |2019-05-15 21:09:27|
|spark/flash/status|success |2019-05-15 21:11:17|
|spark/device/last_reset|user|2019-05-15 21:11:22|
|spark/device/diagnostics/update|{"device":{"power":{"battery":{"charge":100,"state":"charged"},"source":"VIN"},"system":{"uptime":1,|2019-05-15 21:11:22|
|spark/status/safe-mode|{"p":10,"imei":"redacted","iccid":"redacted","m":[{"s":16384,"l":"m","vc":30,"vv"|2019-05-15 21:11:24|
|spark/device/app-hash|redacted|2019-05-15 21:11:24|
|spark/flash/status|started |2019-05-15 21:11:25|
|spark/flash/status|success |2019-05-15 21:12:01|
|spark/device/diagnostics/update|{"device":{"power":{"battery":{"charge":100,"state":"charged"},"source":"VIN"},"system":{"uptime":3,|2019-05-15 21:12:10|
|spark/device/last_reset|user|2019-05-15 21:12:09|
|spark/status/safe-mode|{"p":10,"imei":"redacted","iccid":"redacted","m":[{"s":16384,"l":"m","vc":30,"vv"|2019-05-15 21:12:11|
|spark/flash/status|started |2019-05-15 21:12:12|
|spark/flash/status|failed |2019-05-15 21:18:40|
2

@wesner0019, I had this happen on some device going from 0.9.0 to 1.0.1-rc.1. A device would stay in safe mode after the OTA. Reflashing the original app (again) allowed to complete the automatic healing steps and get back online. However, this was not a “product” device so you would need to push the update again.

3

This type of behavior is unexpected and in many cases we do not know that it is happening when we push updates and they updated at the next hand shake. In the last 2 days we’ve had 3 support calls from customers becasue they couldnt take their medications becuase the device was in safe mode.

Is there anything that can be done on particle side to make OTA more reliable?

4

I just got 2 more devices with this issue.

5

is there a remote api i can call to restart the device when its in safe mode?

6

When trying to downgrade the firmware the Console is stating it flashed sucessfully but the event says it failed. Not sure if this is playing into the above behavoir

image
image1891×907 102 KB

7

How do you perform the downgrade?
Flashing an application that is targeted at a previous device OS version does not downgrade the device OS itself.

8

I use the console to try to lock and flash now to a lower firmware version

9

The problem is that when trying to upgrade to a new OS the firmware flashes fine, but then the device recognizes that a new OS is needed so the device goes into safe mode, but the new OS never gets installed.

So by downgrading to an older firmware where the OS is at the version installed the device then gets out of safemode.

It seems that the device OS should try to flash 1st especially with LTE devices since the files are so large and that download speed is slow it allows for disconnections to happen. I’ve seen cases where it can take upto 6-8 minutes to download a new OS version.