Electron OTa upgrade experience

We are in the process of upgrading a fleet of Electrons spread across the San Francisco Bay area from 0.6.4 to 1.5.2.

The process we are using is:

  1. Place the device to be upgraded in development mode in the integration.
  2. OTA Flash the device with the application compiled for 1.5.2
  3. Let the device / Particle do their thing to upgrade the OS firmware.
  4. Using the Workbench as the development platform

To date we have attempted 4 devices:

1 - Worked as one would expect - Flash worked as expected, Device OS was upgrade, application worked as expected
2 - Failed, recovered the device from the field, attempted to recover the device via the USB using the CLI tools - failed. After discussion with Particle, concluded it could not be recovered = a brick. Particle replaced the device at NC - thank you
3. - Worked as one would expect - Flash worked as expected, Device OS was upgrade, application worked as expected
4. - Failed, recovered the device from the field. Found that the device was in “safe mode”. Connect the device to USB. Each time “reset” was pressed, the device would connect to the cloud and then go to “safe mode”. Ran “Particle doctor” on the device. Application came live and ran as expected.

If this 50% failure rate continues, this is going be a very expensive upgrade process.

Questions:

  1. Is this what others have experienced from the OTA upgrade process?
  2. Are we expecting too much from the Particle tools?
  3. Is there a more reliable way to complete an OTA upgrade?
  4. Are we doing something wrong?

Suggestions?