Bug Bounty: Electron not booting after battery discharges completely

I might have missed it, but going to 0.8.0-rc.x from an “unknown” other version might also require you to flash the bootloader via YMODEM (aka particle flash --serial) - have you done that?

You can go directly to the latest version with electron since the bootloader is built into the system firmware. Photon/P1 you should update to 0.7.0 first, then go to the latest 0.8.0-rc.x.

bko,
I have the dim D7 problem as described in my post of Sep. 3. Can you briefly describe the fix with the JTAG programmer that you mention? Or give a link to a description of the fix?
thanks,
john

It has been long since I last visited. We deployed 10 Electron 3Gs in field in November last year and wrote some application level checks to ensure we never get to the low battery blue LED problem (A similar approach to @rickkas7 but checking every N hours linearly where N=1, 2, 3…).
We have not had any problems so far. We are now going to deploy 45 more and I would like to know if there have been any advances since I was last here. Has the bounty been collected? Is the issue still manifesting itself in new firmware?

2 Likes

Nothing has changed and your battery level check code is still required to prevent any potential damage to the Electrons.

The bounty has not been collected due to this being a hard to replicate issue.

I have just suffered this issue on the 1.1.0 firmware.
Fortunately I had the Programmer Shield and could solve it in the day :slight_smile:

So I just left here the command executed on the windows 10 terminal, if someone needs it

C:\Users\ [name] \.particle\toolchains\openocd\0.11.1-adhoc6ea4372.0\bin>openocd -f interface/ftdi/particle-ftdi.cfg -f target/stm32f2x.cfg -c "init; reset halt; stm32f2x unlock 0; reset halt; flash protect 0 0 0 off; program C:\Users\bootloader-1.0.1-electron.bin verify 0x08000000;flash protect 0 0 0 on; exit"

Edit: After the recovery the blue light flashed very fast. In this post found the solution

2 Likes

I just got this problem, sad. I don’t have any programmer on hand or anything. The interesting part is that I believe that mine was not being provided under voltage.

My setup for data point:
Running on 12V SLA, 5V regulator. 470uF capacitor in. No USB, no LiPo connected.
The voltage of the battery was 12.8V, not low at all, the voltage regulator needs at least 10V to provide 5v. I measured the VIN voltage an it was 4.95V.

Last thing before it died was the red LED blinking red at 1Hz, no OS code. Simply blinking steadily at 1Hz without stopping, this went on for minutes before I unplugged it to reboot it. Dim blue LED showed up after that.

@BDub,

I can reproduce this issue reliably on my custom PCB (I have 1000+ of these in the field) using a programmable power supply with an Electron running tinker or our application and device-os at 1.4.4 or 1.5.2. I accomplish this through the following routine:

  • Power supply set to 19V output
    • Our onboard regulator is a TPS54340 outputting 4.88V
    • No UVLO
    • 19V chosen to mimic a solar panel
  • Current set to 30mA
  • No lipo attached
  • Using a Raspi controlling a KORAD power supply over USB serial I run the following routing:
    • Current decreases by 1mA each second until it reaches 1mA, at which point it begins to increase by 1mA each second until it reaches 30mA
    • At 30mA it pauses for 20 seconds
    • It then increases by 1mA each second until it reaches 60mA, at which point it begins to decrease by 1mA each second until it reaches 30mA
    • At 30mA it pauses for 20 seconds
    • Repeat many times
  • I have not isolated whether our watchdog affects this somehow
  • I have not tested a release later than 1.5.2

After 1-2 hours, I will have an Electron with a damaged bootloader. I can recover the Electron relatively simply by following the SWD guide for flashing bootloaders.

For context, our devices experience a bootloader damaged when they either:

  • Have a solar panel directly connected to them (which mimics the above routine)
  • The 12V sealed lead acid battery they are connected to dies and goes to 4V

From viewing the LED patterns and whatnot, I think what is happening is:

  • PMIC turns on because 19V (4.88V) / 30mA (115mA) passes its input source checks
  • STM32 turns on the modem and PMIC turns on battery charging
  • Overdraw of current from modem brings power down to brownout territory for electron
  • Reset triggered
  • Repeat until bootloader is damaged

I think the same thing is happening in the other scenarios in this thread.

I’m happy to run other tests on this or supply additional data if someone at Particle needs more information.

2 Likes

FYI @BDub
(I am mentioning you just to make sure you get a notification - thanks!)

19v does not represent the 6v Voltaic panels that most people are using and seeing this issue but interesting find never less.

Also can’t remember exactly but how close is 19v to the input limit for that pmic chip? 12v is the max recommended input voltage by Particle.

Also good job on getting 1000 units out into the field!

The 19V is coming in through our on board TPS54340 regulator. The PMIC is seeing 4.88v when everything is operating correctly and something between 4V and 4.88V when the Electron is overdrawing current from the 19V via TPS54340 source. Something very similar is likely to be happening with your 6V panel, but directly instead of through another regulator.

It’s in these brief overdraw events that I think the Electron is able to damage its boot loader. It’s my understanding that the point of this thread is to harden the Electron against damaging its bootloader this way. For example, the board also has an ATMEGA on board that is able to survive these events through its bod.

1 Like

@BDub, would it help to have the bins of the bootloaders before and after they are damaged? Any other data need to be collected?

@hwestbrook thank you so much for providing many details on this. If you can capture the bootloader before and after, that would be useful. You can send those through Support. Better though, if you have a hardware setup that reproduces this, I’d love to take a look personally and see what I can figure out. Particle Support will reach out to you to try to get something sent to me, if you are up for it. Again, thanks for the insight on this… it has not really been reproducible through forced methods. Looking forward to finding the root cause, or in the least a working mitigation :smiley: