Bug Bounty: Electron not booting after battery discharges completely

@marshall, you may want to try the latest version of CLI (1.24.0) which now implements device doctor to recover devices to "factory" state. :wink:

1 Like

device doctor fails on a windows machine

Put the device in DFU mode
Tap RESET/RST while holding MODE/SETUP until the device blinks yellow.
? Select Continue when ready Continue
Okay! New keys in place, your device will not restart.
Error creating keysā€¦ Error: Command failed: C:\WINDOWS\system32\cmd.exe /s /c ā€œopenssl ecparam -name prime256v1 -genkey -out 1234567890987654321_ec_new.pemā€
ā€˜opensslā€™ is not recognized as an internal or external command,
operable program or batch file.

Thatā€™s a PATH issue on your machine, not a CLI shortcoming.

1 Like

Not if you donā€™t have openssl on your machine by default though.

I didnā€™t think that Windows shipped with openssl?

I have got that working now by finding openssl in my mingw dir and running device doctor from there.

Next step it connected to particle and immediately breathes violet, I thought it was getting an OTA upgrade but it has been like that for about 10minutes

any ideas?

@marshall, the Windows installer for CLI also installs openSSL.

1 Like

OK - Iā€™ll check my path issues.

I now have 2 electrons that have been doctored. both seem to connect to particle, (fast cyan flash) but rather than progressing to breathing cyan they are instead progressing to breathing violet, like the OTA / serial listen mode.

Iā€™ll see if I can see them on particle somehow.

@marshall, your Electrons are likely in Safe mode, waiting for your app. If you are connected via USB, you can flash the code over via serial or DFU.

I'll give it a go and report back

Device Doctor said the following on completion

submitting public key succeeded!
Okay! New keys in place, your device should restart.

Flashing the default Particle Tinker app

......

Flash success!

The Doctor has restored your device!

Please visit our community forums if your device still can't connect to the Particle cloud

I assumed that the Particle Tinker app that Device Doctor flashed , would connect to the cloud.

@marshall, you are correct and thatā€™s what it seems like. Flash it over anyway and see what happens.

Yep, I have them both running now.

So to recap.
To recover a bricked unit, when the battery went flash and corrupted the device.

  1. Erase the uC flash (including the bootloader) using the STLINK utility changing any option bits to ensure the entire device is erased.
  2. Reload the bootloader via STLINK
  3. Run Particle Device Doctor in a directory that has openssl in it
  4. Donā€™t try and delete eeprom values when prompted (I suspect that there is no EEPROM on the electron)
  5. When device dr completes, load your own application back or particle flash --usb tinker (perhaps - never tried this)
  6. Confirm all is working - yet to do this but it breathed cyan so thatā€™s a good sign.

Thanks for the support.

Regards

Marshall

2 Likes

@BDub
I can confirm that firmware version 0.7.0-rc7 DOES NOT fix this problem. I run into it running 0.6.4 and followed the guidelines at
https://docs.particle.io/faq/particle-tools/jtag/core/
to re-flash my Electron. I updated the both the bootloader and Electron firmware to 0.7.0-rc.7 using the files at https://github.com/particle-iot/firmware/releases.
This happened at with my Li-ion battery at 3.0V.

1 Like

Thanks for the info. What was your firmware most likely doing at the time? Sleeping and waking or just running on battery down to 3.0V? Any details would be helpful.

1 Like

The system is woken up from sleep using the D1 pin, receives data over the serial port, writes the data to an SD card and resumes the sleep state. The data is sent in by another RF-enabled microcontroller. The Electron is the gateway to a WSN.
System is deployed outdoors with a 1W solar panel. I removed the battery protection IC from the Li-ion battery that comes with the particle because it reduces charging current significantly. I am using a TP4056-based charge-discharge controller which cuts off the system at roughly 3.0V. The Electron wonā€™t reach that far though. We are in the rainy season in East Africa until May so there are many cloudy days. SOC gradually reduces. See figure below. System went off line at SOC=14.6 (or just after, since no more data was logged after this)

A larger solar panel is recommended if you are wanting the battery to keep up with your power needs.

I have some code on the forum that allows you to put the Electron into Sleep Mode once the SOC hits 20% or whatever level you set it at to prevent the Electron from entering a shutdown state. The Electron will wake up every hour to check if the SOC has risen above 20% SOC before going back to your main code loop.

The Electron has a good charge controller built in and many have had success using it along with a solar panel input to recharge the LiPo battery including me.

I don't see any benefit from you using the TP4056 charger over the TI PMIC chip that's built into the Electron. Cutting the battery output off at 3v is basically a dead LiPo.

Using my code you can stop the Electron from pulling large amounts of power at any SOC level you choose while letting the solar panel recharge the battery until it's charged back up.

The 1w Solar panel is not enough power to bring the Bring the Electron back online, it's basically stuck in a dead battery state.

Iā€™ve seen you mention this code, but did not care much to get the link. Now that weā€™re at it, please point me to it.
By ā€œdeadā€, i presume you mean ā€œflatā€. The absolute minimum voltage to discharge Li-ion batteries safely to is about 2.8V, so 3.0V is quite safe.
A 1W solar panel in Uganda is actually overqualified for our deployment, in the sunny season. Itā€™s currently fixed at 1W for research purposes though, so we need to have it in the rainy season as well.
What I removed was the battery protection IC in the battery. Perhaps I should start charging through the TI PMIC as designed. The panel is 6v though, Iā€™ll need to check the maximum voltage it can take.

Yea Flat, empty enough to not support the Electrons startup and connection load which causes the brownout issue your seeing.

It's underqualified based on how you're trying to use it as your graph clearly shows. It can work with proper coding though. You would be better off with a bigger panel, especially in less sunny parts of the year.

The Battery protection IC will prevent short circuits.

The PMIC can accept the solar input directly from the solar panel without any problems, the max input voltage is 12+ volts. My code optimizes the solar panel by not allowing it to operate too far below its maximum power point.

@BDub @RWB This is an update.
I included a simple condition of not turning on the modem if the SOC is below 20%. The electron logs the SOC every about 7 minutes or so. When the SOC was finally OK, the data was uploaded and the SOC graph now looks like below:

I recorded a SOC of 1.1% at one point in time.
What this could indicate is that the Electron is happy going about its other duties (waking up, receiving data over serial and flushing to the SD card) at extremely low SOCs and things go haywire once you decide to turn the modem on.

Since the guys at particle failed to replicate this bug (is this true? up to now?), I will try to deliberately cause this bug to repeat and see (Anyone that has done repeatability tests with results?)

What I do when the battery SOC hits 20% is do nothing but go back to deep sleep for 60 mins and then only wake up and check SOC again to see if it's above 20% before running code normally agian.

If you only turn off the Modem when the SOC is below 20% but still do your other task then there is the possibility of the battery still draining to empty and causing the Electron to brownout and cause memory corruption.

From my testing, I could go at least 1+ weeks in this deep sleep mode waking up every hour before the battery would be empty which was more than enough time to recharge the battery via solar power.

I agree 1W is not enough to keep the electron charged, I would recommend 10W unless you are sleeping a lot... then 2-3W might be enough to get you by. I guess it really depends on how much sun you have (sounds like tons) and how often you are waking and staying awake for.

The battery protection IC in the battery should not limit the default current rate at all. It should allow charging rates of up to 1C easily, and we only charge at 0.25C (512mA) by default.

If you see the first post, with data... we have tried but don't have an exact repeatable test that we can perform over and over. There are many ways to mitigate this issue though as you are testing now.

Please do submit a minimal repeatable test case and collect the bounty :slight_smile:

1 Like

We were having a communications problem and decided to hookup 6 electrons to the same USB hub and power supply. Although all appeared to be charging, when someone went it 2 days later, 3 of them had dropped off the line.

All showed the blue led bug when first inspected. For 2 of them, once they were connected to a better power supply and the devices reset.

However 1 of them has not reset properly. I tried a different battery with no luck.
This one had no board connected at all. The only connections was the USB, battery and Antenna.

We are using the 7.0.0-rc6 firmware. We have ~12 particles in the field (With the same firmware) and all have had the battery run down on all of them (Multiple times). None have exhibited this behaviour before. NB. These all charge off the Vin port where as this one was via the USB. They also have a screen attached that runs off the 3V3 rail.