Tracker in unknown state after wake up from discharge

Edge Systems
,
1

Hi there,

Issue:
After my tracker has been running out of battery, I resurrect it by applying USB power and wait for it to prompt something to the serial monitor.
The status LED pulses white (indicating turned off cellular connection?) but nothing is printed on the serial monitor.
Pressing the reset button brings it back to normal operation but on the packaged version installed in a vehicle I don't have access to the port.

It seems like no code is being executed and that's why I don't where to look for errors in the code.

Expected result:
The tracker should start up normally after power is applied, even if the battery was discharged. It should behave like coming out of shipping mode. Also, the serial monitor should prompt messages saying in which state it is since I cannot debug in the unknown state.

Info:
I am running firmware V119 and DeviceOS 5.3.2 on a Tracker One T524M.

Appreciate any help.

2

Does it ever recover, after the battery recharges enough?

When it's in that state, is it possible to change the device mode, for example with

particle usb dfu

These aren't actual solutions to your problem, but what you're seeing is not a known problem and it's not immediately obvious to me what is happening.

The device is probably running some firmware, because:

  • If there's no bootloader, the LED stays off
  • If there's no user firmware, the LED will blink yellow (DFU mode)
  • If there's a missing firmware dependency, the LED will blink green (on the way to breathing magenta)
3

Hi,

unfortunately, it never recovers even after the battery is charged for 6 hours.

However, it is possible to put it into dfu mode, the LED blinks yellow now as expected.

I am asking this question to the forum because for me too it isn't apparent where I can even start looking for issues.

Have you got any idea where to place a Log.info command in the code so I can narrow done where the device is stopping?

Thanks!

4

Yes, in Tracker.cpp, at the top of Tracker::init(), add a wait for USB connect:

waitFor(Serial.connected, 10000);

Then just sprinkle calls through init().

5

Hi,

after adding

unsigned int last_timestamp = millis();

while(true)
{
    if ( Serial.isConnected() || (millis() - last_timestamp > 10000) )
    {
        break;
    }
}

to tracker.cpp in line 548, the device now prompts the following when waking up from blackout:

0000002034 [ModelGauge] INFO: read original OCV: 152, 8
0000002034 [ModelGauge] INFO: verify model access unlocked: success
0000002344 [ModelGauge] INFO: load model successfully

Even though it should print the battery charge every 5 seconds, it doesn't do so. It seems to be stuck somewhere.

This is my main.cpp:

#include "Particle.h"
#include "tracker_config.h"
#include "tracker.h"

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

#if TRACKER_PRODUCT_NEEDED
PRODUCT_ID(TRACKER_PRODUCT_ID);
#endif // TRACKER_PRODUCT_NEEDED
PRODUCT_VERSION(TRACKER_PRODUCT_VERSION);

STARTUP(
    Tracker::startup();
);

SerialLogHandler logHandler(115200, LOG_LEVEL_TRACE, {
    { "app.gps.nmea", LOG_LEVEL_INFO },
    { "app.gps.ubx",  LOG_LEVEL_INFO },
    { "ncp.at", LOG_LEVEL_INFO },
    { "net.ppp.client", LOG_LEVEL_INFO },
});

double bat_charge = 0.0f;
int bat_state = 0;

void setup()
{
    Tracker::instance().init();

    Particle.variable("bat_charge", bat_charge);
    Particle.variable("bat_state", bat_state);
}

void loop()
{
    Tracker::instance().loop();

    static unsigned int _last_timestamp_ms = 0;

    if ( millis() - _last_timestamp_ms > 5000 )
    {
        bat_charge = (double)System.batteryCharge();
        bat_state = System.batteryState();
        Log.info("Battery charge at %02.1f%%", System.batteryCharge());
        _last_timestamp_ms = millis();
    }
}

Other than the modification of tracker.cpp, I have not changed anything else.

6

UPDATE

The infite loop is in file deviceOS/5.3.2/system/src/system_power_manager.cpp in line 942.

It contains a CONCURRENT_WAIT_FOREVER constant that halts the queue action until it's done, which isn't the case:
os_queue_put(queue_, (const void*)&ev, CONCURRENT_WAIT_FOREVER, nullptr);

The entire function is called PowerManager::setConfig() and starts in line 937.

int PowerManager::setConfig(const hal_power_config* conf) {
  int ret = hal_power_store_config(conf, nullptr);
  if (isRunning()) {
    // Power manager is already running, ask it to reload the config
    Event ev = Event::ReloadConfig;
    os_queue_put(queue_, (const void*)&ev, CONCURRENT_WAIT_FOREVER, nullptr);
  }
  return ret;
}

I exchanged the constant with a smaller number 10000u which worked in my case. However I could need some explanation what is going on here and if this is an actual bug that needs reporting.

Thanks for having a look into this.

7

That is interesting, thank you for debugging. I'll have to have someone take a look because I'm not sure why it's doing that.

8

Thanks so much for helping out on this one, I was expecting a protection mechanism in implemented in the deviceOS in case the battery is depleted.

In the meantime, I implemented a workaround that puts the device in shipping mode when FuelGauge::getSoC() falls below 1% and gracefully shut down the modem at 5%. I believe I couldn't drain the battery fully since the device doesn't draw measurable amounts of current in shipping mode. I'll let it sit and turn it off in a couple of days.

9

Hi, it seems there is one, and you can read more here.

1 Like
10

Hi Gus,

Thank you for checking in! I read your post on my research but the fact that the voltage drops to below 1% was indication enough for me to realize that this mechanism isn't working.
Meanwhile I was recommended to switch from deviceOS 5.3.2 to deviceOS 4.1.0 since it's the long-term-support deviceOS for the Tracker One. I will update this post, if this helps in any way.

Cheers!

2 Likes
11

Update
I performed the fallback to deviceOS 4.1.0 and tracker-edge v18.
Unfortunately the issue persists. The device won't power down when falling below 2% (as this is the setting in tracker.cpp.

I tried "simulating" the battery by connecting a laboratory power supply to the battery pins and dialed the voltage down. However, the device keeps restarting as if it's not getting continuous power. A 100nF capacitor between GND and LI+ for buffering didn't help.

My hope was to manually dial down the voltage until it reaches the threshold of 2% where I would register the device shutdown. If you have any suggestions for trying this, please let me know.

12

Hi, I do not know what to say, I see it working on my side, however with larger values than 2% and 1%.

I set 20% and 10% values on my side, using DeviceOS 3.2.0 and Tracker version 17 (I believe).

Out of curiosity, I went and tested the low batt warning and this is what the warning looks like when set at 80% (for testing):

constexpr int TrackerLowBatteryWarning = 80; // percent of battery charge

image
image1920×1543 151 KB

1 Like
13

Hi Gus,

thanks for testing this on your side.

Maybe I didn't express myself in the best possible way.

I am expecting that the device turns off automatically (shipping mode) when a certain threshold of battery charge is reached. It would be even better if it disconnects gracefully from the Particle cloud and then shuts down.
Just having an alarm is not sufficient and doesn't protect the battery. Since the Tracker is not waking up properly after battery depletion.

Is that working on your side or is it just an alarm that you can set and react to by entering shipping mode on your own?

14

Hey, it is working on my side, the device entered shipping mode on my tests. I remember testing this with 70% (fast test) and 10% values (real-life test).
I am starting again a 10% test to double-check what I have observed in the past.
I provided the batt_warn info for information only.

I am using the internal battery that comes with the tracker one.

I do not make the tracker one enter shipping mode on my firmware, it goes there by itself.

Maybe one percent is too low for this to work? Maybe you can try with 10% just for fun? I would also set the warning to 20%, again, only for this test.

1 Like
15

Hi Gus,

this pretty much sounds like an application firmware topic.

Could you share with me a working code of your test application?

16

Hi Johannes,
my test finished with a pass.
As you can see the device went offline (into shipping mode) right after reporting 10.54%. I set the limit to 10% and is using a somewhat old battery:

image
image2018×1084 277 KB

Half an hour earlier it sent the battery low warning, set at 20%:

image
image2044×458 74.2 KB

17

Hi Gus,
For which reason are you using an old deviceOS and tracker-edge firmware?

I just ran another test on deviceOS 4.1.0 and tracker-edge v18 and it didn't work. The battery got depleted again.

18

hey, no particular reason.

Are you using a new battery? I wonder if it could be the battery that gets depleted way too fast to even be detected as a low batt.

Did you make the changes in the source code and set the min to 80 or 90% just for this test?

19

IMHO, many of the % SOC numbers being discussed are very optimistic with Lithium Chemistries.

Even "IF" the SOC Estimate was accurate to within a couple Percentage Points on the bench, that's unlikely to happen in real world applications. Temperature changes alone will cause voltage swings way outside the SOC accuracies of 1-2%, in my experience. I assume most people wont use a Tracker in an indoor temperature controlled environment.

I would expect the Device could easily enter a continuous BrownOut Loop at these voltage levels when you consider the battery's actual performance curve (available AMPs vs Volts at the end of the discharge curve). And this curve changes with the age and any abuse of the battery. It's only "new" for the first cycle.

Wouldn't 20-30% SOC be a better threshold to start making decisions for power savings?
Possibly consider shipping mode at 10% ?
Even that seems optimistic with any decent amount of time in the wild, and given the SOC is only an estimate.

20

It makes sense. Thank you for all the inputs from real-life scenarios!