Tracker in unknown state after wake up from discharge

Yes, in Tracker.cpp, at the top of Tracker::init(), add a wait for USB connect:

waitFor(Serial.connected, 10000);

Then just sprinkle calls through init().

Hi,

after adding

unsigned int last_timestamp = millis();

while(true)
{
    if ( Serial.isConnected() || (millis() - last_timestamp > 10000) )
    {
        break;
    }
}

to tracker.cpp in line 548, the device now prompts the following when waking up from blackout:

0000002034 [ModelGauge] INFO: read original OCV: 152, 8
0000002034 [ModelGauge] INFO: verify model access unlocked: success
0000002344 [ModelGauge] INFO: load model successfully

Even though it should print the battery charge every 5 seconds, it doesn't do so. It seems to be stuck somewhere.

This is my main.cpp:

#include "Particle.h"
#include "tracker_config.h"
#include "tracker.h"

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

#if TRACKER_PRODUCT_NEEDED
PRODUCT_ID(TRACKER_PRODUCT_ID);
#endif // TRACKER_PRODUCT_NEEDED
PRODUCT_VERSION(TRACKER_PRODUCT_VERSION);

STARTUP(
    Tracker::startup();
);

SerialLogHandler logHandler(115200, LOG_LEVEL_TRACE, {
    { "app.gps.nmea", LOG_LEVEL_INFO },
    { "app.gps.ubx",  LOG_LEVEL_INFO },
    { "ncp.at", LOG_LEVEL_INFO },
    { "net.ppp.client", LOG_LEVEL_INFO },
});

double bat_charge = 0.0f;
int bat_state = 0;

void setup()
{
    Tracker::instance().init();

    Particle.variable("bat_charge", bat_charge);
    Particle.variable("bat_state", bat_state);
}

void loop()
{
    Tracker::instance().loop();

    static unsigned int _last_timestamp_ms = 0;

    if ( millis() - _last_timestamp_ms > 5000 )
    {
        bat_charge = (double)System.batteryCharge();
        bat_state = System.batteryState();
        Log.info("Battery charge at %02.1f%%", System.batteryCharge());
        _last_timestamp_ms = millis();
    }
}

Other than the modification of tracker.cpp, I have not changed anything else.

UPDATE

The infite loop is in file deviceOS/5.3.2/system/src/system_power_manager.cpp in line 942.

It contains a CONCURRENT_WAIT_FOREVER constant that halts the queue action until it's done, which isn't the case:
os_queue_put(queue_, (const void*)&ev, CONCURRENT_WAIT_FOREVER, nullptr);

The entire function is called PowerManager::setConfig() and starts in line 937.

int PowerManager::setConfig(const hal_power_config* conf) {
  int ret = hal_power_store_config(conf, nullptr);
  if (isRunning()) {
    // Power manager is already running, ask it to reload the config
    Event ev = Event::ReloadConfig;
    os_queue_put(queue_, (const void*)&ev, CONCURRENT_WAIT_FOREVER, nullptr);
  }
  return ret;
}

I exchanged the constant with a smaller number 10000u which worked in my case. However I could need some explanation what is going on here and if this is an actual bug that needs reporting.

Thanks for having a look into this.

That is interesting, thank you for debugging. I'll have to have someone take a look because I'm not sure why it's doing that.

Thanks so much for helping out on this one, I was expecting a protection mechanism in implemented in the deviceOS in case the battery is depleted.

In the meantime, I implemented a workaround that puts the device in shipping mode when FuelGauge::getSoC() falls below 1% and gracefully shut down the modem at 5%. I believe I couldn't drain the battery fully since the device doesn't draw measurable amounts of current in shipping mode. I'll let it sit and turn it off in a couple of days.

Hi, it seems there is one, and you can read more here.

1 Like

Hi Gus,

Thank you for checking in! I read your post on my research but the fact that the voltage drops to below 1% was indication enough for me to realize that this mechanism isn't working.
Meanwhile I was recommended to switch from deviceOS 5.3.2 to deviceOS 4.1.0 since it's the long-term-support deviceOS for the Tracker One. I will update this post, if this helps in any way.

Cheers!

2 Likes

Update
I performed the fallback to deviceOS 4.1.0 and tracker-edge v18.
Unfortunately the issue persists. The device won't power down when falling below 2% (as this is the setting in tracker.cpp.

I tried "simulating" the battery by connecting a laboratory power supply to the battery pins and dialed the voltage down. However, the device keeps restarting as if it's not getting continuous power. A 100nF capacitor between GND and LI+ for buffering didn't help.

My hope was to manually dial down the voltage until it reaches the threshold of 2% where I would register the device shutdown. If you have any suggestions for trying this, please let me know.

Hi, I do not know what to say, I see it working on my side, however with larger values than 2% and 1%.

I set 20% and 10% values on my side, using DeviceOS 3.2.0 and Tracker version 17 (I believe).

Out of curiosity, I went and tested the low batt warning and this is what the warning looks like when set at 80% (for testing):

constexpr int TrackerLowBatteryWarning = 80; // percent of battery charge

1 Like

Hi Gus,

thanks for testing this on your side.

Maybe I didn't express myself in the best possible way.

I am expecting that the device turns off automatically (shipping mode) when a certain threshold of battery charge is reached. It would be even better if it disconnects gracefully from the Particle cloud and then shuts down.
Just having an alarm is not sufficient and doesn't protect the battery. Since the Tracker is not waking up properly after battery depletion.

Is that working on your side or is it just an alarm that you can set and react to by entering shipping mode on your own?

Hey, it is working on my side, the device entered shipping mode on my tests. I remember testing this with 70% (fast test) and 10% values (real-life test).
I am starting again a 10% test to double-check what I have observed in the past.
I provided the batt_warn info for information only.

I am using the internal battery that comes with the tracker one.

I do not make the tracker one enter shipping mode on my firmware, it goes there by itself.

Maybe one percent is too low for this to work? Maybe you can try with 10% just for fun? I would also set the warning to 20%, again, only for this test.

1 Like

Hi Gus,

this pretty much sounds like an application firmware topic.

Could you share with me a working code of your test application?

Hi Johannes,
my test finished with a pass.
As you can see the device went offline (into shipping mode) right after reporting 10.54%. I set the limit to 10% and is using a somewhat old battery:

Half an hour earlier it sent the battery low warning, set at 20%:

Hi Gus,
For which reason are you using an old deviceOS and tracker-edge firmware?

I just ran another test on deviceOS 4.1.0 and tracker-edge v18 and it didn't work. The battery got depleted again.

hey, no particular reason.

Are you using a new battery? I wonder if it could be the battery that gets depleted way too fast to even be detected as a low batt.

Did you make the changes in the source code and set the min to 80 or 90% just for this test?

IMHO, many of the % SOC numbers being discussed are very optimistic with Lithium Chemistries.

Even "IF" the SOC Estimate was accurate to within a couple Percentage Points on the bench, that's unlikely to happen in real world applications. Temperature changes alone will cause voltage swings way outside the SOC accuracies of 1-2%, in my experience. I assume most people wont use a Tracker in an indoor temperature controlled environment.

I would expect the Device could easily enter a continuous BrownOut Loop at these voltage levels when you consider the battery's actual performance curve (available AMPs vs Volts at the end of the discharge curve). And this curve changes with the age and any abuse of the battery. It's only "new" for the first cycle.

Wouldn't 20-30% SOC be a better threshold to start making decisions for power savings?
Possibly consider shipping mode at 10% ?
Even that seems optimistic with any decent amount of time in the wild, and given the SOC is only an estimate.

It makes sense. Thank you for all the inputs from real-life scenarios!

We originally set the threshold to 10% to enter shipping mode but got a lot of push back that there was a considerable amount of charge left. We finally came to 2% to give several hours, if fully powered, to perform what is necessary to alert the cloud. The 2% margin also accounts for possible drops in temperature that would decrease the apparent capacity.

1 Like

@johannes.nuwiel Our team here in the US and Europe tried to replicate the shipping mode you had but can't replicate it. I noticed that there is a missing "batt_low" publish that should have gone out right before entering shipping mode and I've filed a bug for that.
The only thing that I can think that could interrupt entering shipping mode would be parasitic current powering the Tracker One. We have FETs that essentially decouple the M8 IO from the MCU and is controlled with the CAN 5V enable signal. The CAN 5V signal is disabled when going into shipping mode to decouple the IO.

Hi @eberseth,

thanks for having a look and finally replicating the issue. I know it's buried deep.

Regarding the CAN 5V Enable, on startup of my Tracker I am executing this piece of code which (if it does what it says) keeps the CAN 5V powered on even in sleep mode:

TrackerConfiguration enableIoCanPowerConf;
enableIoCanPowerConf.enableIoCanPower(true).enableIoCanPowerSleep(false);
Tracker::instance().init(enableIoCanPowerConf);

It would be great if this closes the assumption that you made. Otherwise please state what the function enableIoCanPowerSleep(bool) does.

Cheers!