deviceOS@2.2.0 - Low Battery Condition - State of Charge not getting updated

[chipmc]

With winter approaching, I am starting to think about protecting my batteries when temperatures drop and sunlight is reduced.

My devices sleep most of the hour, waking occasionally to service hardware interrupts and then reporting at the top of the hour. When not doing actual work, I use the Sleep 2.0 API like this:

config.mode(SystemSleepMode::ULTRA_LOW_POWER)
      .gpio(userSwitch,CHANGE)
      .gpio(intPin,RISING)
      .duration(wakeInSeconds * 1000);
    SystemSleepResult result = System.sleep(config);                   // Put the device to sleep

Since deviceOS@2.0.0, I have stopped using the FuelGauge API and started relying on System.batteryCharge to monitor the battery and I dropped all that stuff about fuelGauge.wakeup(); and delays as we had all discussed back in the DeviceOS@1.5.0 days when the fuel gauge was put to sleep.

However, with deviceOS@2.2.0, I am seeing this issue again. Take a look at this console log. I deliberately put a device in a low battery state and, once it was asleep, I fully charged the battery. It failed to wake at the top of the hour so, I reset it and brought it on-line it reported 25% SOC. I then performed a power-cycle and, when it came back on-line only then did it display the correct SoC - 94%.

So, I will keep testing this but it raises some questions:

1. Are we still putting the fuelGauge to sleep in deviceOS@2.2.0?
2. If so, when the battery is charged between samples, does System.batteryCharge() take longer to take a measurement? Do we need to use the fuelGauge.quickStart()?
3. Do we still need to provide fuelGauge.wakeup() and delay(500) like we did starting back at deviceOS@1.5.0 when these issues first surfaced?

I will keep experimenting with solutions but any help is appreciated.

Thanks,

Chip

[Rftop]

@chipmc

It failed to wake at the top of the hour

Chip, wouldn't this be a major concern ?

[chipmc]

@Rftop ,

Yes for sure! The weird part is that this works on deviceOS@2.1.0 but since the battery charge level does not get updated, the device could get fully charged and never come back on-line.

Also strange, when the device is not in a low battery state, it tracks the SoC just fine. Something happens when the charge level gets low.

Chip

[Rftop]

What do you think about this:
Add a Sanity Check to the FSM when it was previously in the low battery state.
Just check the VCell voltage directly when SoC is reported low.
Would that allow you to "over-ride" the bogus SoC and get back on track?
(minus the power cycle requirement for full ops, but it's a start)

@chipmc

Also strange, when the device is not in a low battery state, it tracks the SoC just fine. Something happens when the charge level gets low.

But you decide the "low" State .....correct?
(Not the Device OS)
I might be way off-base in my thinking.

[chipmc]

@Rftop ,

Not off-base at all. I am working to get a handful of low batteries here in my home lab and test the following:

1. Add back the FuelGauge wakeup and delay - I will hold my nose when doing this but, I need to fix this. Long term, all this should be handled in the deviceOS and we should simply consume System.batteryCharge().
2. Another ugly fix - Use my power-cycle command. Perhaps I just do this every other hour. The device keeps counting and it stores the web hooks so, this works - but I don’t like it.
3. Your idea, if I see a Low Battery state, then I take an extra step and “check my answer” using the VCell reading. I should not be surprised but, your idea seems like the most palatable.

Longer term, we need deviceOS to abstract some of these complications from us.

Thanks,

Chip

[Rftop]

Chip, how are you swapping batteries for the test during ULP mode? I assume you have the Solar Panel provide primary power during the hot-swap ?
I'm trying to think how the standard behavior of the PMIC (and DeviceOS's interaction) might explain your results, but it's a stretch for me.

@chipmc

Longer term, we need deviceOS to abstract some of these complications from us

Agreed, but it appears that even the LTS branch is introducing new problems, even with Minor Updates to the LTS Branch.

Per the Particle LTS page:

Our new LTS release branch (Device OS 2.x) is feature-frozen in time – it will not receive updates that include new features, API changes, or improvements that change the function or standard behavior of the device.

This thread, along with the other issue that you've encountered this week (w/ LTS 2.2.0) seem to be a significant deviation from this promise.

Fingers Crossed that LTS gets back to the original intent soon....letting the feature development branch (Device OS 3.x) squash the bugs

[chipmc]

@Rftop ,

The good news is that I am doing all this work in my home office as I do not have any devices in the field with this release. As my father (almost) said “Test twice, deploy once”.

To test this I take the LiPO and discharge it using my LiPO battery station. I typically discharge it to 3.77V (30% SoC) as this is my Low Battery threshold. I then put that battery on a test device and power it on leaving it connected until it reports a value of less than 30% SoC and goes into low battery mode. I Then charge the device through the Vin pin using my bench top power supply set to 6V and up to 900mA. I can tell when the battery is approaching a nominal charge by watching the current supplied drop. Then, I wait for the top of the hour and see what happens. If the device fails to connect, I can bring it on-line and see all the cached messages. Does this make sense?

I am fairly certain this is new behavior with deviceOS@2.2.0 as the low battery code worked in the previous release. I am less certain that the “unlimited reconnect time” issue is new or if I am only now coming to realize it. It was simply luck that I had my serial monitor connected when a device had a reconnection event.

As for the LTS approach, I do think that this approach has made life better. There are a couple new features I would love to see in the next LTS release but agree it needs to be very stable or it looses its reason for existence.

I am currently testing this approach - suggested by you - to look at VCell if a low battery state is called. The code looks like this and it comes right after the low battery value is set:

  if (sysStatus.lowBatteryMode) {                                      // Need to take these steps 
    Log.info("Double checking our answers");
    fuelGauge.wakeup();                                                // Make sure the fuelGauge is woke
    delay(500);
    if (fuelGauge.getVCell() >= 3.77) {                               // Correcponds to 30% SoC - https://blog.ampow.com/lipo-voltage-chart/
      Log.info("Did not pass the smell test, vCell is %4.2f clearing low battery mode", fuelGauge.getVCell());
      sysStatus.lowBatteryMode = false;
      sysStatus.stateOfCharge = int(fuelGauge.getSoC());              // Might as well update the SoC as well - let's see
    }
  }

I will know in the next hour or so if it solves the problem.

Thanks,

Chip

[Rftop]

10-4. Thinking about this a little further, it might not be considered unreasonable for the Boron to “Not” know the updated SoC immediately coming out of ULP mode (during this extreme event), since I assume things are happening pretty fast in your FSM after WakeUp.

Just a note: All my previous testing for Remote Solar Borons was using Manual Mode.
That may help you eliminate a few more possible problems in the future…except for the circumstances when pushing the OS Updates.

[chipmc]

@Rftop @Colleen,

OK so some progress… It seems that @Rftop may be right about the VCell tracking while the SoC is not. Take a look at this console log. Testing with the “check our answers code” in the post above. This could a a relatively simple way around the problem if it bears out under more testing.

So, to recap, SoC is not getting updated - likely because it needs more time once re-awakened to make the state of charge estimate. Will do more testing to confirm.

Thanks, Chip

[Rftop]

@chipmc

So, to recap, SoC is not getting updated - likely because it needs more time once re-awakened to make the state of charge estimate.

I assume you are always sleeping in ULP mode, so how do you know the reported SoC is only Incorrect when coming out of your Low Battery Mode ? IE: The DeviceOS doesn't know or care about your FSM states, correct?
I'm wondering what FSM actions (or runtime) allow the SoC to be updated.... ?

[chipmc]

@Rftop,

Yes, always in ULP sleep. And three things can wake us up:

1. The sensor detects something
2. The user presses the “User Button” - see what I did there?
3. Time reaches the top of the hour

In the last case, the control returns to the IDLE_STATE which notices that the hour has changed and sends execution to the REPORTING_STATE. There, I run a function called takeMeasurements() which is when the data is collected for the hourly report. Here is what that function looks like:

void takeMeasurements()
{
  char data[64];
  if (Cellular.ready()) getSignalStrength();                           // Test signal strength if the cellular modem is on and ready

  getTemperature();                                                    // Get Temperature at startup as well
  
  // Battery Releated actions
  sysStatus.batteryState = System.batteryState();                      // Call before isItSafeToCharge() as it may overwrite the context

  isItSafeToCharge();                                                  // See if it is safe to charge
  sysStatus.stateOfCharge = int(System.batteryCharge());               // Assign to system value

  if (sysStatus.stateOfCharge < 65 && sysStatus.batteryState == 1) {
    System.setPowerConfiguration(SystemPowerConfiguration());          // Reset the PMIC
  }
  if (sysStatus.stateOfCharge < current.minBatteryLevel) current.minBatteryLevel = sysStatus.stateOfCharge; // Keep track of lowest value for the day
  
  if (sysStatus.stateOfCharge < 30) {
    sysStatus.lowBatteryMode = true;                                   // Check to see if we are in low battery territory
    if (!sysStatus.lowPowerMode) setLowPowerMode("1");                 // Should be there already but just in case...      
    snprintf(data, sizeof(data), "Low Battery Mode activated as SoC is %i", sysStatus.stateOfCharge);
    PublishQueuePosix::instance().publish("Diagnostic", data, PRIVATE);             
  }
  else {
    sysStatus.lowBatteryMode = false;                                  // We have sufficient to continue operations
    snprintf(data, sizeof(data), "Low Battery Mode cleared as SoC is %i", sysStatus.stateOfCharge);
    PublishQueuePosix::instance().publish("Diagnostic", data, PRIVATE);                            
  }

  if (sysStatus.lowBatteryMode) {                                      // Need to take these steps 
    PublishQueuePosix::instance().publish("Double checking our answers", PRIVATE);   
    fuelGauge.wakeup();                                                // Make sure the fuelGauge is woke
    delay(500);
    if (fuelGauge.getVCell() >= 3.77) {                               // Correcponds to 30% SoC - https://blog.ampow.com/lipo-voltage-chart/
      snprintf(data, sizeof(data), "Override, SoC: %4.2f, vCell: %4.2f clearing low battery mode", fuelGauge.getSoC(),fuelGauge.getVCell());
      PublishQueuePosix::instance().publish("Diagnostic", data, PRIVATE);   
      sysStatus.lowBatteryMode = false;
      sysStatus.stateOfCharge = int(fuelGauge.getSoC());              // Might as well update the SoC as well - let's see
    }
  }
  
  systemStatusWriteNeeded = true;
  currentCountsWriteNeeded = true;
}

So, I could simply trigger the low battery state based on fuelgauge.getVCell() but this does not solve the riddle of why, when the battery level is low, the state of charge stops updating. In my testing so far, the SoC does not correct itself even after a few hours and a full charge. Unfortunately, the FuelGauge API is not well documented in the Particle docs so, will do some experimentation - perhaps quickStart()?

One other thing to note is that FuelGauge SoC is never equal to System.batteryCharge(). But that is a second order effect at this point.

Chip

[Rftop]

@chipmc

So, I could simply trigger the low battery state based on fuelgauge.getVCell() but this does not solve the riddle of why, when the battery level is low, the state of charge stops updating.

Chip, I took a quick look at some of my Validation Trial Notes (back in the 1.5.x days).
Same thing could happen then with SoC and Boron's would sometimes Not recharge.
That's why I started using the Li-Po Voltage (VCell) instead. I Take the measurement after the current settles down and before turning on the Modem (Manual Mode).

But I must also say that the fuelGauge was impressive with it's accuracy under "general/normal" operating conditions.

One other thought that's related to your OP.
I Love your 3.77 V "low battery" threshold.
But since you're getting ready for wintertime, have you checked your particular Li-Po model to determine the suppressed voltage due to your wintertime target temperature ?
IE: the same SoC at 3.77V at 75 F will be significantly less Voltage at 35 F.
I know that you're already aware of this, just wanted to mention it since you might be considering using the VCell as your trigger.

It's pretty easy to establish the curve for any particular Li-Po.
For your case, you just graph the Temp vs Voltage for a USED (not new) Li-Po that was ~3.75 resting at room temp.

@chipmc

One other thing to note is that FuelGauge SoC is never equal to System.batteryCharge(). But that is a second order effect at this point.

My guess is the System.batteryCharge assumes 4.11V termination charge (Particle Default) and FuelGauge knows that a "full" Li-Po is 4.22V ......just a guess though.

[Rftop]

Chip, I’m Sorry for clogging up your Thread.
I’d also love to hear other peoples opinions and thoughts on this.
I’m getting ready to trial Solar Borons again, so I’m interested in any recent results.

[chipmc]

@Rftop , Very happy to have your engagement here.

Approaching a potential solution. Added two lines to my "check my work" conditional:

  if (sysStatus.lowBatteryMode) {                                      // Need to take these steps 
    PublishQueuePosix::instance().publish("Double checking our answers", PRIVATE);   
    fuelGauge.wakeup();                                                // Make sure the fuelGauge is woke
    delay(500);
    fuelGauge.quickStart();                                            // May help us re-establish a baseline for SoC
    delay(500);
    if (fuelGauge.getVCell() >= 3.77) {                               // Correcponds to 30% SoC - https://blog.ampow.com/lipo-voltage-chart/
      snprintf(data, sizeof(data), "Override, SoC: %4.2f, vCell: %4.2f clearing low battery mode", fuelGauge.getSoC(),fuelGauge.getVCell());
      PublishQueuePosix::instance().publish("Diagnostic", data, PRIVATE);   
      sysStatus.lowBatteryMode = false;
      sysStatus.stateOfCharge = int(fuelGauge.getSoC());              // Might as well update the SoC as well - let's see
    }

The addition of the quickStart command seems to have done the trick. From the MAX17043 data sheet,

A quick-start allows the MAX17043/MAX17044 to restart fuel-gauge calculations in the same manner as initial pow- er-up of the IC. For example, if an application’s power-up sequence is exceedingly noisy such that excess error is introduced into the IC’s “first guess” of SOC, the host can issue a quick-start to reduce the error.

So, now, if a low battery mode condition is detected, I will take the extra time to check to make sure that the Fuel Gauge is restarted and that it takes a fresh look at the battery and sees that it needs to revise its SoC guess.

See this log file that shows the device waking, getting the SoC wrong then checking its work and over-riding the old values for the win.

I have a suggestion for Particle @Colleen , I think that all of this messiness could be avoided if there was a way to prevent ULP sleep from putting the Fuel Gauge to sleep. According to the data sheet, the Fuel Gauge only uses 50uA of current and, I for one, would be willing to simply let it run in order to get a better read on the battery charge.

I am going to try one more thing. I wonder if things would improve if I simply woke the Fuel Gauge each time we left the napping state. The device stays awake for at least one second and perhaps, in aggregate, this would allow the FuelGuage to monitor the battery and have a better guesss at the top of the hour. Would not help at times when there is no visitation but, it is worth a try.

Thanks,

Chip

[Colleen]

@chipmc

think that all of this messiness could be avoided if there was a way to prevent ULP sleep from putting the Fuel Gauge to sleep

That's a good point. I'll share this message with the team internally.

[chipmc]

@Rftop and @Colleen ,

Thank you for your help on this. I will mark this as a solution though it feels more like a workaround. A complete solution would include the following:

• Better documentation of the FuelGauge API in the docs
• System.batteryCharge() - incorporating the wakeup and fast start commands needed for sleepy applications
• An option to not sleep the FuelGauge in the ULP mode of Sleep 2.0

That said, here is what I did which seemed to work:

1. Added the FuelGauge API Upfront

FuelGauge fuelGauge;

2. Added these lines to startup()

  fuelGauge.wakeup();                                                  // Expliciely wake the Feul gauge and give it a half-sec
  delay(500);
  fuelGauge.quickStart();                                              // May help us re-establish a baseline for SoC

3. Added this line after napping or sleeping

    fuelGauge.wakeup();                                                // Make sure the fuelGauge is woke

4. Here are the lines I use when taking measurements:

  if (sysStatus.lowPowerMode) {                                        // Need to take these steps if we are sleeping
    delay(500);
    fuelGauge.quickStart();                                            // May help us re-establish a baseline for SoC
    delay(500);
  }

  sysStatus.stateOfCharge = int(fuelGauge.getSoC());                   // Assign to system value

  if (sysStatus.stateOfCharge < 65 && sysStatus.batteryState == 1) {
    System.setPowerConfiguration(SystemPowerConfiguration());          // Reset the PMIC
  }
  if (sysStatus.stateOfCharge < current.minBatteryLevel) current.minBatteryLevel = sysStatus.stateOfCharge; // Keep track of lowest value for the day
  
  if (sysStatus.stateOfCharge < 30) {
    sysStatus.lowBatteryMode = true;                                   // Check to see if we are in low battery territory
    if (!sysStatus.lowPowerMode) setLowPowerMode("1");                 // Should be there already but just in case...      
    snprintf(data, sizeof(data), "Low Battery Mode SoC: %4.2f, vCell: %4.2f", fuelGauge.getSoC(),fuelGauge.getVCell());
    if (Particle.connected()) Particle.publish("Diagnostic", data, PRIVATE);             
  }
  else {
    sysStatus.lowBatteryMode = false;                                  // We have sufficient to continue operations
    snprintf(data, sizeof(data), "Normal Operations SoC: %4.2f, vCell: %4.2f", fuelGauge.getSoC(),fuelGauge.getVCell());
    if (Particle.connected()) Particle.publish("Diagnostic", data, PRIVATE);                            
  }

  systemStatusWriteNeeded = true;
  currentCountsWriteNeeded = true;
}

With these changes, my device will recognize a change in charge level even when it is charged while sleeping.

I hope this is helpful. Please let me know if you have comments / suggestions.

Thanks,

Chip