Boron Devices Stop Charging deviceOS@1.5.0

[chipmc]

I have a number of devices which, after running normally for a few days, stop accepting a charge. This is causing devices that have run on Solar with STOP mode sleep to discharge their batteries even when there is solar or DC-In voltage to charge.

To fix the issue I have to:

• Turn off the device
• Disconnect LiPO and Solar / DC-In power
• Wait for 60 seconds
• Reconnect and Power the device up / repeat if needed till I see the yellow charging light.

I think this is connected to the this thread on deviceOS@1.5.0 and Sleep.

Sleepy Boron - SOC Not Updating

Its been a while since I read the datasheet for the Fuel Gauge, but the SOC is a calculated value based on many variables. The Fuel Gauge will gather data and eventually return a correct assumption about the charge condition of the Li-Po. The Fuel Gauge doesn’t have the luxury of taking a voltage measurement of a Li-Po in a Resting Condition. The SOC “inaccuracies” for a Sleepy Boron might be simply that the Fuel Gauge is sleeping also (guessing) with the recent Updates in 1.5.0 final ? My …

Here are the relevant snippets of code from my sketch:

In program header:

   SystemPowerConfiguration conf;                      // Initalize the PMIC class so you can call the Power Management functions below.

In the Setup() section:

    conf.powerSourceMaxCurrent(1500)                                  // default is 900mA
    .powerSourceMinVoltage(5080)                                     // Set the lowest input voltage to 5.080 volts best setting for 6V solar panels
    .batteryChargeCurrent(1024)                                      // default is 512mA matches my 3W panel
    .batteryChargeVoltage(4208);                                     // Allows us to charge cloe to 100% - battery can't go over 45 celcius
    System.setPowerConfiguration(conf);

Executed from the main loop() every hour as part of reporting process:

   void getBatteryContext() {
     const char* batteryContext[7] ={"Unknown","Not Charging","Charging","Charged","Discharging","Fault","Diconnected"};  // Battery conect information - https://docs.particle.io/reference/device-os/firmware/boron/#batterystate-
     snprintf(batteryContextStr, sizeof(batteryContextStr),"%s", batteryContext[System.batteryState()]);
   }

Here are some charging profiles from multiple devices. Note that they were charging regularly for days.

I will open a support ticket on this issue.

Any help or suggestions would be welcome.

Thanks,

Chip

[ScruffR]

1.5.1-rc1 has just been released - can you (re)test with that?

[chipmc]

@ScruffR,

Sure, but I checked the feature and bugfix list and saw no mention of anything to do with the setPowerConfiguration API.

Chip

[ScruffR]

True, but some of the peripheral changes might play into the issue (e.g. I2C changes or even ublox settings).

Just a stab in the dark, but shouldn’t hurt

[avtolstoy]

When looking into these types of issues, I suggest getting a dump of the actual PMIC state. Something along the lines of:

void loop() {
    {
        PMIC power(true);
        Log.trace("Current PMIC settings:");
        Log.trace("VIN Vmin: %u", power.getInputVoltageLimit());
        Log.trace("VIN Imax: %u", power.getInputCurrentLimit());
        Log.trace("Ichg: %u", power.getChargeCurrentValue());
        Log.trace("Iterm: %u", power.getChargeVoltageValue());

        int powerSource = System.powerSource();
        int batteryState = System.batteryState();
        float batterySoc = System.batteryCharge();
        
        constexpr char const* batteryStates[] = {
            "unknown", "not charging", "charging",
            "charged", "discharging", "fault", "disconnected"
        };
        constexpr char const* powerSources[] = {
            "unknown", "vin", "usb host", "usb adapter",
            "usb otg", "battery"
        };

        Log.trace("Power source: %s", powerSources[std::max(0, powerSource)]);
        Log.trace("Battery state: %s", batteryStates[std::max(0, batteryState)]);
        Log.trace("Battery charge: %f", batterySoc);
    }
    delay(1000);
}

[Rftop]

Interesting Graphs Chip !

All 3 devices exhibit a constant slope during discharging for the diurnal cycle, as expected.
But all 3 devices show an immediate increase in discharging (steeper slope) prior to the failure.
The next morning the PMIC does source power during Daylight (flattens out) , but isn’t recharging the Li-Po.

Have you noticed that this condition happens 2-3 days after startup (typical in your graphs), or was this just a sample and the run-times vary for the issue ?

My first guess was maybe the PMIC’s charge timer (watchdog) was being triggered, but it would appear that the Increased Demand (maybe the modem eventually not sleeping) preempts the Re-Charging Failure?
The demand increasing before the re-charging fails appears to be related in your sample graphs.

[chipmc]

@Rftop,

Very interesting observation. In fact, I see this pattern in a few more devices:

An even in this one where the steeper slope preceded an unusual increase in charge:

I wonder if I could issue a pmic.resetWatchdog() to a device showing this behavior to get it working again until I figure out what the issue is.

Thanks,

Chip

[Rftop]

@chipmc

I wonder if I could issue a pmic.resetWatchdog() to a device showing this behavior to get it working again until I figure out what the issue is.

You can disable it, but I don't think it's the culprit.....since the increased demand precedes the Charging Issue.

[chipmc]

Going to try using the PMIC commands directly and disabling the setPowerConfiguration API.

    conf.feature(SystemPowerFeature::DISABLE);
    System.setPowerConfiguration(conf);

    pmic.begin();
    pmic.setInputVoltageLimit(5080);
    pmic.setInputCurrentLimit(900) ;        
    pmic.setChargeVoltage(4208);            
    pmic.setChargeCurrent(0, 0, 1, 0, 0, 0);
    pmic.enableDPDM();
    pmic.enableCharging();

Will try this code on a few of the devices that are not charging and see if it works. My first target has resumed charging so fingers crossed.

Has anyone else has success with solar power and sleep on the Boron using deviceOS@1.5.0? If so, any chance you could share your code for either PMIC or setPowerConfiguration API?

Thanks,

Chip

[chipmc]

Thank you and I will incorporate this into my test device here in my home office.

I looked for potential solutions to collect this data from remote cellular connected devices but this does not seem to be practical. There does seem to be a couple 3rd party log handlers but based on the time since their last update on GitHub Papertrail (3 years) and ParticleWebLog (16 months). It would be helpful to be able to collect this type of data from select remote devices. Am I missing something?

Thanks,

Chip

[ccunningham]

@chipmc

any word on this chip? I think I’m having the same or a similar issue. It works great for a while then sucks tons of current and doesn’t charge. I’m not sure about the enableDPDM line too, isn’t that really for usb power negotiation? Don’t take my advice though, I’m sitting in my basement w a bunch of bricked borons…

Also did you ever get a solid blue led when you updated to 1.50 or 1.51?

cc

[chipmc]

@ccunningham,

I don’t think we have closed on this yet. I have a support ticket and am getting help from the Particle support folks.

• If you are not going to use a USB cable to charge your Boron, you can leave the DPDM stuff out.
• I think we made some progress on resetting the PMIC in this thread, worth taking a look as the command to reset the power configuration may help you with your Borons:
  New System BatteryCharge API deviceOS@1.5.0

I will keep you posted if I learn anything on a fix.

[ccunningham]

Thanks chip, I’ll run w this for now. but I’ll probs use an external solar charger on my next fielded devices. This would make sense for me, I was sort of pushing the limits with respect to the current limit, I may have burned one board because of this. I basically had unlimited charge current w a 6w panel… so maybe I was asking for this…

[Rftop]

@chipmc, I was looking back at the PMIC datasheet and this caught my eye:

That condition aligns with your discharge graphs:

Can you think of anything in your System that could be getting into a bad state to trigger the Over-Current Protection ?
It’s a long shot, but worth a post.

[chipmc]

@Rftop,

Thanks for coming back to this. There are not any peripherals which, alone could pull so much current that it would trigger current protection. However, these devices are in Rwanda and are therefore the 2G/3G models so, my working theory is that these drops are caused by repeated connection attempts which, when added to the sensors being switched on may drain the battery between successful connection attempts. I have asked my colleague to monitor the devices and look for the 1Hz flashing yellow light but nothing yet. I need to get a 2G/3G Boron for my testing back here where I could hook up a current logger.

More to come on this.

Thanks, Chip

[Rftop]

@chipmc

my working theory is that these drops are caused by repeated connection attempts which, when added to the sensors being switched on may drain the battery between successful connection attempts.

That would seem reasonable, but it doesn't explain the graphs showing No Charging during the failure. You can see the Boron Sourced the Solar Panel for the daytime....it just didn't charge the Li-Po.

On the other hand, if the Panel was producing "close" to the actual demand during the Cellular Failure (x? mW for 3G), then you might have an easy explanation.
I assume the fix would be Manual Mode with a :
if (waitFor(Particle.connected, 60000))
If it doesn't connect within 1 minute, it's time to go to Sleep, EN Pin Shutdown, or at-least stop searching for a while.

On a side note: Are you positive those Boron's are actually getting into Sleep Mode?
I ask only because burning through >20% SOC overnight, every night, seems excessive for a Sleeping Boron.
The LTE (not sure about 3G) Stop Sleep mode is ~ 5mW .
Assuming:
12 hours of darkness = 60 mWh
2,000 mAh Li-Po = 7,400 mWh
That 60 mWh is less than 1% SOC consumed overnight for Sleeping.
I don't know your duty-cycle, but it appears your Borons aren't getting much sleep

[chipmc]

@Rftop,

I like your thinking and your post raises three questions that bear more investigation:

1. When the device is connecting properly, how much charge are these devices loosing over night? To answer this question I looked at ten devices that had we not exhibiting the “not charging” issue. In Rwanda this time of year, it is dark from about 7pm to 5am so I picked a 7 hour period from this dark period and look at 20 days of data across a number of different sensors. On average, these devices lost about 1.5% of their battery capacity each hour over night so, if there are ~12 hours of not great light each night they would loose about 18%. This is a lot but the 2G/3G Boron can draw a lot more current than the LTE - especially if it falls back to 2G if the 3G connection times out. From the Boron datasheet.
   

2. When the device is struggling to connect (cellular provider outages are more common in Rwanda) is the device trying for too long? Here is my block of code for connecting. I am giving the device up to 5 minutes to connect and it can burn a fair bit of juice when it fails. I looked for periods at night where an hour’s connection was missed. In those cases, the state of charge dropped 20% in two hours. Here is the code I am using now and I think I will test dropping the connecting timeout - looking at the console logs and assuming the devices wake on the hour, it typically takes about 90 seconds to connect and send the webhook.

// These functions control the connection and disconnection from Particle
bool connectToParticle() {
  Cellular.on();
  Particle.connect();
  // wait for *up to* 5 minutes
  for (int retry = 0; retry < 300 && !waitFor(Particle.connected,1000); retry++) {
    Particle.process();
  }
  if (Particle.connected()) return 1;                               // Were able to connect successfully
  else return 0;                                                    // Failed to connect
}

2. Are these devices actually going to sleep? Yes, I think they are but since I made the carrier board to be locally assembled and tried to keep costs down, there is no Real Time Clock so I am stuck with STOP mode sleep. I have a feature called “verbose mode” which I can turn on to report state transitions. From watching the console logs, the devices do report transitioning to the “napping” state. The full code for these devices is here.

Now, the good news. It does seem that this issue is much less likely since updating to deviceOS@1.5.0 on the 11th of May. As of this writing, I only have one device with low battery charge and it started charging this morning (after taking a day off). Here are some plots from recent days.

In addition @no1089 is looking into this issue and has been able to replicate the “not charging” state using his own hardware. He has told me that this issue is actively being investigated.

So, thanks in large part to your prompting, I think there are a couple things I need to implement here:

• reduce the timeout period for connections from 5 minutes to 2 or 3.
• Implement a “sleep” state at night to reduce the number of connections these devices need to make during the dark hours.

Going forward, I may look to add a Real Time Clock to these devices to enable HIBERNATE or ENABLE sleep.

Thank you again for your interest in this topic. I appreciate all that you and the community as a whole to do help work through these issues in an open manner.

Chip

[Rftop]

Good info.
The numbers you reported point to ~30-35 mA Average Current (Sleeping + Operating), assuming you are using a 2,000 mAh Li-Po. If so, that seems way too high.

That’s 110 mWh used for 1 hour (1.5% SOC drop).
Again, I haven’t tested the 2G/3G, but I’ll relate it to the LTE.
A typical LTE cellular connection uses 1 mWh. It also uses 5 mWh in Stop Sleep Mode.
You’re at 110 mWh verse 6 mWh, each hour.

Do you have a board locally that you can check the Sleep Current w/ blank code on a 2G/3G, to determine if Sleeping or Operating Current is causing the major impact?
Then you’d know which area to focus your efforts for Code optimization (sleeping vs connection).

The answer could be (as you mentioned) that the 2G/3G is that much more power hungry.
But that’s also saying your 2G/3G requires ~100 mWh for a connection cycle, if it’s actually Sleeping properly.

Seems like somebody here would have some empirical data on 2G/3G current demands.

[chipmc]

@Rftop,

OK, shame on me for not doing this earlier but I just ordered a Boron 2G/3G and will collect some of this data early next week. I will say that, in my experience, an Electron 3G with poor signal strength can draw a lot of current.

As for the resting current draw, these devices have multiple sensors (soil moisture / water pressure / light sensor / temp / humidity) these device can, in-aggregate, draw 50mA of current when they are activated. I will do some testing to better understand the current draw for each device.

More to come.

Chip

[chipmc]

@All,

Good news, this issue has a short term fix while we wait for the next release of deviceOS.

I added this line to the header:

PMIC pmic;

Then, this line to the Setup() section:

pmic.enableBuck();

The credit for this fix (and explanation of why it works) is from @ielec in this post:

Battery Charge doesn't work if starting when EN grounded

What firmware version are you running? This post by @avtolstoy might be a good starting point for you: [Edit] - This may not help in your case, but for Solar Recharged Boron’s I use the USB connector for the input source. I get the same results as you when using Vin(Vusb) pin, but charging will resume during/after an EN Pin Shutdown when using the USB connector. I have no idea why

As always @Rftop provides great perspective. I wanted to repost this fix on this thread so it would be closed. I have deployed this fix and it has worked very well.

@Rftop the excessive energy drain was finally tracked down to an issue with Particle's providers in Rwanda. We were seeing round-trip times of up to 5000mSec and the devices would hang while attempting to connect and authenticate.

Thanks,

Chip