Boron 2G/3G ULTRA_LOW_POWER, can't get 170uA (720uA instead)

[mtntrx]

I’m launching my first low-power/solar application, and I’ve been reading every thread I could find. Obviously I’m missing something about how to get to 170uA.

I am able to SUCCESSFULLY get there sometimes (when booting with a supply-up) but not able to get there after a button-reset.

Main issue:

• unable to get to 170uA after button-reset in ULTRA_LOW_POWER. (720uA instead)

Side issues:

• Very odd return values from Cellular.isOff() and Cellular.isOn() – sometimes both are false
• ERRORs in the log when booting from supply-up

Setup:

• Device: Boron 2G/3G (tried 2)
• Target: 3.2.0
• Power Source: li-po
• Current Measurement method: HP-DMM in series with lipo battery

My understanding is that DeviceOS should handle shutting the interfaces and getting me to the right low power state with something like:

SystemSleepConfiguration sleepConfig;
sleepConfig.mode(SystemSleepMode::ULTRA_LOW_POWER).duration(SLEEP_TIME);
System.sleep(sleepConfig);

My trimmed back experiment code:

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED);

SerialLogHandler logHandler(LOG_LEVEL_ALL);

#define ACTIVE_MS 10000
#define SLEEP_MS 10000
unsigned long sinceLastSleep;

void setup() {
  delay(5000);
  //Cellular.off();
  delay(5000);
  sinceLastSleep = millis();
}

void loop() {
    if ( ( millis() - sinceLastSleep ) > ACTIVE_MS ) {
        SystemSleepConfiguration sleepConfig;
        sleepConfig.mode(SystemSleepMode::ULTRA_LOW_POWER).duration(SLEEP_MS);
        System.sleep(sleepConfig);
        sinceLastSleep = millis();
    }
}

My current draw during ULTRA_LOW_POWER is 720uA.

What I have tried:

• Instrumenting code with custom LED states
• Connecting to USB and dumping logs for several cycles, live switching to lipo for current measurement during sleep and back to USB to get the next round of logs. Logs:

	0000005288 [system.nm] TRACE: Request to power off the interface. //(when cellular.off() is used)
	0000020289 [system.sleep] TRACE: Entering system_sleep_ext()
	0000020289 [system.sleep] TRACE: Interface 4 is off already
	0000020289 [system.sleep] TRACE: Interface 3 is off already
	…
	0000121129 [system.sleep] TRACE: Entering system_sleep_ext()
	0000121129 [system.sleep] TRACE: Interface 4 is off already
	0000121129 [system.sleep] TRACE: Interface 3 is off already

• Cellular.off() in the setup with 5s delay after. Thought this wasn’t needed as the API should shut this down.
• Booting from supply-up (no reset button) – This generates interesting and inconsistent results. Not sure if this is data or noise.
  Every time I get 150uA (YAY)
  Sometimes I get a clean log showing the modem powering down (and associated blue breathe)
  Sometimes I get a log that loops for 2 minutes on “Deinit modem” … “Modem already off” … “Soft power off modem successfully”
  I get an unexpected TRUE result from
  if( Cellular.isOff() == false && Cellular.isOn() == false ) {

At this point I’m pausing as I feel I might be in the weeds. Any obvious thing I’m doing wrong here?
I could post my fully instrumented code, but that seems to make the initial post to long.

Thanks in advance, and feel free to let me know what

[no1089]

When you reset the device with the button the modem remains on.
Calling cellular.off() should get you to 170uA.

Semi-automatic is tricky in some instances. You need to explicitly tell the system what to do.
You should call cellular.off() before heading to sleep as well.

[mtntrx]

Thanks for the response. I had read that the modem comes on at reset, and I had tried experiments turning it off with no luck (3rd bullet under “things I’ve tried”.) That said, I spent more time messing with the cellular on/off due to your suggestion. I’m still not seeing what either of us expect when including Cellular.off() statements.

Across two borons, I see what looks like the radio ON after a supply-up reset and the radio OFF after a button reset (assuming it was off before the button was pressed) – supporting data here are both logs and current measurements.

Unfortunately I see the 145uA target met after the supply-up reset, but I get >700uA after a button-press reset

I still need help matching up the docs/expectations to real life. Are my Cellular.off() commands ok? Is my ULTRA_LOW_POWER sleep config ok? Any other ideas why I’m seeing elevated sleep current?

data and code:
used two DMMs, used two borons, used two batteries at different states of charge >3.8V

TEST1: Power removed then applied; no reset button; from battery; no serial log

• 40-135 mA pulled for ~ 13-17 secs // corresponds to INIT1_MS, feels like a radio is ON
• 4-5mA pulled for ~ 7-10 secs // corresponds to ACTIVE_MS plus some sleep prep time
• 145uA pulled for 11 secs // corresponds to SLEEP_MS

TEST2: Power removed then applied; no reset button; from usb; serial log enabled; current sampled with hard switchover to battery during sleep

// likely corresponds to the INIT1_MS 13s deli
0000013406 [system.nm] TRACE: Request to power off the interface
0000013406 [net.pppncp] TRACE: NCP event 3
0000013406 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_POWERING_DOWN
0000013407 [ncp.client] TRACE: Try powering modem off using AT command
0000013408 [ncp.client] ERROR: NCP client is not ready
0000013408 [ncp.client] TRACE: Setting UART voltage translator state 0
0000013409 [ncp.client] TRACE: Powering modem off using hardware control
0000013410 [ncp.client] TRACE: Setting UART voltage translator state 0
// ~4sec of system time spent powering things down?
0000017285 [net.pppncp] TRACE: NCP event 3
0000017285 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_DOWN
0000017285 [system.nm] TRACE: Interface 4 power state changed: 1
0000017286 [ncp.client] TRACE: Modem powered off
0000017286 [ncp.client] TRACE: Deinit modem serial.
// ~7sec of ACTIVE_MS time
0000024289 [system.nm] TRACE: Request to power off the interface
0000024289 [system.sleep] TRACE: Entering system_sleep_ext()
0000024290 [system.sleep] TRACE: Interface 4 is off already
0000024291 [system.sleep] TRACE: Interface 3 is off already
// ~11s of sleep + 7s of ACTIVE_MS time 
0000042458 [system.sleep] TRACE: Entering system_sleep_ext()
0000042458 [system.nm] TRACE: Request to power off the interface
0000042459 [system.sleep] TRACE: Interface 4 is off already
0000042459 [system.sleep] TRACE: Interface 3 is off already
// repeats. (in two cycles I hot swapped to batter and confirmed sleep current at 145uA

TEST3 reset via button; from battery; no serial log

• 5mA pulled for ~22s. // corresponds to INIT1_MS(13s) and one “soft delay” round at ACTIVE_MS (7s)
• 700-900 uA pulled for ~11 secs. // corresponds to one SLEEP_MS (11s)
• // repeats

TEST4: reset via button; powered by usb; serial log enabled; current sampled with hard switchover to battery during sleep

// 13s of INIT1_MS
0000013285 [system.nm] TRACE: Request to power off the interface
// 7s of ACTIVE_MS
0000020285 [system.sleep] TRACE: Entering system_sleep_ext()
0000020285 [system.nm] TRACE: Request to power off the interface
0000020286 [system.sleep] TRACE: Interface 4 is off already
0000020287 [system.sleep] TRACE: Interface 3 is off already
// 11s of SLEEP_MS + 7s of ACTIVE MS
0000038455 [system.sleep] TRACE: Entering system_sleep_ext()
0000038455 [system.nm] TRACE: Request to power off the interface
0000038555 [system.sleep] TRACE: Interface 4 is off already
0000038556 [system.sleep] TRACE: Interface 3 is off already
// repeats  -- hot switch over to battery during sleep **confirms 700uA**

My code:

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED);

SerialLogHandler logHandler(LOG_LEVEL_ALL);

#define INIT1_MS  13000
#define CELL_OFF_TIMEOUT  120000
#define ACTIVE_MS 7000
#define SLEEP_MS 11000
unsigned long sinceLastSleep;

void setup() {
  delay(INIT1_MS);  // 13s

  Cellular.off();
  waitFor( Cellular.isOff, CELL_OFF_TIMEOUT);

  sinceLastSleep = millis();
}

void loop() {

    // soft delay (ACTIVE_MS or 11s)
    if ( ( millis() - sinceLastSleep ) > ACTIVE_MS ) {


        // try this each loop just in case
        Cellular.off();
        waitFor(Cellular.isOff, CELL_OFF_TIMEOUT);

        SystemSleepConfiguration sleepConfig;
        sleepConfig.mode(SystemSleepMode::ULTRA_LOW_POWER).duration(SLEEP_MS);
        System.sleep(sleepConfig);

        sinceLastSleep = millis();

    }
}

[mtntrx]

UPDATE: I think I’ve figured something out, and I do think I see two issues with the hardware and/or firmware at reset.

ISSUE1: A button reset puts the radio in a state where it CAN’T get to 150uA on sleep UNLESS you do a Cellular.on() followed by a Cellular.off().

ISSUE2: A power-on reset (no button) can leave the radio in a funny state. It can respond to BOTH .isOff() and .isOn() with false. When you call Cellular.off() you sometimes get into a ~2min loop of “modem is already off” … “Setting UART v trans to 0” … “Soft power off modem success” … “Deinit modem serial” … “modem already off”

To reproduce: Boron, target 3.2.0, use similar code as below

ISSUE1 reproduce:

1. Power up the device with the supply (no reset button)
2. Do a Cellular.off() in the setup function
3. Go to ULTRA_LOW_POWER with .duration only
4. measure current: ~150uA
5. wake - Cellular.off() - sleep - measure - repeat. // always 150uA
6. hit reset button
7. same setup code with Cellular.off()). same sleep command ULTRA_LOW_PWER with .duration
8. measure >700uA
9. wake - cellular.off() - sleep - measure - repeat // always >700uA
10. do a cellular.on() // I did this based on an external pin conditional
11. do a cellular.off()
12. sleep to ULTRA_LOW_POWER with .duration. // 150uA
13. wake - cellular.off() - sleep - measure - repeat. // 150uA

ISSUE2 reproduce (sometimes):

1. power up device with supply (no reset button)
2. check .isOff and .isOn and flag error if both are false
3. Cellular.off(); waitFor( Cellular.isOff, 120000 )
4. Sometimes see one of these odd modes
   a. looping log (included below)
   b. errors and long delays while trying to shut off modem

Whatever the reset button does puts the radio into a state where it doesn’t think it is on, but it sure is burning more current at sleep time. Cellular.off() DOES NOT address it unless you do a Cellular.on() first.

thoughts?

code

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED);
SerialLogHandler logHandler(LOG_LEVEL_ALL);

#define INIT1_MS  5000
#define CELL_OFF_TIMEOUT  120000
#define ACTIVE_MS 10000
#define SLEEP_MS 20000
#define CONNECT_CELL_PIN D12
#define HEARTBEAT_RATE 1000

unsigned long sinceLastSleep = millis();
unsigned long lastHeartBeat = millis();

void setup() {

  pinMode( CONNECT_CELL_PIN, INPUT_PULLUP);

  delay(2000);  // wait 2s of the INIT1_MS before sending log info over serial
  Log.info( "init delay %ds", INIT1_MS );
  delay(INIT1_MS - 2000);

  // check for this weird, unexpected state
  if ( Cellular.isOff() == false && Cellular.isOn() == false ) {
    Log.error("ERROR: Evaluated true >>> ( Cellular.isOff() == false && Cellular.isOn() == false )");
    Log.error("ERROR: Both .isOff() and .isOn() should NOT be false at the same time");
  }

  // do an initial Cellular.off()
  Log.info( "Turning off cellular, and waiting for .isOff()" );
  Cellular.off();
  waitFor( Cellular.isOff, CELL_OFF_TIMEOUT);
  Log.info( "cellular should be off now" );

  // used for a "soft delay"
  sinceLastSleep = millis();

}

void loop() {
    // soft delay (ACTIVE_MS or 7s)
    if ( ( millis() - sinceLastSleep ) > ACTIVE_MS ) {

	// check D12, if low, do a connect (be careful not to get banned from the network)
    	if ( digitalRead( CONNECT_CELL_PIN ) == LOW ) {
    		Log.info( "CELL_CONNECT_PIN(%d) is LOW, Cell connect is desired ",CONNECT_CELL_PIN );
        	Log.info( "Turning Cellular on and waiting for .isON()" );
  		Cellular.on();
  		waitFor(Cellular.isOn, CELL_OFF_TIMEOUT);
      		Log.info( "Cellular should now be on, mabye not fully network connected though" );
      		Log.info( "Will wait 90s for full connection");
		delay(90000);
    	}


       	// try this each loop just in case
        Log.info( "Turning off cellular, and waiting for .isOff()" );
  	Cellular.off();
  	waitFor(Cellular.isOff, CELL_OFF_TIMEOUT);
      	Log.info( "Cellular should now be OFF" );

	// ULTRA_LOW_POWER config and command
        Log.info( "Going to sleep for %d", SLEEP_MS );
    	SystemSleepConfiguration sleepConfig;
    	sleepConfig.mode(SystemSleepMode::ULTRA_LOW_POWER).duration(SLEEP_MS);
    	System.sleep(sleepConfig);

        // reset soft delay timer
        sinceLastSleep = millis();

    }

    // log a heartbeat
    if ( millis() - lastHeartBeat  > HEARTBEAT_RATE ) {
       	Log.info( "Awake, next sleep in %d sec", (int)((ACTIVE_MS - ( millis() - sinceLastSleep )) / 1000 ));
	lastHeartBeat = millis();
    }

}

[rickkas7]

Issue #1 is almost certainly true. The reason this happens is that at boot the modem is slightly on. But in order to turn it off, it needs to be powered on so it can respond to the AT commands to turn it off. But because it was never turned on, Cellular.off assumes it’s off.

Under normal circumstances this is not an issue because usually you want to connect after power-up. After that, sleep works normally. If you have a special use case where you want to be able to go into full power down immediately after cold boot you’ll probably need to power up the modem first before turning it off. This is not necessary when waking from sleep, because the modem will already be really powered off.

Issue #2 is not exactly expected and could be a bug.

[rickkas7]

I happened to be testing sleep code today. This is Device OS 3.1.0 on a Boron LTE (BRN402), with ULP sleep with time and GPIO wake. I just let the sleep code power down the modem after connecting and I’m getting 119 µA.

[mtntrx]

@rickkas7 Thanks for the additional insight. I think you have a few problems here. I won’t pound on it too hard, but please take them into consideration.

1. The half-on state of the modem: I don’t think this fully explains what is happening as I got the 170uA after a power-applied boot but NOT after the following reset button boot. Something unexpected is happening on that button-reset that puts it into a different state than the power-up reset.
2. The half-on state is a problem for low power and solar solutions. If the device powers up (or resets) with a critical battery, it needs to be able to get to 170uA sleep promptly without a full connect that can burn more power or fail due to low battery.
3. The documentation suggests all reset operations are the same ( System.reset(), button, power-up), but I think they are not.

I’ve got the understanding I need to do my best here. Again, thanks for the additional insight that got me there. Still, I think you guys have a few things to work on here.

Thanks again

[rickkas7]

I was unable to reproduce your problem. Tested with a Boron 2G/3G (BRN310) with Device OS 2.3.0.

Here’s the firmware I used.

#include "Particle.h"

SYSTEM_THREAD(ENABLED); // enabled means connecting to cloud is non-blocking
SYSTEM_MODE(SEMI_AUTOMATIC);    // device decides when to start cloud connection

Serial1LogHandler logHandler(115200, LOG_LEVEL_TRACE);

pin_t WAKE_PIN = D2;

retained int wakeCounter = 0;
system_tick_t lastSleep = 0;

void sleepFunction(void);

void setup() {
    pinMode(WAKE_PIN, INPUT_PULLUP);
}

void loop() {

    if (millis() - lastSleep >= 60000) {
        // Time to sleep again
        lastSleep = millis();

        sleepFunction();
    }
}

void sleepFunction(void) {

    {   
        // Copied from Tracker Edge code
        Log.info("Stopping modem");
        if (!Particle.disconnected()) {
            // Explicitly disconnect from the cloud with graceful offline status message
            Particle.disconnect(CloudDisconnectOptions().graceful(true).timeout(5000)); // 5 seconds
            waitUntil(Particle.disconnected);
            Cellular.disconnect();
            waitUntilNot(Cellular.ready);
        }

        // Turn off modem so it won't be turned on again after wake
        Cellular.off();
        waitUntil(Cellular.isOff);
    }
        
    SystemSleepConfiguration config;
    config.gpio(WAKE_PIN, FALLING)
          .duration(60s)
          .mode(SystemSleepMode::ULTRA_LOW_POWER);    // ULP sleep mode
                
    Log.info("going to sleep");
    System.sleep(config);

    lastSleep = millis();

    if (++wakeCounter >= 5) {
        wakeCounter = 0;
    }
    Log.info("wakeCounter=%d", wakeCounter);
    if (wakeCounter == 4) {
        Particle.connect();
    }

}

It more or less stays awake for a minute, then sleeps for a minute. It starts off with cellular off. Even after cold boot and going to sleep, I’m getting 161 uA, so turning the modem off on cold boot seems to work properly for me.

It then wakes up with cellular off (breathing white). In this state I’m getting 5.12 mA average, which is expected. Upon going into ULP sleep with GPIO wake I’m getting 157 uA, which is also expected.

It then repeats this cycle a few more times until wakeCounter ==4 then it connects to cellular. After entering sleep after connecting, I got 159uA.

One difference worth noting: On cold boot, there is a power difference when in breathing white mode. It’s 76.6 mA, instead of 5.12 mA. I believe this is because the code to shut down the cellular modem has not yet been executed yet and the modem is partially powered on. It could probably be fixed by executing the cellular off code in this case.

I also tested hitting the reset button while in sleep mode. This caused the Boron to go into breathing white mode (expected) at 5.4 mA. This is expected, because the modem was previously off. Resetting the nRF52840 does not reset the cellular modem automatically, so this is expected. Once the device went back to sleep it used 155 uA, expected.

Power usage was always around 160 uA in ULP sleep in all of the cases I tested with the code above.

[mtntrx]

I will work with your code a bit and see what I get.

Can you try this on 3.2.0? That is the target I was using as I thought it might be preferred. I can try a downgrade as well.

Thanks

[mtntrx]

Were you able to test on 3.2.0? I think I’m seeing diffs.

possible ISSUE #1: different deep sleep currents in 3.2.0 vs 2.3.0 AFTER reset button is pressed
To reproduce: Use your code unchanged, power up without RST button, press RST button at 280 seconds in, plot currents along the way, repeat on 3.2.0 part and 2.3.0 part. Observe sleep current difference.

possible ISSUE #2: No difference between version here. Device consumes different current (10x) before and after deep sleep. Might not be seen as a bug, but would be good to know where the extra current is going after supply power up, and if that can be addressed with power management calls other than going to sleep.

[rickkas7]

I did try 3.2.0 and noticed an issue after pressing the reset button while in sleep mode (issue 1). Sometimes it was 160 uA, but sometimes I got 720 uA but I didn’t have time to troubleshoot it.

It seems like Issue #2 is the high power after cold boot with cellular off. I ran into that with both 2.3.0 and 3.2.0. After the first sleep the power consumption seems normal so this seemed like an easily worked around issue.