Issues with sleep - Need advice on troubleshooting ideas

All,

I have a sketch I am using with my accelerometer based sensors in the park. All was going well with both counts and stability until I decided to add a “napping” state to the code. The idea was that the device would “nap” during the day between “taps” from the accelerometer. These taps caused D2 to go high which would trigger a hardware interrupt when the device was asleep and would BOTH wake the Electron and trigger an interrupt if the Electron was napping. I differentiate napping from sleeping (which it does at night) as sleeping is DEEP and ends in a reset in the morning.

The full repository for this code is on Github but, as it is over 700 lines long, it is too much to ask for anyone to wade in so I will highlight some items below which I think may be at issue.

First, here is the observed behavior:

  1. The code functions perfectly as long as lowPowerMode is false.
  2. The code functions perfectly for the first hour when put into lowPowerMode - napping and waking as expected.
  3. The device wakes as expected at the top of the hour and sends the data via a webhook, stays awake for 90 seconds (so I can update it if needed) and then starts napping - again just as expected.
  4. After that, the device will wake on tap three times, then it awakes, connects to Particle, sends the 0.8.0rc-4 device health data and becomes unresponsive until the watchdog timer resets it.

So, you don’t have to dig into my code, here are some highlights:

// One line ISR - sensorDetect is defined as volatile
void sensorISR()
{
  sensorDetect = true;                // sets the sensor flag for the main loop
}

// Here is where I pet the watchdog
void watchdogISR()
{
  digitalWrite(donePin, HIGH);                              // Pet the watchdog
  digitalWrite(donePin, LOW);
}

// Here is the napping state code - commenting out the System.sleep() line fixes this issue
  case NAPPING_STATE: {
      if (connectionMode && verboseMode && state != oldState) publishStateTransition();
      if (connectionMode) disconnectFromParticle();                 // If connected, we need to disconned and power down the modem
      watchdogISR();                                                    // Pet the watchdog
      int secondsToHour = (60*(60 - Time.minute()));                    // Time till the top of the hour
      System.sleep(int2Pin, RISING, secondsToHour);             // Sensor will wake us with an interrupt
      delay(30);
      state = IDLE_STATE;                                      // Back to the IDLE_STATE after a nap
  } break;

// Here are the functions this state calls
void publishStateTransition(void)
{
  char stateTransitionString[40];
  snprintf(stateTransitionString, sizeof(stateTransitionString), "From %s to %s", stateNames[oldState],stateNames[state]);
  oldState = state;
  waitUntil(meterParticlePublish);
  Particle.publish("State Transition",stateTransitionString);
  Serial.println(stateTransitionString);
  lastPublish = millis();
}

bool disconnectFromParticle()
{
  Cellular.off();                                           // Turn off the cellular modem
  controlRegister = FRAMread8(CONTROLREGISTER);
  connectionMode = false;
  controlRegister = (0b11101111 & controlRegister);          // Turn off connectionMode
  FRAMwrite8(CONTROLREGISTER, controlRegister);
  return true;
}

I know this is a lot to ask but any help is appreciated on two fronts:

  1. General guidance and advice on how to track these problems down. Serial.print fails when the device sleeps and it is not connected so I can’t publish - tried using Led flashes but need more insight into what is going on.
  2. Comments, advice, criticism of my code is welcome and appreciated. I know I have a lot to learn.

Thank you,

Chip

What exactly are you detecting when the accelerometer is triggered?

Although it’s not exactly addressing your symptoms this issue might be related

Since you are also using attachInterrupt() and stop sleep, this might somehow be caused by the same thing.
You could try detaching the interrupts before entering stop mode sleep.

BTW, if you want to wake on the full hour you can also use something like this.

  const int wakeBoundary = 1*3600 + 0*60 + 0; // 1 hour 0 minutes 0 seconds
  int wakeInSeconds = constrain(wakeBoundary - Time.now() % wakeBoundary, 1 wakeBoundary);
  System.sleep(int2Pin, RISING, wakeInSeconds); 

This ensures that the sleep periode is never too short or even negative (which would cause an “indefinet” sleep) and also allows to alter the boundary down to the second).

1 Like

@RWB,

It is using the tap detection feature to sense a car running over a pneumatic tube across the road. The accelerometer is attached to a small rubber diaphragm that flexes when the car runs over the tube. The rap is detected and the accelerometer raises the interrupt line

Chip

Nice!

Would love to see what that looks like.

Are you measuring traffic in and out or is there just one entrance?

@RWB,

These counters cross the entire entrance so they count both ways.

Here is what is looks like.

Chip

@ScruffR,

Thank you for the link and for the suggestions. I will certainly try detaching the interrupt before sleep.

I will also look at the method to wake at the hour. Always looking for a better way to do things.

Does the Particle premature wake Issue this apply to me? I am suing i2c for the sensor not software serial.

Thanks, Chip

Thanks for the picture.

The device the tube plugs into is totally different than what I was envisioning.

What is that part? A pressure sensor or something custom you built?

As I mentioned in the second post in that issue report, it's not actually ParticleSoftSerial that's causing the problem but the fact that it uses attachInterrupt() and the problem goes away once PSS is ended which does in turn call detachInterrupt().

2 Likes

@ScruffR,

Sorry, missed that point. I can easily detach the interrupt before sleeping and determine what woke the Electron and attach the interrupt. Will see if that makes a difference.

Thanks again, Chip

@RWB,

I am a big fan of not building something if it already exists but, I could not find an inexpensive, low-power, i2c pressure sensor with an interrupt pin. So…

My pressure sensor is cheap and cheerful. I use two brass parts from Lowes - a 1/4 hose barb and a brass fitting. I then hot glue a rubber sheet across the fitting and attach the accelerometer to the rubber diaphragm. It works very well and these simple sensor have counted over 500,000 cars over the past three years.

The accelerometer is very low power, I build my own board so it costs about $5 and it has an interrupt so the Electron can be put to sleep to save power.

Chip

Pretty impressive!

Is that 500,000 cars just for the parks your working with?

How are the parks liking and using this new data about park usage?

@RWB,

That is just in the parks I am monitoring.

The process to collect this data used to be very manual and had limited granularity. I just won another RFP so there is more to come.

Appreciate the interest and support!

Chip

2 Likes

@ScruffR,

OK, back from Peru (and back to sea level!) and was able to give it a try. Detaching the interrupt before sleep and reattaching after sleep does not change the behavior.

To recap:

  1. This code works perfectly until it does an hourly report.
  2. After doing an hourly report, it goes into a weird mode - almost like it cannot read the status of the int2pin. It wakes from sleep on a tap, connects to Particle and then immediately disconnects. I have no idea how it is doing this given the code flow as I understand it.
  3. The device then goes back to sleep with the int2pin HIGH so it does not wake up until the next hour.

Any ideas on troubleshooting would be appreciated.

Full code repository here.

Napping function - where I think the wheels are falling off is here:

  case NAPPING_STATE: {
      if (connectionMode && verboseMode && state != oldState) publishStateTransition();
      stayAwake = debounce;                                           // Ensures that we stay awake long enough to debounce a tap
      if (connectionMode) disconnectFromParticle();                   // If connected, we need to disconned and power down the modem
      watchdogISR();                                                  // Pet the watchdog
      detachInterrupt(int2Pin);                                       // Detach since sleep will monitor the int2Pin
      int wakeInSeconds = constrain(wakeBoundary - Time.now() % wakeBoundary, 1, wakeBoundary);
      System.sleep(int2Pin, RISING, wakeInSeconds);                   // Wake on either int2Pin or the top of the hour
      if (digitalRead(int2Pin)) {                                     // Need to test if Tap or Time woke us up
        awokeFromNap = true;                                          // This flag will allow us to bypass the debounce in the recordCount function
        recordCount();                                                // Count the tap that awoke the device
        stayAwakeTimeStamp = millis();                                // Allows us to ensure we stay awake long enough to debounce
      }
      attachInterrupt(int2Pin,sensorISR,RISING);                      // Reattach Accelerometer interrupt from low to high
      state = IDLE_STATE;                                             // Back to the IDLE_STATE after a nap will come back after the stayAwake time is over
  } break;

Thank you all for your help and advice.

Chip

PS, I liked the suggestion on constraining the sleep seconds. - thank you.

1 Like

Some other things you may need to consider

  • if the pin is already in the state you want to trigger for, the wake interrupt may not work, so wait for the pin to return to passive state before going to sleep
  • I have found some issue with detachInterrupt() which over time tends to barf things up
  • reconnect after sleep depends on the state of the connection prior to sleep, if it was in a limbo state before the system tends to see this as the desired state for next wake :stuck_out_tongue_closed_eyes:
  • while others seem to trust the auto-reconnect, I usually put my own connection monitoring in place and tear down the connection and manually reconnect, when I see odd behaviour

Just some personal views

Over time I came to the conclusion that it’s best to avoid #define for constant definitions wherever possible, due to their “obscure” range of scope, sometimes interfering with otherwise unrelated code.
Hence I’d replace things like this

#define VERSIONNUMBER 9             // Increment this number each time the memory map is changed
#define WORDSIZE 8                  // For the Word size the number of bytes in a "word"
#define PAGESIZE 4096               // Memory size in bytes / word size - 256kb FRAM
#define HOURLYOFFSET 24             // First word of hourly counts (remember we start counts at 1)
#define HOURLYCOUNTNUMBER 4064      // used in modulo calculations - sets the # of hours stored - 256k (4096-14-2)
// First Word - 8 bytes for setting global values
#define VERSIONADDR 0x0             // Where we store the memory map version number
#define SENSITIVITYADDR 0x1         // Sensitivity for Accelerometer sensors
#define DEBOUNCEADDR 0x2            // Where we store debounce in cSec or 1/10s of a sec (ie 1.6sec is stored as 16)
#define RESETCOUNT 0x3              // This is where we keep track of how often the Electron was reset
#define TIMEZONE  0x4               // Store the local time zone data                                    // One byte is open here
#define OPENTIMEADDR 0x5            // Hour for opening the park / store / etc - military time (e.g. 6 is 6am)
#define CLOSETIMEADDR 0x6           // Hour for closing of the park / store / etc - military time (e.g 23 is 11pm)
#define CONTROLREGISTER 0x7         // This is the control register for storing the current state
//Second and Third words bytes for storing current counts
#define CURRENTHOURLYCOUNT 0x8      // Current Hourly Count - 16 bits
#define CURRENTDAILYCOUNT 0xC       // Current Daily Count - 16 bits
#define CURRENTCOUNTSTIME 0xE       // Time of last count - 32 bits
#define HOURLYPOINTERADDR 0x11      // Two bytes for hourly pointer

with an enum that has an explicit scope also allowing for the use of namespaces.

Any other kinds of constants and literals I’d just define as actual const variables

#define HOURLYCOUNTOFFSET 4         // Offsets for the values in the hourly words
#define HOURLYBATTOFFSET 6          // Where the hourly battery charge is stored
// Finally, here are the variables I want to change often and pull them all together here
#define SOFTWARERELEASENUMBER "0.59"

as

const int    HOURLYCOUNTOFFSET     = 4;  // Offsets for the values in the hourly words
const int    HOURLYBATTOFFSET      = 6;  // Where the hourly battery charge is stored
// Finally, here are the variables I want to change often and pull them all together here
const char[] SOFTWARERELEASENUMBER = "0.59";
1 Like

@ScruffR,

First, thank you for taking a look at my code and suggesting a better approach than the #defines. That section now looks like this:

namespace FRAM {                                    // Moved to namespace instead of #define to limit scope
  enum Addresses {
    versionAddr =0x0,                               // Where we store the memory map version number
    sensitivityAddr= 0x1 ,                          // Sensitivity for Accelerometer sensors
    debounceAddr= 0x2,                              // Where we store debounce in cSec or 1/10s of a sec (ie 1.6sec is stored as 16)
    resetCountAddr =0x3 ,                           // This is where we keep track of how often the Electron was reset
    timeZoneAddr = 0x4  ,                           // Store the local time zone data
    openTimeAddr= 0x5 ,                             // Hour for opening the park / store / etc - military time (e.g. 6 is 6am)
    closeTimeAddr =0x6  ,                           // Hour for closing of the park / store / etc - military time (e.g 23 is 11pm)
    controlRegisterAddr =0x7 ,                      // This is the control register for storing the current state
    currentHourlyCountAddr =0x8 ,                   // Current Hourly Count - 16 bits
    currentDailyCountAddr =0xC ,                    // Current Daily Count - 16 bits
    currentCountsTimeAddr =0xE ,                    // Time of last count - 32 bits
  };
};

and when I need one of these values I call it using FRAM::closeTimeAddr. I understand this will reduce the chance of conflict with variable names in libraries and all. This is my first time using a namespace so, I hope I got it write - the code compiles and works anyway.

Now, onto the issue at hand. I read your issue on detachInterrupt() but I am using 0.8.0rc-4 - has this issue been addressed in the 0.8.0 release? I have tried checking the value of the interrupt pin and only going to sleep once it it low but, in the weird state it gets into, these checks don’t seem to work.

You had mentioned not trusting the auto-reconnect and that you team down the connection. What are the steps I need to take in order to try this approach?

Thanks again for all your help,

Chip

1 Like

Since I didn't see this issue getting any attention so far (not even after my bump) I doubt it was addressed yet.
However, if some dev just happened to see and squash that bug without being aware of my issue report, it could still be - worth a try with the code I provided there.

@ScruffR,

Well, I hope this does get addressed.

Looking at your code, it seemed that the change for me to try was to put noInterrupts() before and interrupts() after the disconnectInterrupt() command. I also added a check for the int2Pin before sleeping

case NAPPING_STATE: {
      if (connectionMode && verboseMode && state != oldState) publishStateTransition();
      stayAwake = debounce;                                           // Ensures that we stay awake long enough to debounce a tap
      if (connectionMode) disconnectFromParticle();                   // If connected, we need to disconned and power down the modem
      watchdogISR();                                                  // Pet the watchdog
      noInterrupts();
      detachInterrupt(int2Pin);                                       // Detach since sleep will monitor the int2Pin
      int wakeInSeconds = constrain(wakeBoundary - Time.now() % wakeBoundary, 1, wakeBoundary);
      interrupts();
      if (!digitalRead(int2Pin)) System.sleep(int2Pin, RISING, wakeInSeconds);  // Wake on either int2Pin or the top of the hour
      if (digitalRead(int2Pin)) {                                     // Need to test if Tap or Time woke us up
        awokeFromNap = true;                                          // This flag will allow us to bypass the debounce in the recordCount function
        recordCount();                                                // Count the tap that awoke the device
        stayAwakeTimeStamp = millis();                                // Allows us to ensure we stay awake long enough to debounce
      }
      attachInterrupt(int2Pin,sensorISR,RISING);                      // Reattach Accelerometer interrupt from low to high
      state = IDLE_STATE;                                             // Back to the IDLE_STATE after a nap will come back after the stayAwake time is over
  } break;

Will give this a try.

Thanks,

Chip

1 Like

Update,

No luck. So, here is where I am now:

  1. Goes to sleep and wakes on a hardware pin interrupt - no issues
  2. Goes to sleep and wakes at the hour - does not function as expected
    • Connects to Particle - even though it should not
    • Ignores the state of the int2Pin
    • if the int2Pin is high, ignores the conditional that should prevent going to sleep
    • Goes to sleep with the int2Pin high and therefore cannot wake until the next hour

I am at my wits end with this. At this point, I have to assume there is a bug in the System.sleep(wakeUpPin, edgeTriggerMode, seconds) command or in how it is handing interrupts. I looks like my only option at this point is to give up on sleep which will have a significant impact on battery performance.

If anyone has a suggestion on what else to try, I am all ears.

Thanks, Chip

Hmm, I've not looked for this particular point in your full code, but the NAPPING_STATE does only disconnect when connectionMode == true and hence if it wasn't but a connection was present, then this might play a role

This might be due to a race condition.
Allow for some more time after wake before checking the state.

When using the pin for a RISING edge trigger, the system will implicitly attach the internal pull-down resistor, which - when you don't have pinMode(int2Pin, INPUT_PULLDOWN) in your other code - needs to be removed again after wake.
BTW, checking the pin state after wake isn't a reliable way to actually know whether it was a pin wake or not - there are several threads about this topic.