Fault tolerance and sleep mode weird behavior [SOLVED]

Hello guys.
This is my first post here but I’m using particle products for 3 years now.
I ran into a weird behavior while testing electron sleep mode along with the wonderful library of fault tolerance by @rickkas7.

The code I test is provided below. Basically it’s the example from the library + Going to sleep for 10 seconds and waking up again.

What happens is that it will operate ok but every now and then it would fall to sleep so deep that it just won’t recover from it. I needed to take the battery out and re-plug it to make it work back.

What could be the reason, and what could help it?
Thanks,
Ziv

// Electron Sample Application for fault tolerance and problem debugging techniques
// Requires system firmware 0.6.1 or later!
//
// Original code location:
// https://github.com/rickkas7/electronsample

#include "Particle.h"

#include "AppWatchdogWrapper.h"
#include "BatteryCheck.h"
#include "ConnectionCheck.h"
#include "ConnectionEvents.h"
#include "SessionCheck.h"
#include "Tester.h"

// If you are using a 3rd party SIM card, put your APN here. See also
// the call to Particle.keepAlive in setup()
// STARTUP(cellular_credentials_set("YOUR_APN_GOES_HERE", "", "", NULL));

SerialLogHandler logHandler;

// We use retained memory keep track of connection events; these are saved and later uploaded
// to the cloud even after rebooting
void startup() {
	System.enableFeature(FEATURE_RETAINED_MEMORY);
	System.enableFeature(FEATURE_RESET_INFO);
}
STARTUP(startup);

// System threaded mode is not required here, but it's a good idea with 0.6.0 and later.
// https://docs.particle.io/reference/firmware/electron/#system-thread
SYSTEM_THREAD(ENABLED);

// SEMI_AUTOMATIC mode or system thread enabled is required here, otherwise we can't
// detect a failure to connect
// https://docs.particle.io/reference/firmware/electron/#semi-automatic-mode
SYSTEM_MODE(SEMI_AUTOMATIC);


// Manage connection-related events with this object. Publish with the event name "connEventStats" and store up to 32 events
// in retained memory. This provides better visibility into what your Electron is using but doesn't use too much data.
ConnectionEvents connectionEvents("connEventStats");

// Check session by sending and receiving an event every hour. This can help troubleshoot problems where
// your Electron is online but not communicating
SessionCheck sessionCheck(3600);

// Monitors the state of the connection, and sends this data using the ConnectionEvents.
// Handy for visibility.
ConnectionCheck connectionCheck;

// Tester adds a function that makes it possible exercise some of the feature remotely using functions.
// testerFn is the function and and the second parameter that's a pin to test pin sleep modes.
Tester tester("testerFn", D2);

// BatteryCheck is used to put the device to sleep immediately when the battery is low.
// 15.0 is the minimum SoC, if it's lower than that and not externally powered, it will
// sleep for the number of seconds in the second parameter, in this case, 3600 seconds = 1 hour.
BatteryCheck batteryCheck(15.0, 3600);

// This is a wrapper around the ApplicationWatchdog. It just makes using it easier. It writes
// a ConnectionEvents event to retained memory then does System.reset().
AppWatchdogWrapper watchdog(60000);

//
//
//
void setup() {
	//
	Serial.begin(9600);

	// If you're battery powered, it's a good idea to enable this. If a cellular or cloud connection cannot
	// be made, a full modem reset is first done. If that doesn't resolve the problem, on the second and
	// subsequent failures, the Electron will sleep for this many seconds. The intention is to set it to
	// maybe 10 - 20 minutes so if there is a problem like SIM paused or a network or cloud failure, the
	// Electron won't continuously try and fail to connect, depleting the battery.
	connectionCheck.withFailureSleepSec(15 * 60);

	// We store connection events in retained memory. Do this early because things like batteryCheck will generate events.
	connectionEvents.setup();

	// Check if there's sufficient battery power. If not, go to sleep immediately, before powering up the modem.
	batteryCheck.setup();

	// Set up the other modules
	sessionCheck.setup();
	connectionCheck.setup();
	tester.setup();

	// We use semi-automatic mode so we can disconnect if we want to, but basically we
	// use it like automatic, as we always connect initially.
	
}

void loop() {
	Particle.connect();
    waitFor(Particle.connected, 20*60000);
	
	batteryCheck.loop();
	sessionCheck.loop();
	connectionCheck.loop();
	connectionEvents.loop();
	tester.loop();
	
	// 7. Going to sleep

	// It's necessary to turn on the cellular modem in MANUAL or SEMI_AUTOMATIC mode
	// before going to sleep. This happens because the modem is put to sleep using AT
	// commands, and they don't work when the modem is not on.
	Cellular.on();

	// This workaround appears to be necessary when using SYSTEM_THREAD(ENABLED).
	// The reason is that .on() and .off() are asynchronous in system threaded mode
	// and if the calls don't complete, sleep mode is not properly entered.
	delay(15000);

	// The Cellular.off() requirement is probably a bug. I think it's because when
	// going to sleep in system threaded mode, it's not waiting for the modem to actually
	// go to sleep. As of 0.6.0 it's necessary, otherwise you drop in a weird mode that
	// ends up using 2.8 to 60 mA instead of 132 uA.
	Cellular.off();
	delay(1000);

	// Resulted in 132 uA power consumption, as it should

	System.sleep(SLEEP_MODE_DEEP, 10); // System.sleep(SLEEP_MODE_DEEP, secDifference); //secDifference not including time to enter sleep mode 

	// This part not reached; sleep mode deep starts over again after setup()
}

What have you got connected to WKP?
There is a HW flaw in the STM32 that prevents the device from being woken up from deep sleep when WKP is held HIGH at the moment of entering deep sleep.

1 Like

I believe it’s disconnected, should I tie it to ground?

Having the pin floating may cause issues, so adding a 10k pull-down resistor would at least take that off the list of possible causes.

Thank you for the suggestion.
I’ve connected 10k resistor between WKP and GND. The same behavior I described initially kept happening.
Maybe you have more suggestions?
Thank you,
Omri

Hi again,
I decided to post the code (with the things that don’t work) that I’m using and it’s still having problems.
More specifically: I’m setting daily alarmtimes, in these times, the electron should wake up and:

  1. wait for transmittion from LoRa RF device - works most of the time and if it doesn’t it gets a timeout
    2. Initiate manually cloud connection - works part time
  2. Take measurement from a sensor - works all the time
  3. Gathering the data to be sent to the cloud (basically measure the battery) - works all the time
  4. Publishing to the Cloud - I think it should work all the time but it’s dependent on section 2
    6. Real time calculation for setting the alarm - I think it’s ok but the electron wakes up 2 from 6 of the alarm times in the next day.
  5. Going to sleep - works all the time

Am I doing something wrong?

// This #include statement was automatically added by the Particle IDE.
#include <electronsample.h>

// This #include statement was automatically added by the Particle IDE.
#include <SparkIntervalTimer.h>

// This #include statement was automatically added by the Particle IDE.
#include "MedianFilter.h"



// Connection to the cloud
/////////////////////////////////////////////////////////
/*

// If you are using a 3rd party SIM card, put your APN here. See also
// the call to Particle.keepAlive in setup()
// STARTUP(cellular_credentials_set("YOUR_APN_GOES_HERE", "", "", NULL));

// We use retained memory keep track of connection events; these are saved and later uploaded
// to the cloud even after rebooting
STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));

SYSTEM_THREAD(ENABLED);
// SEMI_AUTOMATIC mode or system thread enabled is required here, otherwise we can't
// detect a failure to connect
// https://docs.particle.io/reference/firmware/electron/#semi-automatic-mode
SYSTEM_MODE(SEMI_AUTOMATIC);
*/


//Bin ID and params
char own_lon[] = "11.9211"; 
char own_lat[] = "57.6978";

char far_lon[] = "35.201329"; 
char far_lat[] = "31.753092";

int customer_id = 4;
int own_bin_id = 51; //The particle
int far_bin_id = 52; //The arduino

unsigned long howLongToWait = 600000; //10 minutes timeout for LoRa connection
//unsigned long howLongToWait = 60000;

//RTC
//int alarmTimes = 7; //5;
//int Hour[]  =  {0, 4, 8, 12, 15, 17, 20};//{ 9, 17, 17, 17, 16};
//int Minute[] = {0, 0, 0,  0,  0,  0,  0};//{ 13, 15, 21, 27, 11};
int alarmTimes = 6;
int Hour[]  =  {0, 4, 8, 12, 16,  20};//{ 9, 17, 17, 17, 16};
int Minute[] = {0, 0, 0, 0,   0,   0};//{ 13, 15, 21, 27, 11};
int alarm = 0;
int secDifference = -1;

//Interval Timer
long updateInterval = 1000; 
IntervalTimer updateTimer;


//EEPROM
int eeAddress1 = 1;   //fullness - Location we want the data to be put.
int eeAddress2 = 101; //battery - Location we want the data to be put.
int eeAddress3 = 1001; //counter good connections.
int eeAddress4 = 10001; //counter good connections.
int counter;
int bug;




int LED_BUILTIN = D7; // This is the LED that is already on your device.



// Connection to the cloud
/////////////////////////////////////////////////////////

////////////////////////////////////////////////////
//fault tolerance:
//   #include "AppWatchdogWrapper.h"
// #include "BatteryCheck.h"
#include "ConnectionCheck.h"
// #include "ConnectionEvents.h"
// #include "SessionCheck.h"
// #include "Tester.h"

// If you are using a 3rd party SIM card, put your APN here. See also
// the call to Particle.keepAlive in setup()
// STARTUP(cellular_credentials_set("YOUR_APN_GOES_HERE", "", "", NULL));

SerialLogHandler logHandler;

// We use retained memory keep track of connection events; these are saved and later uploaded
// to the cloud even after rebooting
void startup() {
	System.enableFeature(FEATURE_RETAINED_MEMORY);
	System.enableFeature(FEATURE_RESET_INFO);
}
STARTUP(startup);

// System threaded mode is not required here, but it's a good idea with 0.6.0 and later.
// https://docs.particle.io/reference/firmware/electron/#system-thread
SYSTEM_THREAD(ENABLED);

// SEMI_AUTOMATIC mode or system thread enabled is required here, otherwise we can't
// detect a failure to connect
// https://docs.particle.io/reference/firmware/electron/#semi-automatic-mode
SYSTEM_MODE(SEMI_AUTOMATIC);


// Manage connection-related events with this object. Publish with the event name "connEventStats" and store up to 32 events
// in retained memory. This provides better visibility into what your Electron is using but doesn't use too much data.
//ConnectionEvents connectionEvents("connEventStats");

// Check session by sending and receiving an event every hour. This can help troubleshoot problems where
// your Electron is online but not communicating
// SessionCheck sessionCheck(3600);

// Monitors the state of the connection, and sends this data using the ConnectionEvents.
// Handy for visibility.
ConnectionCheck connectionCheck;

// Tester adds a function that makes it possible exercise some of the feature remotely using functions.
// testerFn is the function and and the second parameter that's a pin to test pin sleep modes.
// Tester tester("testerFn", D2);

// BatteryCheck is used to put the device to sleep immediately when the battery is low.
// 15.0 is the minimum SoC, if it's lower than that and not externally powered, it will
// sleep for the number of seconds in the second parameter, in this case, 3600 seconds = 1 hour.
//BatteryCheck batteryCheck(15.0, 3600);

// This is a wrapper around the ApplicationWatchdog. It just makes using it easier. It writes
// a ConnectionEvents event to retained memory then does System.reset().
//AppWatchdogWrapper watchdog(60000);


/////////////////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////////////////

void setup() {
    
//When using SYSTEM_THREAD(ENABLED) you must be careful of when you register your variables. 
//At the beginning of setup(), before you do any lengthy operations, delays, or things like waiting for a key press, is best.
//   Particle.connect();
//   while (!Particle.connected());

  
  Serial.begin(9600); // Open serial monitor at 115200 baud to see ping results.
  //delay(500);
 
  // AUTO allocate updateFlag to run every 1000ms using hmSec timescale (1000 * 0.5ms period)
  // On Core the first allocated timer is TIMER2.  On Photon, it is TIMER3.
  updateTimer.begin(updateFlag, 1000, hmSec);
  
  
  ////////////////////////////////////////////////////
  //fault tolerance:
  
  	// If you're battery powered, it's a good idea to enable this. If a cellular or cloud connection cannot
	// be made, a full modem reset is first done. If that doesn't resolve the problem, on the second and
	// subsequent failures, the Electron will sleep for this many seconds. The intention is to set it to
	// maybe 10 - 20 minutes so if there is a problem like SIM paused or a network or cloud failure, the
	// Electron won't continuously try and fail to connect, depleting the battery.
	connectionCheck.withFailureSleepSec(4 * 60);

	// We store connection events in retained memory. Do this early because things like batteryCheck will generate events.
//	connectionEvents.setup();

	// Check if there's sufficient battery power. If not, go to sleep immediately, before powering up the modem.
//	batteryCheck.setup();

	// Set up the other modules
// 	sessionCheck.setup();
	connectionCheck.setup();
// 	tester.setup();

	// We use semi-automatic mode so we can disconnect if we want to, but basically we
	// use it like automatic, as we always connect initially.
}

void blinkLED()
{
  static bool LedStatus = LOW;

  digitalWrite(LED_BUILTIN, LedStatus);
  LedStatus = !LedStatus;
}

void updateFlag(void) {
	uFlag = HIGH;
}



void loop() {

/*


If 10 minutes have passed OR new Lora data was received continue
2. Particle Electron Internet Connection (This might take some time so we start it before the sensor)

4. Gathering the data to be sent to the cloud: fullness & battery for local and far bins
5. Publishing to the Cloud
6. Real time calculation for setting the alarm
7. Going to sleep

*/



  
 

// 2. Particle Electron Internet Connection
//      batteryCheck.loop();
// 	sessionCheck.loop();
//	connectionCheck.loop();
//	connectionEvents.loop();
// 	tester.loop();

    
	Particle.connect();
    waitFor(Particle.connected, 3*60000);
	

	Serial.print("time local: "); Serial.println(Time.local());
	if (Particle.connected()){
     Serial.println("hoorah");
     EEPROM.get(eeAddress3, counter);
     counter++;
     EEPROM.put(eeAddress3, counter);
} else {
     Serial.println("not connected!");
          EEPROM.get(eeAddress4, bug);
     bug++;
     EEPROM.put(eeAddress4, bug);
}
     Serial.print("counter: "); Serial.println(counter);
     Serial.print("bug: "); Serial.println(bug);
 
/*
   Particle.connect();
    waitFor(Particle.connected, 20*60000);  // this won't be necessary when 0.6.1 is released // waitUntil(Particle.connected(), 60000*10); //assuring connection before sending data. if not connected waite 10 minutes more before let it go
  // while (!Particle.connected());

    /////////////////////////////////////////////////////////
  */  
    

  
// 4. Gathering the data to be sent to the cloud: fullness & battery for local and far bins	  
	//own_fullness.number = (float) distance_filtered;
	//EEPROM.get(eeAddress1, far_fullness.number);
	//EEPROM.get(eeAddress2, far_battery.number);
  
	//Show battery level
	FuelGauge fuel;
	own_battery.number = fuel.getSoC();


	//battery_level = fuel.getSoC();
	//String battery = "{\"battery_fullness\": \"" + String(battery_fullness) + "\"}";
	//delay(10000);

	
// 5. Publishing to the Cloud

//New JSON format, triggering the "batbinData" webhook
String own_data = String::format("{ \"id\": %d, \"fullness\": %d, \"battery_fullness\": %f}", own_bin_id, distance_filtered, own_battery.number);
String far_data = String::format("{ \"id\": %d, \"fullness\": %d, \"battery_fullness\": %f}", PacketRX.sensor.sensorId, PacketRX.sensor.fullness, PacketRX.sensor.battery_level);

if (uFlag){
	if (!GRAPH){
	   Serial.print("publishing: ");
	   Serial.println(own_data);
	   Serial.println(far_data);
	}

	if (waitFor(Particle.connected, 90000)) {
		
		Particle.publish("batbinData",own_data,60,PRIVATE); //Particle.publish("binData",own_data,60,PRIVATE);
		delay(2000);
	if (new_message_flag)	Particle.publish("batbinData",far_data,60,PRIVATE); //Particle.publish("binData",far_data,60,PRIVATE);
		
		uFlag = LOW;
		blinkLED();
	}
}  
  


// 6. Real time calculation for setting the alarm
/*
int alarmTimes = 6;
int Hour[]  =  {0, 4, 8, 12, 16,  20};
int Minute[] = {0, 0, 0, 0,   0,   0};
int alarm = 0;
int secDifference = -1;
*/
while (secDifference < 0){
	Time.zone(+2);
	int secNow = Time.local() % 86400; 
	Serial.print("time local: "); Serial.println(Time.local());
	// for 18:23:00
	int secThen = Hour[alarm]*3600 + Minute[alarm]*60; 
	//check if this time past
	secDifference = secThen - secNow; 
	Serial.print("alarm: "); Serial.print(" = "); Serial.println(alarm); 
	Serial.print("Next alarm //only the last print: "); Serial.print(Hour[alarm]); Serial.print(":"); Serial.println(Minute[alarm]);
	if (alarm == alarmTimes){
	//adding one day (86400 seconds) when list back to first alarm
	secDifference = Hour[0]*3600 + Minute[0]*60 - secNow + 86400;
	}
	alarm += 1;
}

//Serial.print("\tdistance_filtered: ");  Serial.println(distance_filtered);
Serial.print("secDifference: "); Serial.print(" = "); Serial.println(secDifference);




// 7. Going to sleep

	// It's necessary to turn on the cellular modem in MANUAL or SEMI_AUTOMATIC mode
	// before going to sleep. This happens because the modem is put to sleep using AT
	// commands, and they don't work when the modem is not on.
	Cellular.on();

	// This workaround appears to be necessary when using SYSTEM_THREAD(ENABLED).
	// The reason is that .on() and .off() are asynchronous in system threaded mode
	// and if the calls don't complete, sleep mode is not properly entered.
	delay(15000);

	// The Cellular.off() requirement is probably a bug. I think it's because when
	// going to sleep in system threaded mode, it's not waiting for the modem to actually
	// go to sleep. As of 0.6.0 it's necessary, otherwise you drop in a weird mode that
	// ends up using 2.8 to 60 mA instead of 132 uA.
	Cellular.off();
	delay(1000);

	// Resulted in 132 uA power consumption, as it should
//    System.sleep(SLEEP_MODE_DEEP, 10); 
	System.sleep(SLEEP_MODE_DEEP, secDifference); //secDifference not including time to enter sleep mode 

	// This part not reached; sleep mode deep starts over again after setup()
	
}


Unless you really want to sleep more than 24 hours you want another modulo operation there

  secDifference = (Hour[0]*3600 + Minute[0]*60 - secNow + 86400) % 86400;

Thank you so much! that did the trick :slight_smile: