Unexpected lockup of Photon with SparkFun PowerShield

[frlobo]

Well. I tried this and it seems that the power up worked for a week, After that the system went to a stall mode, where the photon needs reset in order to work again.

So the fix… does not work. Or the issue might not be related to power. I have been having sunny days for the whole week. The lowest power logged it’s about 60% of the lipo battery according to the spark fun monitoring system.

I haven’t been able to keep the system working for more than 5 days now.

Code attached. Let me know if you catch something that I should be concerned with my code:

#include "LEDControl.h"
#include "SparkFunMAX17043/SparkFunMAX17043.h"
#include "SparkFun_Photon_Weather_Shield_Library/SparkFun_Photon_Weather_Shield_Library.h"
#include "math.h"

#define PIN_SOLARNRG A2
#define PIN_POWERALERT_ISR D6
#define PIN_PROGRAMMODE_ISR D4

#define version "2.0a"


float humidity = 0;
float tempc = 0;
float tempf = 0;
float dewtf = 0;
float dewtc = 0;
float pascals = 0;
float baroTempc = 0;
float baroInches = 0;
float altimeter = 0;

Weather sensor;


//OPTIONS
const int pulseLEDEverySeconds = 30;
const int pulseLEDDuration = 1;
const int reportToCloudEverySeconds = 59;
const int deepSleepForHours = 1;

const int lowPowerAlert = 30;

//STATION SETUP
const int station_elevation_m = 1638;

char WU_SERVER [] = "weatherstation.wunderground.com";   //Standard server - for sends once per minute or less
char WU_WEBPAGE [] = "GET /weatherstation/updateweatherstation.php?";

//Station Identification
char WU_ID [] = "xxx"; //Your station ID here
char WU_PASSWORD [] = "yyyy"; //your Weather Underground password here

TCPClient client;




//VARIABLE DECLARATIONS
unsigned long int pulseSeconds=0;
unsigned long int seconds=0;

//Battery Shield
float voltage = 0.0; 
float soc = 0.0; 
float solarNrg = 0.0;
float solarNrgRaw = 0.0;
bool hybernateMode=false;
bool alert = false;
bool programMode = false;


//SYSTEM_MODE(SEMI_AUTOMATIC);	
Timer heartTimer(1000, heartBeat);

//BROWNOUT
void setBrowoutResetLevel() {
  const uint8_t desiredBOR = OB_BOR_LEVEL2; //2 = 2.40~2.70 // 1 = 2.10~2.40  // 3 = 2.70~3.60
  if(FLASH_OB_GetBOR() != desiredBOR) {
    /* Steps from http://www.st.com/web/en/resource/technical/document/programming_manual/CD00233952.pdf */
    /* See also https://github.com/spark/firmware/blob/aefb3342ed50314e502fc792f673af7a74f536f9/platform/MCU/STM32F2xx/STM32_StdPeriph_Driver/src/stm32f2xx_flash.c#L615 */
    /* To run any operation on this sector, the option lock bit (OPTLOCK) in the Flash option control register (FLASH_OPTCR) must be cleared. */
    FLASH_OB_Unlock();
    /* Modifying user option bytes */
    /* 1. Check that no Flash memory operation is ongoing by checking the BSY bit in the FLASH_SR register */
    FLASH_WaitForLastOperation();
    /* 2. Write the desired option value in the FLASH_OPTCR register */
    FLASH_OB_BORConfig(desiredBOR);
    /* 3. Set the option start bit (OPTSTRT) in the FLASH_OPTCR register */
    FLASH_OB_Launch();
    /* disable the FLASH option control register access (recommended to protect the Option Bytes against possible unwanted operations) */
    FLASH_OB_Lock();
  } 
  
}


void heartBeat() {
    if (seconds>=UINT_MAX) {seconds=0;}
    if (pulseSeconds>=UINT_MAX) {seconds=0;}
    
    seconds++;
    pulseSeconds++;
}


//DUTY METHODS
void publishBatteryEvents() 
{
   String msg = String::format("{ \"SOLNRG\":%f, \"VTS\":%f, \"SOC\":%f, \"LOWP\":%d, \"HYBM\":%d}", solarNrg, voltage, soc, alert, hybernateMode);
   Particle.publish("POWSTAT", msg); 
}

void publishWeatherEvents() 
{
   String msg = String::format("{ \"TEMP\":%f, \"DEW\":%f, \"PASC\":%f, \"ALT\":%f, \"HUM\":%f}", tempc, dewtc, pascals, altimeter, humidity);
   Particle.publish("WXSTAT", msg); 
}

void publishToWunderground()
{
    
    if (client.connect(WU_SERVER, 80)) {
        client.print(WU_WEBPAGE);
        client.print("ID=");
        client.print(WU_ID);
        client.print("&PASSWORD=");
        client.print(WU_PASSWORD);
        client.print("&dateutc=now");      //can use 'now' instead of time if sending in real time
        client.print("&tempf=");
        client.print(tempf);
        client.print("&dewptf=");
        client.print(dewtf);
        client.print("&humidity=");
        client.print(humidity);
        client.print("&baromin=");
        client.print(altimeter);
        client.print("&action=updateraw");    //Standard update rate - for sending once a minute or less
        client.println();
  } 
  
        delay(1000);
        client.stop();

}

// SHIELDS >>>>>>>>>>>>>>>>  >>>>>>>>>>>>>>>>  >>>>>>>>>>>>>>>>

//BATTERY-POWER
void readBattery()
{
    lipo.wake();
    delay(300);
	voltage = lipo.getVoltage();
	soc = lipo.getSOC();
	alert = lipo.getAlert(true);
	solarNrgRaw = ((analogRead(PIN_SOLARNRG) * 3.0)/4095);
    solarNrg = (100 / 3) * solarNrgRaw;
}

//WEATHER
void readWeather()
{
    
  humidity = sensor.getRH();
  
  tempf = 0.0;
  float rawTempc = sensor.getTemp();
  
  baroTempc = sensor.readBaroTemp();
  tempc = (rawTempc+baroTempc)/2;
  tempf = (tempc * 9)/5 + 32;
  
  pascals = sensor.readPressure();
  baroInches = pascals * 0.0002953;
  
  dewtc = dewPoint(tempc, humidity);
  dewtf = (dewtc * 9.0)/ 5.0 + 32.0;
  altimeter = altimeterSetting(pascals);
}

double dewPoint(double celsius, double humidity)
{
	// (1) Saturation Vapor Pressure = ESGG(T)
	double RATIO = 373.15 / (273.15 + celsius);
	double RHS = -7.90298 * (RATIO - 1);
	RHS += 5.02808 * log10(RATIO);
	RHS += -1.3816e-7 * (pow(10, (11.344 * (1 - 1/RATIO ))) - 1) ;
	RHS += 8.1328e-3 * (pow(10, (-3.49149 * (RATIO - 1))) - 1) ;
	RHS += log10(1013.246);

  // factor -3 is to adjust units - Vapor Pressure SVP * humidity
	double VP = pow(10, RHS - 3) * humidity;

  // (2) DEWPOINT = F(Vapor Pressure)
	double T = log(VP/0.61078);   // temp var
	return (241.88 * T) / (17.558 - T);
}

float altimeterSetting(float pascalsIn)
{
    float pressure = pascalsIn;
    pressure /= 100; //pressure is now in millibars

    float part1 = pressure - 0.3; //Part 1 of formula
  
    const float part2 = 8.42288 / 100000.0;
    float part3 = pow((pressure - 0.3), 0.190284);
    float part4 = (float)station_elevation_m / part3;
    float part5 = (1.0 + (part2 * part4));
    float part6 = pow(part5, (1.0/0.190284));
    float altimeter_setting_pressure_mb = part1 * part6; //Output is now in adjusted millibars
    float baroin = altimeter_setting_pressure_mb * 0.02953;

  return baroin;
}

void setSafeMode() {
    setLED(ORANGE);
    programMode = true;
}



// >>>>>>>>>>>>>>>>  >>>>>>>>>>>>>>>>  >>>>>>>>>>>>>>>>
// PHOTON METHODS
void setup() 
{
    
    //Protect for low voltage conditions.
    setBrowoutResetLevel();

    pinMode(PIN_SOLARNRG, INPUT);
    pinMode(PIN_POWERALERT_ISR, INPUT_PULLUP);
    pinMode(PIN_PROGRAMMODE_ISR, INPUT_PULLUP);
    pinMode(D7, INPUT_PULLDOWN);
    
    attachInterrupt(PIN_PROGRAMMODE_ISR, setSafeMode, FALLING);
    
    Time.zone(-5);
    
    Serial.begin(9600);
    
    //BatteryShield
    lipo.begin(); 
	lipo.quickStart();
	lipo.setThreshold(lowPowerAlert);
	
	//WeatherShield
	sensor.begin();
    sensor.setModeBarometer();
    sensor.setOversampleRate(100);
    sensor.enableEventFlags(); 
    RGB.control(true);
    
	String bootStatusString; 
	String ip = WiFi.localIP();
	Serial.println("*************************");
    Serial.println("        WXMAN");
    Serial.print("Version: ");
    Serial.println(version);
    Serial.println("By Lobo:Labs 2015");
    Serial.println("System Ready......");
    Serial.println("*************************");
    
    bootStatusString  = String::format("{\"VER\": \"%s\", \"RSSI\": %d, \"SSID\":\"%s\", \"IP\":\"%s\"}", version, WiFi.RSSI(), WiFi.SSID(), ip.c_str()); 
    Serial.println(bootStatusString);
    
    //waitUntil(Particle.connected);
	Particle.publish("BOOT", bootStatusString);
	
	readBattery();
	readWeather();
	delay(1000);
	WiFi.off();
    heartTimer.start();
}

void performReportDuties() {
    setLED(WHITE);
    
    if (!Particle.connected()) {
        Serial.println("[Cloud] Connecting to cloud");
        Particle.connect();
    }
        
    setLED(BLUE);
    waitUntil(Particle.connected);
    Serial.println("[Cloud] Transmitting Events...");
    publishBatteryEvents();
    delay(1500);
    publishWeatherEvents();
    delay(3000);
    setLED(GREEN);
    publishToWunderground();
    delay(1500);
    setLED(OFF);
    
    if (soc<99) { WiFi.off(); } //Only shut down when not at full charge.. If at full chargo don't waste time.
    
}

void enterProgramMode() {
        setLED(CYAN);
        if (Particle.connected() == false) {
             Particle.connect();
        }
        
       while (true) {
        setLED(MAGENTA);
         delay(1000);
         Particle.process();
         setLED(ORANGE);
         delay(1000);
        }
        
}

void timedEvents() {
    
    // if (pulseSeconds==pulseLEDEverySeconds) { setLED(GREEN); }
    // if (pulseSeconds>=pulseLEDEverySeconds+pulseLEDDuration) { setLED(OFF); pulseSeconds=0; }
    
    if (seconds % 2 == 0) { //Read Values every 2 seconds. 
        readBattery();
        readWeather();
    } 
    
    
    if (programMode==true) {
        programMode=false;
       enterProgramMode();
    }
    
    if (seconds % reportToCloudEverySeconds == 0) {
        hybernateMode = false;
        if (soc<=60 && solarNrg <= 58) { hybernateMode = true; }
        performReportDuties();   
        
        if (hybernateMode) {
            setLED(RED);
            lipo.sleep();
            
            delay(1500);
            if (soc>45) {  
                System.sleep(SLEEP_MODE_DEEP, 60*60*deepSleepForHours); 
            } else { //If battery very low... Sleep for longer!
                System.sleep(SLEEP_MODE_DEEP, 60*60*(deepSleepForHours*2)); 
            }
        }
        
    }
}

void loop() {
    timedEvents();
}

[mdma]

Can you say why you think the stall has anything to do with eeprom persistence?

(ScruffR: Since it hasn’t, I’ve moved this to a seperate thread)

[ScruffR]

Just a general suspicion with long running code and the use of local/automatic String variables.
Could you try to log the addresses of msg.c_str() in your publish functions to see if you had run out of heap space prior to your code freezing.

For my part I would say away from String wherever I can. (Sometimes libraries require the use of String, so I have to, but otherwise )

I’d also suspect that your timedEvents() is not really that timed at all.
Since you call it from loop several hundreds if not thousands of times, your seconds % 2 will hit numerous times in a row for the same second.
You might want to either add a if (seconds == prevSec) return; or get your timedEvents() only called once per sec (e.g. via soft timers).

[frlobo]

@mdma

Can you say why you think the stall has anything to do with eeprom persistence?

I don't... I said that I thought it had nothing to do with power, because the battery never ran bellow 60% power... I didn't mention anything about eeprom persistence.

[frlobo]

@ScruffR

For my part I would say away from String wherever I can. (Sometimes libraries require the use of String, so I have to, but otherwise )

Il move away from String's then. Thanks

@ScruffR

I'd also suspect that your timedEvents() is not really that timed at all.

I will modulate this, but I don't see why timedEvents() could be the cause of the crash. It ran smooth for 5 days continuously. Anyway. I will try to modulate or dosify the calls.

I really appreciate the help!

[ScruffR]

@frlobo

Can you say why you think the stall has anything to do with eeprom persistence?

I don't... I said that I thought it had nothing to do with power, because the battery never ran bellow 60% power... I didn't mention anything about eeprom persistence.

This thread is entitled "'EEPROM' persistence issue", so your post will be see in that context and so it was a fair question of Mat.

---

The thing about timedEvents() was a side-note that just caught my eye, but might be unrelated to your problem.
On the other hand, how will your sensors deal with being hammered this rapidly over extended periods of time?

[RWB]

When you say it stalls do you mean that the Photon powers up with the LED White and just stays in that state instead of moving on to the green flashing and then cyan breathing after wifi connection?

I’m asking because I’m doing some testing with a Photon + 1100mAh LiPo + Low Energy Harvesting Chip + Solar panel for harvesting power from indoor lighting conditions. Sometimes I see the Photon try to start up but it get’s stuck with the White LED on startup and stays that way until I reset it. At this point I’m not 100% sure what is causing this due to lack of time to test it properly but I’m wondering if this is also what your seeing with your solar setup.

I’m assuming it has to do with the voltage being fed to the Photon, maybe the battery is to low in my case.

[frlobo]

If you check the code, you will see that they get hammered every 2 seconds.. this is because the Seconds variable is updated using an interval timer ISR.

Maybe I am missing somethings, but this is the intention and how the tests look like.

As far as the title, you are correct. I use the thread, because is where the brownout solution was posted for me.

@RWB

When you say it stalls do you mean that the Photon powers up with the LED White and just stays in that state instead of moving on to the green flashing and then cyan breathing after wifi connection?

When the Photon goes out, it goes out.. No LEDs are on. I have to press reset and it gets it going. This might be related to the SparkFun Power Shield. Maybe there's an issue with the board that I don't know about?

Anyway.. I will keep testing... And see what I can find.

[ScruffR]

@frlobo

If you check the code, you will see that they get hammered every 2 seconds.. this is because the Seconds variable is updated using an interval timer ISR.

I might be missing something in your code too, but I'm looking at this

Timer heartTimer(1000, heartBeat);

void heartBeat() {
    if (seconds>=UINT_MAX) {seconds=0;}      // *** what is this for? uint wraps automatically
    if (pulseSeconds>=UINT_MAX) {seconds=0;} // *** thats one of the points of using uint
    
    seconds++;
    pulseSeconds++;
}

void timedEvents() {
    // *** since seconds does NOT change for ... (guess) ... one second
    // *** each iteration of loop() in that same second will satisfy the condition again and again
    if (seconds % 2 == 0) { //Read Values every 2 seconds. // *** no! multiple times in every other second
        readBattery();
        readWeather();
    } 
    ...
    // *** possible delays in this function which would resolve the issue are conditional
    // *** so don't help in all cases
}

void loop() { 
    timedEvents();  // this potentially happens thousands of times in one second
}

[frlobo]

You are absolutely right. Let me try fixing that with q flag inside the if… And try again and also remove the Strings to see if that fixes it.

[frlobo]

So this is the WUnderground station address… let’s hope that it keeps online with the fix…

http://www.wunderground.com/personal-weather-station/dashboard?ID=INUEVOLE47

This is the attempt to fix the code:

if ((seconds % 2 == 0) && (lastReadSeconds!=seconds)) {  //Read Values every 2 seconds. 
        lastReadSeconds = seconds;
        readBattery();
        readWeather();
}

[ScruffR]

Just out of interest, why are you using a soft timer to push your private seconds counter forward when there is an RTC that already does that for your?
And to read the value you’d just use Time.now() this is a uint32_t which counts the seconds since 1.1.1970 - or Time.second() would be just as good for what I can see in your code.

[frlobo]

@ScruffR

Time.second()

This is because of ignorance or lack of knowledge from myself. Your solution seems much nicer.

Let me try it.

So:

if (Time.second()!=lastSecond) 
{
 lastSecond = Time.second();
//DO EVERY SECOND EVENTS
}

??

[ScruffR]

This should work

[frlobo]

Just an update to report that it’s been 21 days without a crash. The system is up and running using solar cell, and its waking up as expected and reporting weather as expected. So so far so good! I guess the fix was that I was locking something in the loop.

[frlobo]

Another update:

The WX Station finally stopped responding again. After almost three months of uninterrupted work, it is actually very reliable for my standards.

However I would like to explore further the issue and try to find where the problem might be. It happened during a slept period that wasn’t a deep sleep because it happened at 11am… We have been having pure sunny days without issues in the last couple of weeks.

11AM Happens to be a perfect hour on sunny days. I am guessing battery was at 100% at that time.

When I reseted the device the battery was at 80% charge, so that means that the panel and the battery are still working. Wireless radio sleep happens anyway between reports to conserve battery between calls.

And DeepSleep happens when the battery is below 50%. Which is means it’s either very cloudy or night. In which case it will start reporting once per hour and then at 40% once every 3 hours… And at 35% Once every 5 hours to prevent brownouts due to low power conditions.

So… Any clues on where to start looking for answers? Any logging ideas or leads would be very welcomed!

I have some code improvements (posted later today here) that I was afraid to upload since it was very reliable and wanted to stress test it… Now Il use this chance to upgrade firmware and try new mechanisms of tracking time, such as what was suggested here to use the time keeper with Hours Seconds instead of micros

I am attaching a screenshot for fun of the graphics of the temperature reporting to UNDERGROUND.

[epierre]

Hello @frlobo,l

Finally I find interesting posts such as yours, I am doing the same experiments... have you a current version of your script ? I've seen on WU that you were no longer posting ?

Solar Powered Spark Core

Hello, The one with deep deep is still alive, the other stopped yesterday noon. Both have same battery and solar panel. Will try same code on both.

[frlobo]

I am no longer posting, because I am getting very erratic results and unpredictable. Once it work for months… And then stopped working in weeks and sometimes days. Don’t know…

[epierre]

@frlobo have you upgraded through DFU to latest firmware ?

[frlobo]

I did. But to be fair, I haven’t have the time to properly test it and document my findings. I also upgraded my firmware, so I need to debug it first very deeply and then try again. I removed some timers I was using that I was suspecting interfered with the deep sleep. I also added a forced reset of the MCU every week to clear memory and prevent integer overflows that I accidentally forgot to reset.

So… I will work with the new OS and new firmware and document my findings here, next week.

Thanks for all your help!

Ps: what changed in firmware that you think was affecting this behaviors?