Hard Fault on Electron (after about an hour)

Troubleshooting
1

Hello,

for the last couple days I was working on resolving an issue of which I couldn’t find the cause yet.

My Electron is running a code which reads temperature and humidity data from an SHT31 sensor (I2C connected) which is accessible via an Android App. Moreover, I am using a CC1101 (SPI connected) radio module which receives the same data from an outside located Arduino Nano. This data is then being added to a string variable after a defined period of time which is used for monitoring the temperature of the last 24h.

So the issue is that after about an hour (more or less) the Electron is having a hard fault which then leads to the device reseting itself and to loosing the data of the variables including the monitoring of the temperature. After this hard fault the Electron works fine again, apart from the data being lost, until the same issue occurs in the next one ore two hours.

I would very much appreciate your advice about what could eventually cause this problem and how I could avoid it.

Thanks,
David

2

How many red blinks are you seeing?

3

@kennethlimcp

The thing is that this issue occurs quite randomly after about an hours as I said. So as I didn’t want to watch the Electron for an hour I integrated a publish function which (using IFTTT) notifies me whenever the setup code has ran through.

After receiving this notification I had a look at the console which shows me “panic, hard_fault”.

I am sorry I can’t tell you more on this point but if its necessary I will have to watch it for quite a while.

1 Like
4

Will you be able to share the code that you are running?

1 Like
5

Sure I can… but I have to warn you that it’s not very cleaned up yet

I added a screenshot to give you an idea of what the Android App looks like after the code was running for two hours.

Screenshot_20170518-164408.png1440×2560 113 KB 

    // This #include statement was automatically added by the Particle IDE.
    #include "CC1101Radio.h"

    // This #include statement was automatically added by the Particle IDE.
    //#include "RCSwitch/RCSwitch.h"
    #include "adafruit-sht31/adafruit-sht31.h"
    #include "math.h"


    // -------------------------------------------- CC1101 ----------------------------------------------
    #ifdef ARDUINO
    #include "SPI.h"
    #endif

    // comment next line if packets should not be send automatically
    #define _sendPackets

    // comment next line if packets should not be read automatically
    #define _readPackets

    // comment next line if Photon should not publish automatically
    #define _PhotonStuff


    #include "CC1101Radio.h"
    CC1101Radio cc1101;


    // ----------------------
    // Stats to publish
    // ----------------------

    uint16_t packetsSend = 0;
    uint16_t packetsReceived = 0;
    uint32_t packetsSendBytes = 0;
    uint32_t packetsReceivedBytes = 0;

    // ----------------------
    // Counters and timers
    // ----------------------

    uint16_t packetNum = 0;
    uint32_t lastSendMs = 0;
    #define msBetweenSending 2000

    uint32_t lastPublishMs = 0;
    #define msBetweenPublish 60000 //5000 prior value

    uint8_t publishDevices = 0; // publish a list of devices or a stat of send/receive packets

    // ----------------------
    // Interrupt procedures
    // ----------------------

    void cc1101signalsInterrupt(void){
    	cc1101.packetAvailable=true;
    }

    void attachIntr() {
    	#if defined(PARTICLE)
    	attachInterrupt(cc1101.GDO0pin, cc1101signalsInterrupt, FALLING);
    	#else
    	attachInterrupt(digitalPinToInterrupt(cc1101.GDO0pin), cc1101signalsInterrupt, FALLING);
    	#endif
    }

    void detachIntr() {
    	#if defined(PARTICLE)
    	detachInterrupt(cc1101.GDO0pin);
    	#else
    	detachInterrupt(digitalPinToInterrupt(cc1101.GDO0pin));
    	#endif
    }



    /*
    RCSwitch mySwitch = RCSwitch();
    int inputPin = D3;
    */

    unsigned long ULBefore=2200;
    unsigned long ULNow=2200;

    Adafruit_SHT31 sht31IN = Adafruit_SHT31();
    //Adafruit_SHT31 sht31OUT = Adafruit_SHT31();

        int iRelayTest=D5;
        
        int iMoveDetect=D6;

        String sTEMPIN = "";
        String sTEMPOUT = "";
        
        String sHUMIDIN = "";
        String sHUMIDOUT = "";
        
        String sTEMPIN_P = "";
        String sTEMPOUT_P = "";
        
        String sHUMIDIN_P = "";
        String sHUMIDOUT_P = "";
        
        //Change "aaaaa" instead of ""
        String sTEMPINARRAY="aaaaa";
        String sTEMPOUTARRAY="aaaaa";
        String sTEMPINARRAY_V="";
        String sTEMPOUTARRAY_V="";
     
        int iCountArray=0;
        String sCountArray="0";
        
        const int iArray=123;
        
        double dTEMPIN = 22;
        double dTEMPOUT = 22;
        
        double dHUMIDIN = 50;
        double dHUMIDOUT =50;
        
        int iCountM=0;
        int iCountP=0;
        
        int iCountBeweg=0;
        
        int iArraySubString=0;

        String sRelayStatus="LOW";
        
        int iTimeUnix=0;
        int iTimeDif=0;
        int iMovePrev=LOW;
        int iMove=LOW;
        
        int iCountUNIX=0;
        
        int iTimeUnixStart=0;
        int iTimeStartDif=0;
        
        double dCompareTime1=0;
        double dCompareTime2=0;
        
        int iCountTime1=1; //changed to 1
        int iCountTime2=0;
        
        double dCountInterval1=702.439;
        double dCountInterval2=60;
        
        int iTimerLongRelay=0;
        
        int iCountWrong=0;
        
        double dTEMPOUTREC=2500;


    void setup() {
        
        Particle.syncTime();
        
        Particle.keepAlive(30);

        Particle.variable("TEMPIN", sTEMPIN_P);
        Particle.variable("TEMPOUT", sTEMPOUT_P);
        Particle.variable("HUMIDIN", sHUMIDIN_P);
        Particle.variable("HUMIDOUT", sHUMIDOUT_P);
        Particle.variable("RELAY", sRelayStatus);
        
        Particle.variable("TEMPINA", sTEMPINARRAY);
        Particle.variable("TEMPOUTA", sTEMPOUTARRAY);
        Particle.variable("COUNTARRAY", sCountArray);

        Particle.function("RelayT", RelayT);
        Particle.function("TimeMove", TimeMove);
        
        sht31IN.begin(0x44);
        //sht31OUT.begin(0x45);
        
        pinMode(iRelayTest, OUTPUT);
        digitalWrite(iRelayTest, HIGH);

        pinMode(iMoveDetect, INPUT);

        iArraySubString=(iArray-1)*5;

delay(30000);

    iTimeUnixStart = Time.now();

    while(iTimeUnixStart<1482945639)
    {
        Particle.syncTime();
        
        delay(10000);
        
        iTimeUnixStart=Time.now();
        
        iCountUNIX++;
        Particle.publish("UNIX", "FALSE UNIX Time! New UNIX Time: " + String(iTimeUnixStart) + " / Count: " + String(iCountUNIX));
    }
   
   
   
   
   //------------------------------- CC1101 -------------------------
   
  Serial.begin(57600);

	cc1101.init(); // initialize to default setup values

	// change what is needed
	
	// Photon, Arduino Uno, Mini, Mega & Due is given a uniq device Address in CC1101Radio.h
	// If more than one of each board is used, they will get the samme address. I that case change device Address below
	// cc1101.deviceData.deviceAddress = 99;
	#ifdef ARDUINO
		pinMode(9,INPUT);
		if(digitalRead(9)==LOW)
		cc1101.deviceData.deviceAddress++;
		
		Serial.println("\r\nAdded 1 to deviceAddress becourse D9 is connected to ground");
	#endif
	
	
	
	
	// remoteDeviceAddress is set to '0'. This means broadcast to all cc1101 devices
	cc1101.deviceData.remoteDeviceAddress = 0;
	
	// We don't let the CC1101 filter packets for this deviceAddress, we receive everything
	cc1101.deviceData.addressCheck = false;
	
	// Use the same frequency, network and channel for alle CC1101 modules
	cc1101.deviceData.carrierFreq = CFREQ_433;
	cc1101.deviceData.txPower = PA_ShortDistance;
	cc1101.deviceData.channel = 5 ;
	cc1101.deviceData.syncWord[0] = 19;
	cc1101.deviceData.syncWord[1] = 9;

	// write deviceData to cc1101 module
	cc1101.begin();
	
	// start reading the GDO0 pin through the ISR function
	attachIntr();


    sendPacket();

    Particle.publish("SETUP", "SETUP function triggered");
    
    
  }

void loop() {

 iMove=digitalRead(iMoveDetect);
   
   if(iMovePrev==LOW && iMove==HIGH)
   {
       iTimeUnix=Time.now();
       
       iCountBeweg++;
       
       Particle.publish("BEWEG", "Bewegung erkannt! Nummer: " + String(iCountBeweg));
       
   }
   
   iMovePrev=iMove;

   
   //----------------------- new CODE ---------------------------------
   //Interval from Start
   iTimeStartDif = Time.now()-iTimeUnixStart;
   
   dCompareTime1 = dCountInterval1*iCountTime1;
   dCompareTime2 = dCountInterval2*iCountTime2;
   
   
   
   
   if(iTimeStartDif>=dCompareTime2)
   {
       
       
       dTEMPIN = sht31IN.readTemperature(); 
       
       while(-0.01<dTEMPIN<0.01 || dTEMPIN > 150 || dTEMPIN<-50) //new
       {
           delay(2000);
           
           Particle.publish("S_FAIL_TEMP", "Wert des Sensors: " + String(dTEMPIN));
           
           dTEMPIN = sht31IN.readTemperature();
       }
       
       //dTEMPOUT = sht31OUT.readTemperature();
       delay(200);
       dHUMIDIN = sht31IN.readHumidity();
       
       
       while(dHUMIDIN>100 || dHUMIDIN<=0)
       {
           delay(500);
           
           Particle.publish("S_FAIL_HUMID", "Wert des Sensors: " + String(dHUMIDIN));
           
           dHUMIDIN = sht31IN.readHumidity();
       }
       
       
       //dHUMIDOUT = sht31OUT.readHumidity();
       
       sTEMPIN = String(dTEMPIN).substring(0,5);
       sHUMIDIN = String(dHUMIDIN).substring(0,5);
       sTEMPOUT = String(dTEMPOUT).substring(0,5);
       sHUMIDOUT = String(dHUMIDOUT).substring(0,5);
  
        sTEMPIN_P = String(dTEMPIN).substring(0,4);
       sHUMIDIN_P = String(dHUMIDIN).substring(0,2);
       sTEMPOUT_P = String(dTEMPOUT).substring(0,4);
       sHUMIDOUT_P = String(dHUMIDOUT).substring(0,2);   
       
   
   
   
       
       iCountTime2++;
       
       //Particle.publish("TEMP", "TEMP Data measured! Count: " + String(iCountTime2));
   }
   
   
   if(iTimeStartDif>=dCompareTime1)
   {
       if(iCountArray<iArray)
       {
       sTEMPINARRAY_V=sTEMPIN+sTEMPINARRAY_V;
       sTEMPOUTARRAY_V=sTEMPOUT+sTEMPOUTARRAY_V;
       
       sTEMPOUTARRAY="aaaaa"+sTEMPOUTARRAY_V;
       sTEMPINARRAY="aaaaa"+sTEMPINARRAY_V;
       }
       
       if(iCountArray>=iArray)
       {
       sTEMPINARRAY_V=sTEMPIN+sTEMPINARRAY_V.substring(0,iArraySubString);
       sTEMPOUTARRAY_V=sTEMPOUT+sTEMPOUTARRAY_V.substring(0,iArraySubString);
       
       sTEMPOUTARRAY="aaaaa"+sTEMPOUTARRAY_V;
       sTEMPINARRAY="aaaaa"+sTEMPINARRAY_V;
       
       iCountArray=iArray;
       }
       
        sCountArray=String(iCountArray);
        iCountArray++;
       
       
       
       
       iCountTime1++;
       
       //Particle.publish("ARRAY", "Array Data added! Count: " + String(iCountTime1));
   }

if(sRelayStatus=="HIGH")
    {
        if(Time.now()>=iTimerLongRelay)
        {
         digitalWrite(iRelayTest, HIGH);
         sRelayStatus="LOW";
        }
    }
    
    
    
    // ---------------------------------------------------- CC1101 -----------------------------------------------------
   
   
   	if(!receivePacket()) {

		// If not packet available, check if we shoud send one
		if(millis() > lastSendMs + msBetweenSending) {
			//sendPacket();
			lastSendMs = millis();
		}
	}
	
   
   
    #ifdef PARTICLE
    #ifdef _PhotonStuff
	if(millis() > lastPublishMs + msBetweenPublish) {
	    publishStats();
	    lastPublishMs = millis();
	}
    #endif
    #endif
    
    delay(100); //changed to 10 instead of 100
}

struct weatherData_s {
	uint32_t temperature; // 4 bytes
	uint16_t humidity; // 2 bytes
};

void sendPacket() {

    weatherData_s wd;
    
    // replace with your sensor readings
    wd.temperature = dTEMPIN; //random(0,40000); 
    wd.humidity = dHUMIDIN; //random(10,100);
    
	CC1101Radio::CCPACKET pkt;
	
	pkt.length = 4 + sizeof(wd);
	
	pkt.data[0] = cc1101.deviceData.remoteDeviceAddress;
	pkt.data[1] = cc1101.deviceData.deviceAddress;
	pkt.data[2] = lowByte(packetNum);
	pkt.data[3] = highByte(packetNum++);
	
	// copy weather data to packet variable
	memcpy(pkt.data+4, &wd, sizeof(wd));
	
	
	
	// do the actual transmitting
	cc1101.sendData(pkt);

	packetsSend++;
	packetsSendBytes+=pkt.length;

	Serial.print("\r\nSend packet #");
	Serial.print(packetNum);
	Serial.print(" -");
	cc1101.printCCPACKETdata(&pkt);
	
}

bool receivePacket() {

	if(!cc1101.packetAvailable) {
		return false;
	}

	// The cc1101 has received a package
	detachIntr();
	cc1101.packetAvailable = false;

	CC1101Radio::CCPACKET pkt;

	// read the packet
	if(cc1101.receiveData(&pkt)) {
		
		// do we have valid packet ?
		if((pkt.crc_ok==1) && (pkt.length > 0)) {
			
			packetsReceived++;
			packetsReceivedBytes+=pkt.length;

			Serial.println("\r\n--------------------------------");
			Serial.print("CC1101 have news! - ");
			Serial.print("Package len = ");
			Serial.print(pkt.length);
			Serial.print(" lgi=");
			Serial.print(pkt.lqi);
			Serial.print(" rssi=");
			Serial.print(pkt.rssi);
			Serial.print(" CRC=");
			Serial.println(pkt.crc_ok);
			
            weatherData_s wd;
		    memcpy(&wd, pkt.data+4, sizeof(wd));	
		
		    uint16_t msgNum = pkt.data[2] + (pkt.data[3]*255);
		
		    Serial.print("Message num (");
		    Serial.print(msgNum);

		    Serial.print(") from device #");
            Serial.print(pkt.data[1]);
            Serial.print(", sendt to device #");
            Serial.print(pkt.data[0]);
            Serial.print(", data length: ");
            Serial.println(pkt.length-4);
            
            Serial.print("\tTemperature = ");
            Serial.println(wd.temperature/100);
            Serial.print("\tHumidity = ");
            Serial.println(wd.humidity);
           
        
            Serial.println();
            
            dHUMIDOUT=wd.humidity;
            dTEMPOUTREC=wd.temperature;
            dTEMPOUT=dTEMPOUTREC/100;
            
		} // crc & len>0
		
	} // cc1101.readData
	
	attachIntr(); // re-attach interrupt so packetAvailable will be set true in ISR

	return true;

}

// ----------------------
//  Photon publish
// ----------------------

#ifdef PARTICLE


void publishStats() {
    
    sendPacket();
    
    char txt[64];
    
    if(((publishDevices % 4) > 0) && !cc1101.newDeviceAdded) {
        sprintf(txt,"Device #%d Sended %lu & received %lu packets (%lu/%lu bytes)",cc1101.deviceData.deviceAddress,packetsSend,packetsReceived,packetsSendBytes,packetsReceivedBytes);
    }
    else {
        sprintf(txt,"I am cc1101 address #%d, the others are ",cc1101.deviceData.deviceAddress);
        for(int idx = 0; idx < maxDevices; idx ++) {
            if(cc1101.deviceList[idx] != -1) sprintf(txt,"%s #%d",txt,cc1101.deviceList[idx]);
        }
    }
    
    cc1101.newDeviceAdded = false;
    Particle.publish("CC1101 Stats",txt);
    publishDevices ++;
    

 }
 
 #endif

 
int RelayT(String extra) {

if(extra.substring(0,4)=="HIGH")
{
    digitalWrite(iRelayTest, LOW);
    
    if(extra.substring(4).toInt()>0)
    {
        iTimerLongRelay = extra.substring(4).toInt()*60+Time.now();
    }
    else
    {
        iTimerLongRelay = 300+Time.now();
    }
    
    sRelayStatus="HIGH";
    return 1;
}
else
{
    digitalWrite(iRelayTest, HIGH);
    sRelayStatus="LOW";
    return 0;
}
 return -1;
}

int TimeMove(String extra2)
{

    if(extra2=="1")
    {
        
        if(iTimeUnix>0)
        {
        iTimeDif=Time.now()-iTimeUnix;           
        return iTimeDif;
        }        
        else
        {
        return -1;
        }
       
    }
}
6

Unexpected intermittent hard faults may indicate heap fragmentation caused by “heavy” use of String objects.
The common cure (or action to exclude it from the list of suspects) is to replace String with oldfashioned C strings (char[]) in conjunction with strcpy(), snprintf() and the likes.

(Grüße aus Salzburg nach Wien)

1 Like
7

Ok thanks

..that could indeed be the cause as I am using two strings for storing the temperatures over the last 24h. The only thing I was wondering about is that the same code, regarding the string use for the monitoring, was working fine for several hours as I tested it a while ago but now the problem occurs after only one to two hours.

Is it possible that it doesn't fail at a specific length of the string but randomly?

Another question: What exactly would you replace the following code with? ..I am not that experienced in using char with strcpy() snprintf() to be honest

(Grüße nach Salzburg )
:wink:

8

That exactly is the annoying thing about String or rather heap fragmentation that it happens sort of randomly.
It depends how often String objects get created, destroyed, resized, reallocated and since that is all happening behind the scenes you never really know when and how often it happens.

It's not only your two variables but actually everywhere you use String as datatype for variables or even just for conversions like String(someNum). Also several String methods like substring() cause dynamic mem allocation and consequently probable fragmentation.

About your syntax question, this would be a possible way for what you want (although I'm not fully sure about the meaning of iArraySubString and the current content of the variables used - without looking at your whole code)

  char szTEMPINARRAY_V[64];
  char szTEMPIN[16];
  
  snprintf(szTEMPIN, sizeof(szTEMPIN), "%.2f", dTEMPIN);  // floating point number with two decimal places
  snprintf(szTEMPINARRAY, sizeof(szTEMPINARRAY), "%s,%s", szTEMPIN, szTEMPINARRAY);

But instead of building the string up step-by-step I'd rather store the numeric values in a ring buffer and just build the full string on demand.

I'd also expose the numeric values as Particle.variable() direct instead of doing it via an intermediate string.

1 Like
9

So by talking about a ring buffer..do you mean to create a double array in which I can store the temperature values and then use a function thats called by the cloud which builds the string then and returns it?

...i will try this if replacing the string data monitoring code doesn't help.

Another idea I thought about is that the CC1101 radio module (SPI connected) might cause the problem as the code worked well for several hours by the time I didn't include this module into my project. Do you think that this could be a potential issue as well?

Thanks

10

Something like it.

const int count = 10;
double dRingBuffer[count];
int ringIndex = 0;

void addValue(double newVal) {
  ringIndex++;
  if (ringIndex >= count) ringIndex = 0;
  dRingBuffer[ringIndex] = newVal;
}

It's possible that the SPI device causes the issue, but String should be avoided in any case :wink:

1 Like
11

Ok sure :wink: ..I will try the code when disconnecting the CC1101

1 Like
12

I managed to record the Electron while its having a hard fault and it blinked the following way:

SOS pattern (3 short blinks, 3 long blinks, 3 short blinks)
1 blink (indicating hard fault)
SOS pattern (3 short blinks, 3 long blinks, 3 short blinks)
1 blink (indicating hard fault)

And then it goes back to searching for a connection and after that breathing cyan..

13

I did some testing the last couple hours (20h) and the result is that without having the CC1101 module connected, the Electron is not having this issue any more.

I am now thinking about what exactly could be the problem. Do you think that it might be caused by receiving false data (not numeric or even from other devices) or by the SPI connection between the module and the Electron itself?