Code Not Working after 15 Days in the Field


#1

I deployed a particle electron for 15 days so far in the field (System Firmware 0.6.1). The device starts up, takes a sensor measurement, publishes it to the cloud that is connected to a web-hook, and then goes into deep sleep mode for about 24 hours. (And by the way I’m using a 3rd party sim card).

It was working fine for 15 days, but then the last two days it stopped publishing and posting to my database. I took the device and pressed reset, and watching the LED, it connected to the cellular network, appeared to have taken a measurement, and then went into deep sleep mode as expected. However, I watched the logs in that product, and nothing ever published. I tried it multiple times with the exact same result.

I finally got it publishing and posting again by simply re-uploading the same code fresh onto the device. However, if I have 100 of these in the field I can’t go re-uploading code for all of these devices manually, and re-flashing code when it shakes hands with the cloud could be costly at scale.

What is going on here? Is it something wrong with the device? Is the code expiring somehow? Is it a Particle Cloud bug?

Thanks in advance for the help!


#2

I’ve been testing an Electron outside running on a 3w solar panel for many months now.

What I found is the same as you, eventually, something would happen that would cause the Electron to stop sending data even though it looked as if everything is fine which would require 1 or 2 resets to get it back to normal operation.

I’m using only Particle SIM’s. I’m sending data to Ubidots instead of the Particle Cloud because it databased the data and give me the ability to create a dashboard for the info and send text and email alerts.

I eventually added the WatchDog code to reset the Electron if it ever did freeze up.

With the latest code below I was able to keep it online the longest and so far no resets have been required.

I’ll post the code I’ve ended up with below for to look over if you desire.

SYSTEM_MODE(SEMI_AUTOMATIC);
//SYSTEM_THREAD(ENABLED);
// This #include statement was automatically added by the Particle IDE.
#include "Ubidots/Ubidots.h"

#define TOKEN "YourTokenGoesHere" // Put here your Ubidots TOKEN
#define DATA_SOURCE_NAME "ElectronSleepNew"

SerialLogHandler logHandler(LOG_LEVEL_ALL);  //This serial prints system process via USB incase you need to debug any problems you may be having with the system.

Ubidots ubidots(TOKEN); // A data source with particle name will be created in your Ubidots account.
//

int button = D0;         // Connect a Button to Pin D0 to Wake the Electron when in System Sleep mode. 
int ledPin = D7;         // LED connected to D1
int sleepInterval = 60;  // This is used below for sleep times and is equal to 60 seconds of time. 

ApplicationWatchdog wd(660000, System.reset); //This Watchdog code will reset the processor if the dog is not kicked every 11 mins which gives time for 2 modem reset's. 

void setup() {
 //Serial.begin(115200);
 pinMode(button, INPUT_PULLDOWN);  // Sets pin as input
 pinMode(ledPin, OUTPUT);          // Sets pin as output

 ubidots.setDatasourceName(DATA_SOURCE_NAME); //This name will automatically show up in Ubidots the first time you post data. 
 
 PMIC pmic; //Initalize the PMIC class so you can call the Power Management functions below. 
 pmic.setChargeCurrent(0,0,1,0,0,0); //Set charging current to 1024mA (512 + 512 offset)
 pmic.setInputVoltageLimit(4840);   //Set the lowest input voltage to 4.84 volts. This keeps my 5v solar panel from operating below 4.84 volts.  
}

void loop() {
    
FuelGauge fuel; // Initalize the Fuel Gauge so we can call the fuel gauge functions below. 
 
    
if(fuel.getSoC() > 20) // If the battery SOC is above 20% then we will turn on the modem and then send the sensor data. 
  {
   
   float value1 = fuel.getVCell();
   float value2 = fuel.getSoC();
   
  ubidots.add("Volts", value1);  // Change for your variable name
  ubidots.add("SOC", value2);    

  Cellular.connect();  // This command turns on the Cellular Modem and tells it to connect to the cellular network. 
  
   if (!waitFor(Cellular.ready, 600000)) { //If the cellular modem does not successfuly connect to the cellular network in 10 mins then go back to sleep via the sleep command below. After 5 mins of not successfuly connecting the modem will reset.  
    
    System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); //Put the Electron into Sleep Mode for 2 Mins + leave the Modem in Sleep Standby mode so when you wake up the modem is ready to send data vs a full reconnection process.  
    
}  
  
     ubidots.sendAll(); // Send fuel gauge data to your Ubidots account. 

     digitalWrite(ledPin, HIGH);   // Sets the LED on
     delay(250);                   // waits for a second
     digitalWrite(ledPin, LOW);    // Sets the LED off
     delay(250);                   // waits for a second
     digitalWrite(ledPin, HIGH);   // Sets the LED on
     delay(250);                   // waits for a second
     digitalWrite(ledPin, LOW);    // Sets the LED off
  
     System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); //Put the Electron into Sleep Mode for 2 Mins + leave the Modem in Sleep Standby mode so when you wake up the modem is ready to send data vs a full reconnection process.  
    
  }
  else //If the battery SOC is below 20% then we will flash the LED 4 times so we know. Then put the device into deep sleep for 1 hour and check SOC again. 
  {
      
  //The 6 lines of code below are needed to turn off the Modem before sleeping if your using SYSTEM_THREAD(ENABLED); with the current 0.6.0 firmware. It's a AT Command problem currently. 
  //Cellular.on();
  //delay(10000);
  //Cellular.command("AT+CPWROFF\r\n");
  //delay(2000);
  //FuelGauge().sleep();
  //delay(2000);
  
  
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  
  System.sleep(SLEEP_MODE_DEEP, 3600);  //Put the Electron into Deep Sleep for 1 Hour. 
  
  }
}    

#3

Well, the code doesn’t expire, but the fact that reflashing your code fixes it, while a reset does not, might indicate that your app firmware is being corrupted. This can happen a number of ways in C. Anything that causes you to write to an address that you shouldn’t, like a buffer overflow, could corrupt your code. Seeing your code would help us to see if that may be the problem.


#4

Also, batteries dying has been known to cause some Electrons to lose all their memory. This is the reason I put the Electron to Deep Sleep before the battery has a chance to go flat.


#5
    // This #include statement was automatically added by the Particle IDE.
    #include <HC_SR04.h>
    #include "Particle.h"
    #include "cellular_hal.h"
    PRODUCT_ID(4139); 
    PRODUCT_VERSION(1);
    STARTUP(cellular_credentials_set("hologram", "", "", NULL));

    // initializing variables
    double cm = 0.0;
    double inches = 0.0;
    double initial_distance = 0.0;

    int trigPin = D4;
    int echoPin = D5;
    int num_meas = 10; 
    double temp_val; 

    //This calls and initializes the sensor 
    HC_SR04 rangefinder = HC_SR04(trigPin, echoPin, 2.0, 250.0);

    void setup() {
        // enables the solid state relay to turn on so the Sensor can get power.
        pinMode(D3,OUTPUT);
        digitalWrite(D3, HIGH); 
        delay(100);
       
        // initialize the measurement value
    double meas_val = 0.0;
        double percentage_val= 0.0;
        
        // for loop to take measurements
        for(int i = 0; i < num_meas; i++) {
            meas_val += rangefinder.getDistanceCM();
            delay(100);
        }
        cm = meas_val / num_meas;
        
        // turns relay off so no power is leaking to the sensor.
        digitalWrite(D3, LOW);

        // converting stuff to strings.
         int y = (int)cm;
         String rdg = String(y);
         String status = "I";

        char buf[256];
        snprintf(buf, sizeof(buf), "{\"m\":\"%s\",\"est\":\"%s\"}", rdg.c_str(), status.c_str());
        Particle.publish("M",buf,PRIVATE);
       
        delay(5000);
      
        // goes into deep sleep mode for 30 seconds and then resets the device.
        System.sleep(SLEEP_MODE_DEEP, 86300);
    }

    void loop() {
      
    }

#6

To format your code in the post add this at the top above your first line:

 ```cpp

 and this on the line below your last line: 


Sometimes it takes a few tries to get it to work.

#8

@RWB Thanks! I checked the batteries and they are still going very strong. Is the ubidots solving the problem for your issue, or is it the watchdog? I don’t understand that fully. I’m having my device go to deep sleep for nearly 24 hours.

@Ric I uploaded my code, but I’m not sure where I would be writing to that is causing the corruption. I’d love to hear your insights.


#9

I’m not really sure when the Watchdog kicks in so it’s hard telling but I do feel better that it’s there to reset it should the code ever lock up.

The thing with the publish-loop running normally but not actually showing up on the receiver end would not cause the watchdog to kick in and reset the device because it’s not actually causing the code to block.

I would suggest just trying your code with the additions and seeing if your device locks up.

I’m using the Particle SIM so I have no idea if the Hologram service could be the cause of your issue or not. I do know that using the 3rd part SIM does require you to send the Keep-Alive PING more often if you’re using the Particle SIM. If you’re sending data to Ubidots then you do not need to send the Keep-Alive Ping since that is just for the Particle Cloud and Ubidots does not rely on the Particle Cloud.

Ubidots seemed to use less data and have much quicker connect times when compared to Particle. I just use Ubidots because for me it was really easy to setup and gave me the ability to create custom dashboards and SMS / Email message alerts based on my pre-defined thresholds.

It’s worth trying out if your looking for a way to database your data and create a dashboard.


#10

@RWB yes. Thank you! I’m glad you pointed me to ubidots. I signed up to try it out.

I’ll try out your suggestions from your code.


#11

Do you see anything in my code that would corrupt the firmware?


#12

I’m not sure that this would cause any problems, but sometimes using Strings can cause heap fragmentation. I don’t know if that would cause the symptoms you see. Anyway, there’s no need to create Strings, then convert them to c_strings in your snprint function. You could delete the two lines above where you create rdg and status, and do this,

 char buf[256];
 snprintf(buf, sizeof(buf), "{\"m\":\"%d\",\"est\":\"%s\"}", y, "I");

I don’t see anywhere else in your code that would cause your publishes to stop working.


#13

Since deep sleep wakes with a reset heap fragmentation should not be an issue.
Also mere RAM write operations won’t be able to corrupt the flash based firmware either.

But I’d make sure to have a valid cloud connection before attempting a Particle.publish() by use of waitFor(Particle.connected, 300000).
Although AUTOMATIC mode should take care of that, adding this safe guard and maybe forcing a cellular module reset in case of failing waitFor() might help.

BTW, with recent system versions you won’t need delay(5000) after the publis on Electrons (on Photons you still do).


#14

Hello, you can use the webhook of particle to send data to Ubidots too :), please check this article:

http://help.ubidots.com/connect-your-devices/how-to-use-particle-webhooks-with-ubidots


#15

Could you explain how Ubidots uses less data, when it connects into particle anyways first? Are you not using Webhooks and connecting directly to ubidots somehow? I don’t know how to do that.

Thanks!


#16

Using the Ubidots library sends your data Unencrypted directly to Ubidots without using the Particle cloud and this uses less data.

I have some example code showing how I do it if your interested.


#17

Oh okay. I think I see that in your code above, or are there other important things I should know about that?

Thanks!


#18

Yea that’s the code above.


#19

@liddlem - I am having this exact symptom. My code is also very similar. I am also using hologram SIMs in my electrons.
I have a support issue opened with Particle as I believed it to be a cloud issue but after reflashing my binary to the device, it took off running again. This has happened twice on one of my test devices but happens so infrequently that it is impossible to troubleshoot.
The idea that something in the code is causing some sort of firmware corruption is troubling but it’s the best I have to go on at the moment. Sadly, it could be pretty much anything and there is no way to test it.
If you happen to figure anything out, I would love to hear from you.
@ScruffR - I already have a waitFor(Particle.connected but that hasn’t seemed to resolved the issue.


#20

How long are you waiting for?
Cellular connection can take up to 5 minutes (300000ms)


#21

Hi @ScruffR! First off, let me say that you are always an incredible help. I don’t think I would have been able to make it this far without the help you provide myself and others.

Regarding this issue, let me recount the issue as I believe it lines up with the issue that @liddlem is having.

Running a particle electron with system firmware 0.6.1. Using a hologram sim card. SEMI_AUTOMATIC.
This device, like many others, checks a couple of pressure sensors every so often and then goes to sleep using SLEEP_MODE_DEEP. (The sleep time is tied to the battery level so it is variable). It reports in with data via a particle.publish every so often.

Twice now, the device has stopped reporting in (months apart). When I checked the last reporting data each time, the device had good signal strength and plenty of battery but simply wasn’t reporting in.

I grabbed the device from it’s location and pressed the physical reset button. The device dutifully ran through it’s process, but no data was sent.

I ran it again while plugged in via USB as I have very thorough debugging set up on these devices. Everything was working as it should. The device turned on, took it’s measurements, it turned on the cell modem (blinking green), started it’s cloud connection (flashing cyan and then ultimately breathing cyan), and sent it’s data via particle.publish (according to the debugging). However, as I watched the stream via SSE, no data was being sent.

I spoke with Rick at particle and he checked the logs. The logs had simply stopped at the last successful connection. He said there was a device broker issue at that time, but it reset itself and everything should be working perfectly. But still, the particle.publish just simply wasn’t coming across the stream.