Electron sometimes freezes on reconnect - Led Solid Cyan

Your issue sounds similar to this

3 Likes

After much debugging (running different code variations on ~8 Electrons continuously/without sleep overnight) and discussing with more experienced hardware developers/electrical engineers, I can post an initial update on the issue (not the least, for future developers experiencing the same issue):

These freezes, with solid LED, is not specific states in the firmware/hardware (as other community threads also conclude). Instead, this seems to typically be caused by the batteries not having the capability to handle the power surges that may occur during normal operation – the most demanding of which occur when connecting the Electron to Cellular/Cloud :zap:️. This seems to be very much in line with what we’ve experienced.

As it turns out, we run our Electrons mostly with batteries that are slightly underpowered. However, the batteries have “burst” abilities that are sufficient for even the highest specified power surge that can occur; as per Particle’s specification. We’ve not yet been able to completely iron out these freezes with code modifications (e.g. restricting successive disconnects, and more). Although, as of last night we’ve also experienced this freeze with the included Battery from Particle that definitely should be able to handle the load :sweat: Our rationale thus is that it might be code related still.

Our search continues, and we’ll try to find a code setup that let’s our Electrons run smoothly. Stay tuned.

Whether the battery will be able to cope or not also depends on your extra components powered off this same battery.

And 2G is known to have higher current demands than 3G

1 Like

Good points worth pointing out @ScruffR :slight_smile: we’ve considered this as well.

Of the components listed in my (very long) initial post; the buttons are passive components, and the screen is only 25 mA. So this should not cause issues since the battery can handle 1,5 A continuous discharge with burst/pulse discharge at 3 A (which our battery provider claims can keep up for ~10 minutes at a time). Particle specifies that their hardware can at most require 1,8 A.

As for the 2G v. 3G current demands; this has also come to light. We actually ordered 3G electrons at the start of this week, and will hopefully get them early next week.

Okay so this is weird.

We couldn’t pinpoint where in our code history the freezes started to occur (we managed to freeze one unit with code that we thought was completely safe). We reduced our code to a very barebones setup to see if the units would even manage that. Out of eight units, one had its LED stuck in solid green :scream: Four among them powered down because of low battery, so there might actually have been more stuck units (will look into the logs and update this post accordingly) – but 1/8 is still way too bad, especially for such a short cycle :dizzy_face:

So, we had 8 units trying out the following code overnight; the only demanding thing it does is basically to connect to the cloud, and after 2 minutes disconnect and connect again. Apart from this, it also keeps track of manual/dropped disconnects and handshakes, and prints them to the connected screen:

#include "Adafruit_SSD1306.h"
#include "Adafruit_GFX.h"
#include "fonts.h"

using namespace std;

#include "application.h"


#define LOGGING_ON TRUE

#if (LOGGING_ON)
   SerialLogHandler logHandler(LOG_LEVEL_ALL);
#endif


/* ============== MAIN =====================*/

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

// New screen
#define OLED_DC     D0
#define OLED_CS     A4
#define OLED_RESET  D1
Adafruit_SSD1306 oled(OLED_DC, OLED_RESET, OLED_CS);

FuelGauge fuel;
PMIC pmic;

char version[] = "0.1";
char ID[] = "8015";

unsigned long lastDisconnectTimestamp;
unsigned long unix_timestamp;

volatile int handshakes = 0;

unsigned long checkBattery = 0;
unsigned long lastUSBCheck = 0;
unsigned long lastPublish = 0;
unsigned long lastUpdatedTS = 0;

int statusUSB = 0;
int lastUSB = 0;

bool updateScreen = false;
bool isManualDisconnect = true;
volatile bool connectionStatus = false;
bool lastConnectionStatus = false;
bool isConnecting = false;
int batteryStatus = 0;

const String clickerIDString = String(System.deviceID());

String deviceID;
String firmwareVersion;

int panicCount = 0;
int manualDisconnectCount = 0;
int droppedDisconnectCount = 0;

static const unsigned char batteri_4of4 [] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0xBF,
0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_3of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_2of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_1of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_0of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char charging[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x82, 0x40, 0x00, 0x86,
0x40, 0x00, 0x8E, 0x40, 0x00, 0x8C, 0x40, 0x00, 0x9C, 0x40, 0x00, 0xBC, 0x40, 0x00, 0xBF, 0x40,
0x00, 0x8F, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x9C, 0x40, 0x00, 0x98, 0x40, 0x00,
0x90, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char gprs_4of4[] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x07, 0x08, 0x00, 0x07, 0x08,
0x00, 0x07, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x07, 0x77, 0x08, 0x07,
0x77, 0x08, 0x07, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77,
0x08, 0x77, 0x77
};

static const unsigned char gprs_off [] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x00, 0x08, 0x00, 0x00, 0x08,
0x07, 0x00, 0x08, 0x08, 0x80, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x08,
0x80, 0x08, 0x07, 0x40, 0x08, 0x00, 0x20, 0x08, 0x00, 0x10, 0x08, 0x00, 0x08, 0x08, 0x00, 0x00,
0x08, 0x00, 0x00
};

void setup() {

    System.on(cloud_status, cloudStatusEventHandler);
    System.on(network_status, networkStatusEventHandler);

    // SPI
    oled.begin(SSD1306_SWITCHCAPVCC);

    delay(50);
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);

    delay(100);
    fuel.quickStart(); // Start Fuel Gauge
    delay(200);

    deviceID = System.deviceID();
    firmwareVersion = System.version();
    Log.info("Wake up");

    delay(200);
    connectCellular();

    Particle.connect();
    isConnecting = true;

    Log.info("Setup battery");
    batteryStatus = fuel.getSoC();
    batteryStatus = map(batteryStatus, 0, 84, 0, 100);

    updateScreen = true;
    checkBattery = lastUSBCheck = millis();
}

void connectCellular() {
    Cellular.on();
    delay(2500);
    Cellular.connect(WIFI_CONNECT_SKIP_LISTEN); // Fix to ignore simcard glitches (?)
    delay(500);
}

void restartCellular() {
    Cellular.off();
    // this waitUntil was a mistake, but it should't cause the solid LED right?
    // (shouldn't the code get stuck in an infinite loop? Or is it run too quickly after 
    // Cellular.off() for the respective bools to get properly set?) 
    waitUntil(Cellular.ready);
    connectCellular();
}

    // --------------------------------------------------------- Check connection status
void cloudStatusEventHandler(system_event_t event, int param) {
    int e = event;
    int p = param;
    Log.info("*** CloudStatus: Got event %d with %d", e, p);

    if (p == 8) {
        connectionStatus = true;
    } else if ( p == 1 ) {
        handshakes++;
        connectionStatus = false;
    }
    else {
        connectionStatus = false;
    }
}

void networkStatusEventHandler(system_event_t event, int param){
    int e = event;
    int p = param;
    Log.info("*** Networkstatus: Got event %d with %d", e, p);
}

void loop() {

    // disconnect after 2 minutes
    if (connectionStatus) {
        if (millis() - lastDisconnectTimestamp > 2*60*1000) {
            lastDisconnectTimestamp = millis();
            isConnecting = false;
            isManualDisconnect = true;
            manualDisconnectCount++;
            Particle.disconnect();
            updateScreen = true;
        }
    } else if (!isConnecting) {
        restartCellular();
        isConnecting = true;
        Particle.connect();
    }

    // --------------------------------------------------------- Check connection status
    if (lastConnectionStatus != connectionStatus) {
        if(connectionStatus) {
            updateScreen = true;
            Log.info("Successfully connected");
        } else {
            updateScreen = true;
            lastDisconnectTimestamp = millis();
            if (!isManualDisconnect) {
                droppedDisconnectCount++;
                Log.info("Disconnected. Number of disconnects: %u", droppedDisconnectCount);
            }
            isManualDisconnect = false;
        }
        lastConnectionStatus = connectionStatus;
    }

    if ((millis() - lastUSBCheck > 2*1000)){
        statusUSB = pmic.getSystemStatus();
        if (lastUSB != statusUSB) {
            updateScreen = true;
            batteryStatus = fuel.getSoC();
            batteryStatus = map(batteryStatus, 0, 84, 0, 100);
            lastUSB = statusUSB;
        }
        lastUSBCheck = millis();
    }

    // --------------------------------------------------------- Update battery SoC every 20 minutes
    if ((millis() - checkBattery > 10*60*1000)) {
        updateScreen = true;
        batteryStatus = fuel.getSoC();
        batteryStatus = map(batteryStatus, 0, 84, 0, 100);
        checkBattery = millis();
    }

    if (updateScreen) {
        printDisplay();
        updateScreen = false;
    }
}

// --------------------------------------------------------- Print onto OLED
void printDisplay(){
    oled.clearDisplay();
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);
    oled.drawLine(0, 17, 128, 17, WHITE);

    // Print accumulated disconnect/handshakes status
    oled.setCursor(0,5);
    oled.print(ID);
    oled.setCursor(0,20);
    oled.print("Handshakes:");
    oled.setCursor(70,20);
    oled.print(handshakes);
    oled.setCursor(0,35);
    oled.print("Dropped:");
    oled.setCursor(70,35);
    oled.print(droppedDisconnectCount);
    oled.setCursor(0,50);
    oled.print("Manual:");
    oled.setCursor(70,50);
    oled.print(manualDisconnectCount);

    // Print battery status
    if((statusUSB == 4) || (statusUSB ==  36) || (statusUSB ==  100) || (statusUSB ==  116)) {
        oled.drawBitmap(51, -1, charging, 24, 24, WHITE);
    } else {
        if (batteryStatus > 75) {
            oled.drawBitmap(51, -1, batteri_4of4, 24, 24, WHITE);
        } else if (batteryStatus >= 50) {
            oled.drawBitmap(51, -1, batteri_3of4, 24, 24, WHITE);
        } else if (batteryStatus >= 25) {
            oled.drawBitmap(51, -1, batteri_2of4, 24, 24, WHITE);
        } else if (batteryStatus >= 10) {
            oled.drawBitmap(51, -1, batteri_1of4, 24, 24, WHITE);
        } else if (batteryStatus < 10) {
            oled.drawBitmap(51, -1, batteri_0of4, 24, 24, WHITE);
        }
    }

    oled.setCursor(63,4);
    if (batteryStatus > 100) {
        batteryStatus = 100;
    }
    oled.print(batteryStatus);
    oled.print("%");

    // Print connection status
    if (!connectionStatus) {
        oled.drawBitmap(102, 0, gprs_off, 24,17, WHITE);
    } else {
        oled.drawBitmap(102, 0, gprs_4of4, 24, 17, WHITE);
    }

    oled.display();
}

Observe the erroneous waitUntil in the restartCellular-function (and the accompanying comments).

Is there an obvious oversight even in this simple setup? How could this code possibly cause the units to freeze? Or is it actually an hardware/firmware (i.e. Particle Firmware) issue after all?

1 Like

I think you got the right idea about this waitUntil(Cellular.ready)

Thanks for your continuous responses @ScruffR :slight_smile:

Could you be more specific on what I’ve got the right idea about? (to clarify, I had multiple ideas about that particular code block). Or do you mean the code works as initially intended?

Thanks!

Sorry, I was refering to the quoted bit of your post about waitUntil() and your related comment in code

Hence I linked to another topic dealing with the same issue.

1 Like

Cool :thumbsup: We have updated the code and will continue the debugging process

1 Like

I do not use any other additional source code to maintain the connection except watcdog timer and for now works well. I use Firmware 0.6.1, SYSTEM_THREAD (ENABLED), and
SYSTEM_MODE (SEMI_AUTOMATIC). Sometimes a problem occurs with the loss of connection when it starts flashing rapidly green, and once blinking red, but then continues to operate normally.
Sometimes a problem occurs with the loss of connection with LED starts flashing rapidly cyan and then once flashes red. But then continue to work without any problem and without any intervention.

Hey @developer_bt,

So you never get the freeze that we’re experiencing? Are implying that the Watchdog kicks in when your loss of connection problem occurs?

We use the internal Watchdog, but this doesn’t help in our case since it crashes along with the rest of RTOS. Do you use 2G or 3G electrons btw?

Debug update:

We’ve run more tests and have had units freezing with the following slightly tweaked code from before. The essential code change (apart from keeping track and printing some additional data points) is the removal of the erroneous waitUntil from before and the addition of some generous delays during connect/reconnect:

#include "Adafruit_SSD1306.h"
#include "Adafruit_GFX.h"
#include "fonts.h"

using namespace std;

#include "application.h"


#define LOGGING_ON TRUE

#if (LOGGING_ON)
   SerialLogHandler logHandler(LOG_LEVEL_ALL);
#endif


/* ============== MAIN =====================*/

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

// New screen
#define OLED_DC     D0
#define OLED_CS     A4
#define OLED_RESET  D1
Adafruit_SSD1306 oled(OLED_DC, OLED_RESET, OLED_CS);

FuelGauge fuel;
PMIC pmic;

char version[] = "0.3b";
char ID[] = "xxxx";

unsigned long lastDisconnectTimestamp;
unsigned long unix_timestamp;

unsigned long checkBattery = 0;
unsigned long lastUSBCheck = 0;
unsigned long lastPublish = 0;
unsigned long lastUpdatedTS = 0;

int statusUSB = 0;
int lastUSB = 0;

bool updateScreen = false;
volatile bool connectionStatus = false;
bool lastConnectionStatus = false;
bool isConnecting = false;
int batteryStatus = 0;
bool isManualDisconnect = true;


retained int manualDisconnectCount;
retained int droppedDisconnectCount;
retained int panicCount;
retained volatile int handshakes;


const String clickerIDString = String(System.deviceID());

String deviceID;
String firmwareVersion;


int brickToggleTimestamp = 0;
bool brickToggle = false;   // just toggles true/false to move point on screen for easy visual brick-check

static const unsigned char batteri_4of4 [] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0xBF,
0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_3of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_2of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_1of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_0of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char charging[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x82, 0x40, 0x00, 0x86,
0x40, 0x00, 0x8E, 0x40, 0x00, 0x8C, 0x40, 0x00, 0x9C, 0x40, 0x00, 0xBC, 0x40, 0x00, 0xBF, 0x40,
0x00, 0x8F, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x9C, 0x40, 0x00, 0x98, 0x40, 0x00,
0x90, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char gprs_4of4[] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x07, 0x08, 0x00, 0x07, 0x08,
0x00, 0x07, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x07, 0x77, 0x08, 0x07,
0x77, 0x08, 0x07, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77,
0x08, 0x77, 0x77
};

static const unsigned char gprs_off [] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x00, 0x08, 0x00, 0x00, 0x08,
0x07, 0x00, 0x08, 0x08, 0x80, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x08,
0x80, 0x08, 0x07, 0x40, 0x08, 0x00, 0x20, 0x08, 0x00, 0x10, 0x08, 0x00, 0x08, 0x08, 0x00, 0x00,
0x08, 0x00, 0x00
};

void setup() {

    if (System.resetReason() == RESET_REASON_PANIC) { 
        panicCount++;
    } else {
        manualDisconnectCount = 0;
        droppedDisconnectCount = 0;
        panicCount = 0;
        handshakes = 0;
    }

    System.on(cloud_status, cloudStatusEventHandler);
    System.on(network_status, networkStatusEventHandler);

    // SPI
    oled.begin(SSD1306_SWITCHCAPVCC);

    delay(50);
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);

    delay(100);
    fuel.quickStart(); // Start Fuel Gauge
    delay(200);

    deviceID = System.deviceID();
    firmwareVersion = System.version();
    Log.info("Wake up");

    connectCellular();

    Particle.connect();
    isConnecting = true;

    Log.info("Setup battery");
    batteryStatus = fuel.getSoC();
    batteryStatus = map(batteryStatus, 0, 84, 0, 100);

    updateScreen = true;
    checkBattery = lastUSBCheck = millis();
}

void connectCellular() {
    delay(2000);
    Cellular.on();
    delay(2000);
    Cellular.connect(WIFI_CONNECT_SKIP_LISTEN); // Fix to ignore simcard glitches (?)
    delay(2000);
    waitUntil(Cellular.ready);
}

void restartCellular() {
    Cellular.off();
    connectCellular();
}

    // --------------------------------------------------------- Check connection status
void cloudStatusEventHandler(system_event_t event, int param) {
    int e = event;
    int p = param;
    Log.info("*** CloudStatus: Got event %d with %d", e, p);

    if (p == 8) {
        connectionStatus = true;
    } else if ( p == 1 ) {
        handshakes++;
        connectionStatus = false;
    }
    else {
        connectionStatus = false;
    }
}

void networkStatusEventHandler(system_event_t event, int param){
    int e = event;
    int p = param;
    Log.info("*** Networkstatus: Got event %d with %d", e, p);
}

void loop() {
    // update "brick status toggle" to visually show that unit isn't bricked
    if (millis() - brickToggleTimestamp > 1*1000) {
      brickToggleTimestamp = millis();
      brickToggle = !brickToggle;
      updateScreen = true;
    }

    // disconnect after 2 minutes
    if (connectionStatus) {
        if (millis() - lastDisconnectTimestamp > 1*45*1000) {
            lastDisconnectTimestamp = millis();
            isConnecting = false;
            isManualDisconnect = true;
            manualDisconnectCount++;
            Particle.disconnect();
            updateScreen = true;
        }
    } else if (!isConnecting) {
        restartCellular();
        isConnecting = true;
        Particle.connect();
    }

    // --------------------------------------------------------- Check connection status
    if (lastConnectionStatus != connectionStatus) {
        if(connectionStatus) {
            updateScreen = true;
            Log.info("Successfully connected");
        } else {
            updateScreen = true;
            lastDisconnectTimestamp = millis();
            if (!isManualDisconnect) {
                droppedDisconnectCount++;
                Log.info("Disconnected. Number of disconnects: %u", droppedDisconnectCount);
            }
            isManualDisconnect = false;
        }
        lastConnectionStatus = connectionStatus;
    }

    if ((millis() - lastUSBCheck > 2*1000)){
        statusUSB = pmic.getSystemStatus();
        if (lastUSB != statusUSB) {
            updateScreen = true;
            batteryStatus = fuel.getSoC();
            batteryStatus = map(batteryStatus, 0, 84, 0, 100);
            lastUSB = statusUSB;
        }
        lastUSBCheck = millis();
    }

    // --------------------------------------------------------- Update battery SoC every 20 minutes
    if ((millis() - checkBattery > 10*60*1000)) {
        updateScreen = true;
        batteryStatus = fuel.getSoC();
        batteryStatus = map(batteryStatus, 0, 84, 0, 100);
        checkBattery = millis();
    }

    if (updateScreen) {
        printDisplay();
        updateScreen = false;
    }
}

// --------------------------------------------------------- Print onto OLED
void printDisplay(){
    oled.clearDisplay();
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);
    oled.drawLine(0, 17, 128, 17, WHITE);

    // Print accumulated disconnect/handshakes status
    oled.setCursor(0,5);
    oled.print(ID);
    oled.setCursor(0,20);
    oled.print("Handshakes:");
    oled.setCursor(70,20);
    oled.print(handshakes);
    oled.setCursor(0,30);
    oled.print("Dropped:");
    oled.setCursor(70,30);
    oled.print(droppedDisconnectCount);
    oled.setCursor(0,40);
    oled.print("Manual:");
    oled.setCursor(70,40);
    oled.print(manualDisconnectCount);
    oled.setCursor(0,50);
    oled.print("Panic:");
    oled.setCursor(70,50);
    oled.print(panicCount);

    if (brickToggle) {
      oled.setCursor(108,20);
    } else {
      oled.setCursor(120,20);
    }
    oled.print("?");

    // Print battery status
    if((statusUSB == 4) || (statusUSB ==  36) || (statusUSB ==  100) || (statusUSB ==  116)) {
        oled.drawBitmap(51, -1, charging, 24, 24, WHITE);
    } else {
        if (batteryStatus > 75) {
            oled.drawBitmap(51, -1, batteri_4of4, 24, 24, WHITE);
        } else if (batteryStatus >= 50) {
            oled.drawBitmap(51, -1, batteri_3of4, 24, 24, WHITE);
        } else if (batteryStatus >= 25) {
            oled.drawBitmap(51, -1, batteri_2of4, 24, 24, WHITE);
        } else if (batteryStatus >= 10) {
            oled.drawBitmap(51, -1, batteri_1of4, 24, 24, WHITE);
        } else if (batteryStatus < 10) {
            oled.drawBitmap(51, -1, batteri_0of4, 24, 24, WHITE);
        }
    }

    oled.setCursor(63,4);
    if (batteryStatus > 100) {
        batteryStatus = 100;
    }
    oled.print(batteryStatus);
    oled.print("%");

    // Print connection status
    if (!connectionStatus) {
        oled.drawBitmap(102, 0, gprs_off, 24,17, WHITE);
    } else {
        oled.drawBitmap(102, 0, gprs_4of4, 24, 17, WHITE);
    }

    oled.display();
}

The units run in an environment where the Cellular coverage is very unreliable (disconnects occurs sporadically), which is a plausible setup where the units will be operating.

At this stage we have no choice but to recall all 2G Electrons since we cannot even guarantee their continuous operation in a basic setup, which is very unfortunate. We haven’t yet received the 3G Electrons, but are expecting them shortly and will straight away run them through our test suites.

As earlier, any help on how to fix or debug these issues properly will be greatly appreciated.

2 Likes

Hey there! I did a little digging and it sounds like you're working with our customer support team in parallel.

These freezes, with solid LED, is not specific states in the firmware/hardware (as other community threads also conclude). Instead, this seems to typically be caused by the batteries not having the capability to handle the power surges that may occur during normal operation – the most demanding of which occur when connecting the Electron to Cellular/Cloud :zap:. This seems to be very much in line with what we've experienced.

Do you still believe that the battery is the root cause here? How are your batteries connected to the product?

2 Likes

Hey Will,

Me and @robinandersson are working on this together, so I’ll chip in here.

We had this complete freeze occurring both with the battery you provide, at 2000mAh, and the ones we stocked at 1500mAh. The ones we stock lacks a JST male connector, so we solder them onto Li+ and GND, and the Particle ones we just connect using the connector. This should point us in the direction that the battery is not the issue here, unless there is a hardware-related issue with the specific Electron that was running the 2000mAh battery.

About a week back to started to systematically run the same code on multiple devices simultaneously. Our test setup consists of 14 Electrons, with eleven of them being powered by a 1500mAh battery and the rest by the 2000mAh one. Since then we’ve logged seven complete crashes on the Electron, with one of them running the bigger battery. But as we started logging data so recently, we still think we lack the quantity necessary to draw any proper conclusions on whether the battery is the root cause or not.

1 Like

@ftideman sounds good, and thank you for providing updates. I’m going to make sure @mohit and @bdub from the Particle team are aware of this thread in the event that the data you collect is suggestive of a hardware or firmware issue.

1 Like

One more follow-up – I see you have multithreading turned on…are you seeing the same hangups when you run our out-of-box “Tinker” firmware with multithreading active?

I have not heard any consistent reports of this the Electron 2G freezing, so my guess is that it’s not a hardware issue but probably an issue with either your user application or our system firmware.

@will we’ll set up our Electrons running the Tinker firmware to run this weekend to see if the same hangup occurs. I’m not sure what you mean by “with multithreading active” in this context though, since we can’t seem to find any option to activate multithreading in the Tinker app. Do you mean that it’s enabled by default, or do we need to do something else than simply flash the Tinker firmware OTA through the app?

1 Like

The Tinker app source is available on Web IDE and if you add SYSTEM_THREAD(ENABLED) to the top of the code you’ll have multithreaded Tinker to flash OTA from Web IDE or download the binary and flash via USB.

2 Likes

Thanks @ScruffR. We guessed that’s what he meant so we just did that. :slight_smile:

We’ll report back when we have any news.

3 Likes

Thank you! Looking forward to hearing the results!