Electron sometimes freezes on reconnect - Led Solid Cyan

robinandersson · May 7, 2017, 9:44pm

Hello Particle Community!

This is my first post! So, while I’ve been reading through the guidlines etc. I might get some things wrong, so please gently point me in the right direction if I misbehave I’m also a web developer by heart and super fresh into the Particle sphere, so I apologize in advance for potentially using the wrong vocabulary, etc.

To the issue!

We’re experiencing spontaneous freezing (see below for my definition of freezing) in my company’s Electrons (2G) that are being used by our customers, requiring manual restart through the onboard-reset button which entails opening of the devices’ chassis – not ideal since the devices are business critical to our customers, and customers themselves are located all over the country we’re operating in.

After a bunch of debugging I believe I’ve narrowed the freezing down to the devices loosing cellular/cloud connection and then trying (but seemingly failing) to reconnect. The freeze MIGHT then suddenly occur (i.e. sometimes it reconnects successfully, although I have yet to confirm this).

I managed to reproduce the issue while having the device connected to a terminal. The following logs were the last ones printed before this particular device froze:

0000761078 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 0
0000761178 [comm.protocol] ERROR: Event loop error 3
0000761178 [system] WARN: Communication loop error, closing cloud socket
0000761278 [system] INFO: Cloud: connecting
0000761278 [system] TRACE: sparkSocket Now =0
0000761278 [system] TRACE: Close Attempt
socketClose(0)
   761.277 AT send      12 "AT+USOCL=0\r\n"
   761.329 AT read OK    6 "\r\nOK\r\n"
socketFree(0)
0000761330 [system] TRACE: socket_close()=success
0000761330 [system] TRACE: HAL_FLASH_Read_ServerAddress() = type:1,domain:$id.udp.particle.io,ip: 46.100.105.36, port: 65535
   761.331 AT send      56 "AT+UDNSRN=0,\"33003e000351353337353037.udp.particle.io\"\r\n"
   763.025 AT read  +   28 "\r\n+UDNSRN: \"54.86.250.117\"\r\n"
   763.037 AT read OK    6 "\r\nOK\r\n"
0000763039 [system] INFO: Resolved host 33003e000351353337353037.udp.particle.io to 54.86.250.117
socketSocket(UDP)
   763.038 AT send      18 "AT+USOCR=17,5684\r\n"
   763.080 AT read  +   13 "\r\n+USOCR: 0\r\n"
   763.090 AT read OK    6 "\r\nOK\r\n"
Socket 0: handle 0 was created
0000763091 [system] TRACE: socketed udp=1, sparkSocket=0, 1
0000763091 [system] TRACE: connection attempt to 54.86.250.117:5684
0000763093 [system] INFO: Cloud socket connected
0000763093 [system] INFO: Starting handshake: presense_announce=0
0000763095 [comm.protocol.handshake] INFO: Establish secure connection
0000763117 [comm.dtls] TRACE: restore size mismatch 1: 0/220
0000763119 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=2
0000763121 [system] TRACE: send 107
socketSendTo(0,54.86.250.117,5684,,107)
   763.121 AT send      37 "AT+USOST=0,\"54.86.250.117\",5684,107\r\n"
0000763122 [app] INFO: Disconnected. Number of disconnects: 1
   763.172 AT read  >    3 "\r\n@"
   763.222 AT send     107 "\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x00\x00^\x01\x00\x00R\x00\x00\x00\x00\x00\x00\x00R\xfe\xfdI\x18Q7I\xf2\x0e\xfc\xb2\xe1_c\xe8\x9e|\xd4]Bk[\x06\xb9\x88B\x94\xac\xec\xe1\x89\x10\xe28\x00\x00\x00\x04\xc0\xae\x00\xff\x01\x00\x00$\x00\r\x00\x06\x00\x04\x04\x03\x03\x03\x00\n\x00\x04\x00\x02\x00\x17\x00\v\x00\x02\x01\x00\x00\x13\x00\x02\x01\x02\x00\x14\x00\x02\x01\x02"
   763.268 AT read  +   17 "\r\n+USOST: 0,107\r\n"
   763.278 AT read OK    6 "\r\nOK\r\n"
0000763673 [app] INFO: *********************** System.resetReason(): 0
0000763673 [app] INFO: *********************** System.freeMemory(): 87512
   764.208 AT read  +   17 "\r\n+UUSORD: 0,60\r\n"
Socket 0: handle 0 has 60 bytes pending
   764.218 AT send      16 "AT+USORF=0,813\r\n"
   764.268 AT read  +   98 "\r\n+USORF: 0,\"54.86.250.117\",5684,60,\"\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x00\x00/\x03\x00\x00#\x00\x00\x00\x00\x00\x00\x00#\xfe\xff Y\x0fN\x9f\xdb\x7f\x99\xb4\r\x1a\xa4\x9b}\xaa\x9f\xca\x13\nM\xaf\xbf\xfe\x81HN\xe9\xe5\xc0\xb7*\x8aW\""
   764.284 AT read UNK   2 "\r\n"
   764.294 AT read OK    6 "\r\nOK\r\n"
0000764295 [system] TRACE: received 60
0000764295 [system] TRACE: send 139
socketSendTo(0,54.86.250.117,5684,,139)
   764.296 AT send      37 "AT+USOST=0,\"54.86.250.117\",5684,139\r\n"
   764.348 AT read  >    3 "\r\n@"
   764.398 AT send     139 "\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x01\x00~\x01\x00\x00r\x00\x01\x00\x00\x00\x00\x00r\xfe\xfdI\x18Q7I\xf2\x0e\xfc\xb2\xe1_c\xe8\x9e|\xd4]Bk[\x06\xb9\x88B\x94\xac\xec\xe1\x89\x10\xe28\x00 Y\x0fN\x9f\xdb\x7f\x99\xb4\r\x1a\xa4\x9b}\xaa\x9f\xca\x13\nM\xaf\xbf\xfe\x81HN\xe9\xe5\xc0\xb7*\x8aW\x00\x04\xc0\xae\x00\xff\x01\x00\x00$\x00\r\x00\x06\x00\x04\x04\x03\x03\x03\x00\n\x00\x04\x00\x02\x00\x17\x00\v\x00\x02\x01\x00\x00\x13\x00\x02\x01\x02\x00\x14\x00\x02\x01\x02"
   764.446 AT read  +   17 "\r\n+USOST: 0,139\r\n"
   764.456 AT read OK    6 "\r\nOK\r\n"
0000767297 [system] TRACE: send 139
socketSendTo(0,54.86.250.117,5684,,139)
   767.296 AT send      37 "AT+USOST=0,\"54.86.250.117\",5684,139\r\n"
   767.348 AT read  >    3 "\r\n@"
   767.398 AT send     139 "\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x02\x00~\x01\x00\x00r\x00\x01\x00\x00\x00\x00\x00r\xfe\xfdI\x18Q7I\xf2\x0e\xfc\xb2\xe1_c\xe8\x9e|\xd4]Bk[\x06\xb9\x88B\x94\xac\xec\xe1\x89\x10\xe28\x00 Y\x0fN\x9f\xdb\x7f\x99\xb4\r\x1a\xa4\x9b}\xaa\x9f\xca\x13\nM\xaf\xbf\xfe\x81HN\xe9\xe5\xc0\xb7*\x8aW\x00\x04\xc0\xae\x00\xff\x01\x00\x00$\x00\r\x00\x06\x00\x04\x04\x03\x03\x03\x00\n\x00\x04\x00\x02\x00\x17\x00\v\x00\x02\x01\x00\x00\x13\x00\x02\x01\x02\x00\x14\x00\x02\x01\x02"
   767.448 AT read  +   17 "\r\n+USOST: 0,139\r\n"
   767.458 AT read OK    6 "\r\nOK\r\n"
0000767793 [app] INFO: *********************** System.resetReason(): 0
0000767794 [app] INFO: *********************** System.freeMemory(): 87444
0000771945 [app] INFO: *********************** System.resetReason(): 0
0000771945 [app] INFO: *********************** System.freeMemory(): 87444
lib/mbedtls/library/ssl_tls.c:2334: handshake timeout
lib/mbedtls/library/ssl_tls.c:3939: mbedtls_ssl_fetch_input() returned -26624 (-0x6800)
lib/mbedtls/library/ssl_cli.c:1526: mbedtls_ssl_read_record() returned -26624 (-0x6800)
0000773461 [comm.dtls] ERROR: handshake failed -6800
0000773463 [comm.protocol.handshake] ERROR: handshake failed with code 17
0000773463 [system] WARN: Cloud handshake failed, code=17
0000773713 [system] INFO: Cloud: disconnecting
0000773713 [system] TRACE: Close Attempt
socketClose(0)
   773.712 AT send      12 "AT+USOCL=0\r\n"
   773.754 AT read OK    6 "\r\nOK\r\n"
socketFree(0)
0000773755 [system] TRACE: socket_close()=success
0000773755 [system] INFO: Cloud: disconnected
0000774557 [system] INFO: Cloud: connecting
0000774557 [system] TRACE: sparkSocket Now =-1
0000774557 [system] TRACE: HAL_FLASH_Read_ServerAddress() = type:1,domain:$id.udp.particle.io,ip: 46.100.105.36, port: 65535
   774.558 AT send      56 "AT+UDNSRN=0,\"33003e000351353337353037.udp.particle.io\"\r\n"
   775.820 AT read  +   28 "\r\n+UDNSRN: \"54.164.79.148\"\r\n"
   775.832 AT read OK    6 "\r\nOK\r\n"
0000775834 [system] INFO: Resolved host 33003e000351353337353037.udp.particle.io to 54.164.79.148
socketSocket(UDP)
   775.833 AT send      18 "AT+USOCR=17,5684\r\n"
   775.875 AT read  +   13 "\r\n+USOCR: 0\r\n"
   775.885 AT read OK    6 "\r\nOK\r\n"
Socket 0: handle 0 was created
0000775886 [system] TRACE: socketed udp=1, sparkSocket=0, 1
0000775888 [system] TRACE: connection attempt to 54.164.79.148:5684
0000775888 [system] INFO: Cloud socket connected
0000775888 [system] INFO: Starting handshake: presense_announce=0
0000775890 [comm.protocol.handshake] INFO: Establish secure connection
0000775912 [comm.dtls] TRACE: restore size mismatch 1: 0/220
0000775914 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=2
0000775916 [system] TRACE: send 107
socketSendTo(0,54.164.79.148,5684,,107)
   775.917 AT send      37 "AT+USOST=0,\"54.164.79.148\",5684,107\r\n"
   775.969 AT read  >    3 "\r\n@"
   776.019 AT send     107 "\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x00\x00^\x01\x00\x00R\x00\x00\x00\x00\x00\x00\x00R\xfe\xfdT*\x1a\x1c%\b\x817\x93\x85\x1f\xef\x1e\xf6rl\x81w\x9f\x1c\xca\xad\x18\x82`:\xe6bo(\xc9\v\x00\x00\x00\x04\xc0\xae\x00\xff\x01\x00\x00$\x00\r\x00\x06\x00\x04\x04\x03\x03\x03\x00\n\x00\x04\x00\x02\x00\x17\x00\v\x00\x02\x01\x00\x00\x13\x00\x02\x01\x02\x00\x14\x00\x02\x01\x02"
   776.067 AT read  +   17 "\r\n+USOST: 0,107\r\n"
   776.077 AT read OK    6 "\r\nOK\r\n"
0000776195 [app] INFO: *********************** System.resetReason(): 0
0000776195 [app] INFO: *********************** System.freeMemory(): 87512
   776.857 AT read  +   17 "\r\n+UUSORD: 0,60\r\n"
Socket 0: handle 0 has 60 bytes pending
   776.869 AT send      16 "AT+USORF=0,813\r\n"
   776.919 AT read  +   98 "\r\n+USORF: 0,\"54.164.79.148\",5684,60,\"\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x00\x00/\x03\x00\x00#\x00\x00\x00\x00\x00\x00\x00#\xfe\xfd Y\x0fN\xac\x04\xfail\x83C\xc1zA\x8f\x8f^\x10>\x81jv\xf44\x17\xe4\xce\x8b\xefs\xa47\x1a\""
   776.935 AT read UNK   2 "\r\n"
   776.945 AT read OK    6 "\r\nOK\r\n"
0000776946 [system] TRACE: received 60
0000776946 [system] TRACE: send 139
socketSendTo(0,54.164.79.148,5684,,139)
   776.947 AT send      37 "AT+USOST=0,\"54.164.79.148\",5684,139\r\n"
   776.999 AT read  >    3 "\r\n@"
   777.049 AT send     139 "\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x01\x00~\x01\x00\x00r\x00\x01\x00\x00\x00\x00\x00r\xfe\xfdT*\x1a\x1c%\b\x817\x93\x85\x1f\xef\x1e\xf6rl\x81w\x9f\x1c\xca\xad\x18\x82`:\xe6bo(\xc9\v\x00 Y\x0fN\xac\x04\xfail\x83C\xc1zA\x8f\x8f^\x10>\x81jv\xf44\x17\xe4\xce\x8b\xefs\xa47\x1a\x00\x04\xc0\xae\x00\xff\x01\x00\x00$\x00\r\x00\x06\x00\x04\x04\x03\x03\x03\x00\n\x00\x04\x00\x02\x00\x17\x00\v\x00\x02\x01\x00\x00\x13\x00\x02\x01\x02\x00\x14\x00\x02\x01\x02"
   777.099 AT read  +   17 "\r\n+USOST: 0,139\r\n"
   777.109 AT read OK    6 "\r\nOK\r\n"
   777.949 AT read  +   18 "\r\n+UUSORD: 0,118\r\n"
Socket 0: handle 0 has 118 bytes pending
   777.961 AT send      16 "AT+USORF=0,813\r\n"
   778.011 AT read  +  157 "\r\n+USORF: 0,\"54.164.79.148\",5684,118,\"\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x01\x00i\x02\x00\x00]\x00\x01\x00\x00\x00\x00\x00]\xfe\xfdY\x0fN\xad\x05C=\x14\x8c@\xc7\xdbg\x8b\xb2(MX\x93\xd9\x13\xac8\xbe\x10Zi^\x10X\xdf\x1a \x00z-\xc3\x86\xb3\x91D\xa0\xea\x04\xd4\\\xa9dS\x03/N\xc6\xc5*\xb7\x92\xd1=\xd7r7y-V\xc0\xae\x00\x00\x15\xff\x01\x00\x01\x00\x00\v\x00\x02\x01\x00\x00\x13\x00\x01\x02\x00\x14\x00\x01\x02\""
   778.031 AT read UNK   2 "\r\n"
   778.041 AT read OK    6 "\r\nOK\r\n"
0000778042 [system] TRACE: received 118
   778.041 AT read  +   18 "\r\n+UUSORF: 0,228\r\n"
Socket 0: handle 0 has 228 bytes pending
   778.053 AT send      16 "AT+USORF=0,813\r\n"
   778.113 AT read  +  209 "\r\n+USORF: 0,\"54.164.79.148\",5684,170,\"\x16\xfe\xfd\x00\x00\x00\x00\x00\x00\x00\x02\x00\x9d\f\x00\x00\x91\x00\x02\x00\x00\x00\x00\x00\x91\x03\x00\x17A\x04\xb2\xa1\x1a\x03\xd45Hv\x85H\xf27\x9cD\x03\x86\x04\x0e\xd1K\x00+oG\n\xdf\x9f\xb9\xe1\xdf)v\x1eBC\x11\x81p\x80,\xc6\xacF\x1a5\xc7}\xae\xa9pj\r\xde\xee<\xbaJ\x02\x89\x9e\x85MIY\x04\x03\x00H0F\x02!\x00\xe0pO0\xb2\xd5d8\xedre.\xe3\xfe\xe6\xf7\x8a\x8c9\x9f\x1e\xcd\xae\xe5\x85\xbf:;\xa0\xad\xf6\xdc\x02!\x00\x9e\tZ\xe9\xd1l\xda2\xb6B\x16\xb0\xbeYA\xb1\x8a\xa7\xd9\f\xe2w\x87u6\xe2\x1b\x17\x11\xcdu\xca\""
   778.137 AT read UNK   2 "\r\n"
   778.147 AT read OK    6 "\r\nOK\r\n"
0000778148 [system] TRACE: received 170

I am looking for advice in how to fix this, or of how to better debug the problem. It is quite difficult to reproduce the issue, as the devices can run flawlessly for long periods of time . After hours of scouring the community forum (and Google) I’ve found several helpful advice, although but one reference to the same problem. As far as I can tell, this is the same issue as posted on github here (by @BDub?): https://github.com/spark/firmware/issues/1175, although what we’re experiencing seems to be more severe since the devices are effectively bricked in the eyes of our customers.

If this is indeed a known problem, are there other potential solution than as outlined in the github issue? Unfortunately, restricting the devices only to 3G (as suggested, if I understood it correctly) is not an option. [Edit: Although, possibly as last resort]

Thank you in advance!

Ps.

We’re running:

Firmware 0.6.1
SYSTEM_THREAD(ENABLED), and
SYSTEM_MODE(SEMI_AUTOMATIC).

To clarify what I mean by freezing (please correct me if freezing is not the correct term to use here) and what the setup is – the Electron is enclosed in a plastic casing, and attached to the Electron are a display and a few buttons for user interaction. The display still shows “reasonable” data after the freeze (i.e. no glitches – numbers from a valid state when the freeze occurred is shown), but the device does not respond to button clicks or data sent from the Particle cloud. Nothing is logged, and of course nothing changes on the display. Also, the cyan LED is solid on the Electron (red LED is off), rather than breathing (although this might be coincidental, sometimes the LED is actually completely off while the device/display is still on). [UPDATE: managed to produce another freeze, this one had it’s LED stuck in solid Green]. The serial monitor does not get disconnected when this freeze happens, rather the logs simply stops flowing.

Also, I suspect the code running on our Electrons needs to be posted, although I’m quite paranoid and reluctant to do so straight away (even though it is definitely not rocket science). If the issue can be solved without posting the code it would be tremendous (if not possible, well then by all means I’ll post necessary code).

Futhermore, I do have more logs as well as photographs etc. if needed.

Ds.

ScruffR · May 9, 2017, 10:15am

Your issue sounds similar to this

robinandersson · May 11, 2017, 10:36am

After much debugging (running different code variations on ~8 Electrons continuously/without sleep overnight) and discussing with more experienced hardware developers/electrical engineers, I can post an initial update on the issue (not the least, for future developers experiencing the same issue):

These freezes, with solid LED, is not specific states in the firmware/hardware (as other community threads also conclude). Instead, this seems to typically be caused by the batteries not having the capability to handle the power surges that may occur during normal operation – the most demanding of which occur when connecting the Electron to Cellular/Cloud ️. This seems to be very much in line with what we’ve experienced.

As it turns out, we run our Electrons mostly with batteries that are slightly underpowered. However, the batteries have “burst” abilities that are sufficient for even the highest specified power surge that can occur; as per Particle’s specification. We’ve not yet been able to completely iron out these freezes with code modifications (e.g. restricting successive disconnects, and more). Although, as of last night we’ve also experienced this freeze with the included Battery from Particle that definitely should be able to handle the load Our rationale thus is that it might be code related still.

Our search continues, and we’ll try to find a code setup that let’s our Electrons run smoothly. Stay tuned.

ScruffR · May 11, 2017, 11:18am

Whether the battery will be able to cope or not also depends on your extra components powered off this same battery.

And 2G is known to have higher current demands than 3G

robinandersson · May 11, 2017, 11:46am

Good points worth pointing out @ScruffR we’ve considered this as well.

Of the components listed in my (very long) initial post; the buttons are passive components, and the screen is only 25 mA. So this should not cause issues since the battery can handle 1,5 A continuous discharge with burst/pulse discharge at 3 A (which our battery provider claims can keep up for ~10 minutes at a time). Particle specifies that their hardware can at most require 1,8 A.

As for the 2G v. 3G current demands; this has also come to light. We actually ordered 3G electrons at the start of this week, and will hopefully get them early next week.

robinandersson · May 12, 2017, 9:06am

Okay so this is weird.

We couldn’t pinpoint where in our code history the freezes started to occur (we managed to freeze one unit with code that we thought was completely safe). We reduced our code to a very barebones setup to see if the units would even manage that. Out of eight units, one had its LED stuck in solid green Four among them powered down because of low battery, so there might actually have been more stuck units (will look into the logs and update this post accordingly) – but 1/8 is still way too bad, especially for such a short cycle

So, we had 8 units trying out the following code overnight; the only demanding thing it does is basically to connect to the cloud, and after 2 minutes disconnect and connect again. Apart from this, it also keeps track of manual/dropped disconnects and handshakes, and prints them to the connected screen:

#include "Adafruit_SSD1306.h"
#include "Adafruit_GFX.h"
#include "fonts.h"

using namespace std;

#include "application.h"


#define LOGGING_ON TRUE

#if (LOGGING_ON)
   SerialLogHandler logHandler(LOG_LEVEL_ALL);
#endif


/* ============== MAIN =====================*/

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

// New screen
#define OLED_DC     D0
#define OLED_CS     A4
#define OLED_RESET  D1
Adafruit_SSD1306 oled(OLED_DC, OLED_RESET, OLED_CS);

FuelGauge fuel;
PMIC pmic;

char version[] = "0.1";
char ID[] = "8015";

unsigned long lastDisconnectTimestamp;
unsigned long unix_timestamp;

volatile int handshakes = 0;

unsigned long checkBattery = 0;
unsigned long lastUSBCheck = 0;
unsigned long lastPublish = 0;
unsigned long lastUpdatedTS = 0;

int statusUSB = 0;
int lastUSB = 0;

bool updateScreen = false;
bool isManualDisconnect = true;
volatile bool connectionStatus = false;
bool lastConnectionStatus = false;
bool isConnecting = false;
int batteryStatus = 0;

const String clickerIDString = String(System.deviceID());

String deviceID;
String firmwareVersion;

int panicCount = 0;
int manualDisconnectCount = 0;
int droppedDisconnectCount = 0;

static const unsigned char batteri_4of4 [] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0xBF,
0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_3of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_2of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_1of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_0of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char charging[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x82, 0x40, 0x00, 0x86,
0x40, 0x00, 0x8E, 0x40, 0x00, 0x8C, 0x40, 0x00, 0x9C, 0x40, 0x00, 0xBC, 0x40, 0x00, 0xBF, 0x40,
0x00, 0x8F, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x9C, 0x40, 0x00, 0x98, 0x40, 0x00,
0x90, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char gprs_4of4[] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x07, 0x08, 0x00, 0x07, 0x08,
0x00, 0x07, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x07, 0x77, 0x08, 0x07,
0x77, 0x08, 0x07, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77,
0x08, 0x77, 0x77
};

static const unsigned char gprs_off [] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x00, 0x08, 0x00, 0x00, 0x08,
0x07, 0x00, 0x08, 0x08, 0x80, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x08,
0x80, 0x08, 0x07, 0x40, 0x08, 0x00, 0x20, 0x08, 0x00, 0x10, 0x08, 0x00, 0x08, 0x08, 0x00, 0x00,
0x08, 0x00, 0x00
};

void setup() {

    System.on(cloud_status, cloudStatusEventHandler);
    System.on(network_status, networkStatusEventHandler);

    // SPI
    oled.begin(SSD1306_SWITCHCAPVCC);

    delay(50);
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);

    delay(100);
    fuel.quickStart(); // Start Fuel Gauge
    delay(200);

    deviceID = System.deviceID();
    firmwareVersion = System.version();
    Log.info("Wake up");

    delay(200);
    connectCellular();

    Particle.connect();
    isConnecting = true;

    Log.info("Setup battery");
    batteryStatus = fuel.getSoC();
    batteryStatus = map(batteryStatus, 0, 84, 0, 100);

    updateScreen = true;
    checkBattery = lastUSBCheck = millis();
}

void connectCellular() {
    Cellular.on();
    delay(2500);
    Cellular.connect(WIFI_CONNECT_SKIP_LISTEN); // Fix to ignore simcard glitches (?)
    delay(500);
}

void restartCellular() {
    Cellular.off();
    // this waitUntil was a mistake, but it should't cause the solid LED right?
    // (shouldn't the code get stuck in an infinite loop? Or is it run too quickly after 
    // Cellular.off() for the respective bools to get properly set?) 
    waitUntil(Cellular.ready);
    connectCellular();
}

    // --------------------------------------------------------- Check connection status
void cloudStatusEventHandler(system_event_t event, int param) {
    int e = event;
    int p = param;
    Log.info("*** CloudStatus: Got event %d with %d", e, p);

    if (p == 8) {
        connectionStatus = true;
    } else if ( p == 1 ) {
        handshakes++;
        connectionStatus = false;
    }
    else {
        connectionStatus = false;
    }
}

void networkStatusEventHandler(system_event_t event, int param){
    int e = event;
    int p = param;
    Log.info("*** Networkstatus: Got event %d with %d", e, p);
}

void loop() {

    // disconnect after 2 minutes
    if (connectionStatus) {
        if (millis() - lastDisconnectTimestamp > 2*60*1000) {
            lastDisconnectTimestamp = millis();
            isConnecting = false;
            isManualDisconnect = true;
            manualDisconnectCount++;
            Particle.disconnect();
            updateScreen = true;
        }
    } else if (!isConnecting) {
        restartCellular();
        isConnecting = true;
        Particle.connect();
    }

    // --------------------------------------------------------- Check connection status
    if (lastConnectionStatus != connectionStatus) {
        if(connectionStatus) {
            updateScreen = true;
            Log.info("Successfully connected");
        } else {
            updateScreen = true;
            lastDisconnectTimestamp = millis();
            if (!isManualDisconnect) {
                droppedDisconnectCount++;
                Log.info("Disconnected. Number of disconnects: %u", droppedDisconnectCount);
            }
            isManualDisconnect = false;
        }
        lastConnectionStatus = connectionStatus;
    }

    if ((millis() - lastUSBCheck > 2*1000)){
        statusUSB = pmic.getSystemStatus();
        if (lastUSB != statusUSB) {
            updateScreen = true;
            batteryStatus = fuel.getSoC();
            batteryStatus = map(batteryStatus, 0, 84, 0, 100);
            lastUSB = statusUSB;
        }
        lastUSBCheck = millis();
    }

    // --------------------------------------------------------- Update battery SoC every 20 minutes
    if ((millis() - checkBattery > 10*60*1000)) {
        updateScreen = true;
        batteryStatus = fuel.getSoC();
        batteryStatus = map(batteryStatus, 0, 84, 0, 100);
        checkBattery = millis();
    }

    if (updateScreen) {
        printDisplay();
        updateScreen = false;
    }
}

// --------------------------------------------------------- Print onto OLED
void printDisplay(){
    oled.clearDisplay();
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);
    oled.drawLine(0, 17, 128, 17, WHITE);

    // Print accumulated disconnect/handshakes status
    oled.setCursor(0,5);
    oled.print(ID);
    oled.setCursor(0,20);
    oled.print("Handshakes:");
    oled.setCursor(70,20);
    oled.print(handshakes);
    oled.setCursor(0,35);
    oled.print("Dropped:");
    oled.setCursor(70,35);
    oled.print(droppedDisconnectCount);
    oled.setCursor(0,50);
    oled.print("Manual:");
    oled.setCursor(70,50);
    oled.print(manualDisconnectCount);

    // Print battery status
    if((statusUSB == 4) || (statusUSB ==  36) || (statusUSB ==  100) || (statusUSB ==  116)) {
        oled.drawBitmap(51, -1, charging, 24, 24, WHITE);
    } else {
        if (batteryStatus > 75) {
            oled.drawBitmap(51, -1, batteri_4of4, 24, 24, WHITE);
        } else if (batteryStatus >= 50) {
            oled.drawBitmap(51, -1, batteri_3of4, 24, 24, WHITE);
        } else if (batteryStatus >= 25) {
            oled.drawBitmap(51, -1, batteri_2of4, 24, 24, WHITE);
        } else if (batteryStatus >= 10) {
            oled.drawBitmap(51, -1, batteri_1of4, 24, 24, WHITE);
        } else if (batteryStatus < 10) {
            oled.drawBitmap(51, -1, batteri_0of4, 24, 24, WHITE);
        }
    }

    oled.setCursor(63,4);
    if (batteryStatus > 100) {
        batteryStatus = 100;
    }
    oled.print(batteryStatus);
    oled.print("%");

    // Print connection status
    if (!connectionStatus) {
        oled.drawBitmap(102, 0, gprs_off, 24,17, WHITE);
    } else {
        oled.drawBitmap(102, 0, gprs_4of4, 24, 17, WHITE);
    }

    oled.display();
}

Observe the erroneous waitUntil in the restartCellular-function (and the accompanying comments).

Is there an obvious oversight even in this simple setup? How could this code possibly cause the units to freeze? Or is it actually an hardware/firmware (i.e. Particle Firmware) issue after all?

ScruffR · May 12, 2017, 11:46am

I think you got the right idea about this waitUntil(Cellular.ready)

robinandersson · May 12, 2017, 12:09pm

Thanks for your continuous responses @ScruffR

Could you be more specific on what I’ve got the right idea about? (to clarify, I had multiple ideas about that particular code block). Or do you mean the code works as initially intended?

Thanks!

ScruffR · May 12, 2017, 12:19pm

Sorry, I was refering to the quoted bit of your post about waitUntil() and your related comment in code

Hence I linked to another topic dealing with the same issue.

robinandersson · May 12, 2017, 12:38pm

Cool We have updated the code and will continue the debugging process

developer_bt · May 12, 2017, 5:28pm

I do not use any other additional source code to maintain the connection except watcdog timer and for now works well. I use Firmware 0.6.1, SYSTEM_THREAD (ENABLED), and
SYSTEM_MODE (SEMI_AUTOMATIC). Sometimes a problem occurs with the loss of connection when it starts flashing rapidly green, and once blinking red, but then continues to operate normally.
Sometimes a problem occurs with the loss of connection with LED starts flashing rapidly cyan and then once flashes red. But then continue to work without any problem and without any intervention.

robinandersson · May 17, 2017, 10:03am

Hey @developer_bt,

So you never get the freeze that we’re experiencing? Are implying that the Watchdog kicks in when your loss of connection problem occurs?

We use the internal Watchdog, but this doesn’t help in our case since it crashes along with the rest of RTOS. Do you use 2G or 3G electrons btw?

robinandersson · May 17, 2017, 10:36am

Debug update:

We’ve run more tests and have had units freezing with the following slightly tweaked code from before. The essential code change (apart from keeping track and printing some additional data points) is the removal of the erroneous waitUntil from before and the addition of some generous delays during connect/reconnect:

#include "Adafruit_SSD1306.h"
#include "Adafruit_GFX.h"
#include "fonts.h"

using namespace std;

#include "application.h"


#define LOGGING_ON TRUE

#if (LOGGING_ON)
   SerialLogHandler logHandler(LOG_LEVEL_ALL);
#endif


/* ============== MAIN =====================*/

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

// New screen
#define OLED_DC     D0
#define OLED_CS     A4
#define OLED_RESET  D1
Adafruit_SSD1306 oled(OLED_DC, OLED_RESET, OLED_CS);

FuelGauge fuel;
PMIC pmic;

char version[] = "0.3b";
char ID[] = "xxxx";

unsigned long lastDisconnectTimestamp;
unsigned long unix_timestamp;

unsigned long checkBattery = 0;
unsigned long lastUSBCheck = 0;
unsigned long lastPublish = 0;
unsigned long lastUpdatedTS = 0;

int statusUSB = 0;
int lastUSB = 0;

bool updateScreen = false;
volatile bool connectionStatus = false;
bool lastConnectionStatus = false;
bool isConnecting = false;
int batteryStatus = 0;
bool isManualDisconnect = true;


retained int manualDisconnectCount;
retained int droppedDisconnectCount;
retained int panicCount;
retained volatile int handshakes;


const String clickerIDString = String(System.deviceID());

String deviceID;
String firmwareVersion;


int brickToggleTimestamp = 0;
bool brickToggle = false;   // just toggles true/false to move point on screen for easy visual brick-check

static const unsigned char batteri_4of4 [] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0xBF,
0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_3of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_2of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_1of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0xBF, 0x40, 0x00, 0xBF, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char batteri_0of4[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x80, 0x40, 0x00, 0x80,
0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40,
0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00, 0x80, 0x40, 0x00,
0x80, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char charging[] = {
0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x21, 0x00, 0x00, 0xFF, 0xC0, 0x00, 0x82, 0x40, 0x00, 0x86,
0x40, 0x00, 0x8E, 0x40, 0x00, 0x8C, 0x40, 0x00, 0x9C, 0x40, 0x00, 0xBC, 0x40, 0x00, 0xBF, 0x40,
0x00, 0x8F, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x8E, 0x40, 0x00, 0x9C, 0x40, 0x00, 0x98, 0x40, 0x00,
0x90, 0x40, 0x00, 0xFF, 0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

static const unsigned char gprs_4of4[] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x07, 0x08, 0x00, 0x07, 0x08,
0x00, 0x07, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x00, 0x77, 0x08, 0x07, 0x77, 0x08, 0x07,
0x77, 0x08, 0x07, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77, 0x08, 0x77, 0x77,
0x08, 0x77, 0x77
};

static const unsigned char gprs_off [] = {
0xFF, 0x80, 0x00, 0x49, 0x00, 0x00, 0x2A, 0x00, 0x00, 0x1C, 0x00, 0x00, 0x08, 0x00, 0x00, 0x08,
0x07, 0x00, 0x08, 0x08, 0x80, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x10, 0x40, 0x08, 0x08,
0x80, 0x08, 0x07, 0x40, 0x08, 0x00, 0x20, 0x08, 0x00, 0x10, 0x08, 0x00, 0x08, 0x08, 0x00, 0x00,
0x08, 0x00, 0x00
};

void setup() {

    if (System.resetReason() == RESET_REASON_PANIC) { 
        panicCount++;
    } else {
        manualDisconnectCount = 0;
        droppedDisconnectCount = 0;
        panicCount = 0;
        handshakes = 0;
    }

    System.on(cloud_status, cloudStatusEventHandler);
    System.on(network_status, networkStatusEventHandler);

    // SPI
    oled.begin(SSD1306_SWITCHCAPVCC);

    delay(50);
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);

    delay(100);
    fuel.quickStart(); // Start Fuel Gauge
    delay(200);

    deviceID = System.deviceID();
    firmwareVersion = System.version();
    Log.info("Wake up");

    connectCellular();

    Particle.connect();
    isConnecting = true;

    Log.info("Setup battery");
    batteryStatus = fuel.getSoC();
    batteryStatus = map(batteryStatus, 0, 84, 0, 100);

    updateScreen = true;
    checkBattery = lastUSBCheck = millis();
}

void connectCellular() {
    delay(2000);
    Cellular.on();
    delay(2000);
    Cellular.connect(WIFI_CONNECT_SKIP_LISTEN); // Fix to ignore simcard glitches (?)
    delay(2000);
    waitUntil(Cellular.ready);
}

void restartCellular() {
    Cellular.off();
    connectCellular();
}

    // --------------------------------------------------------- Check connection status
void cloudStatusEventHandler(system_event_t event, int param) {
    int e = event;
    int p = param;
    Log.info("*** CloudStatus: Got event %d with %d", e, p);

    if (p == 8) {
        connectionStatus = true;
    } else if ( p == 1 ) {
        handshakes++;
        connectionStatus = false;
    }
    else {
        connectionStatus = false;
    }
}

void networkStatusEventHandler(system_event_t event, int param){
    int e = event;
    int p = param;
    Log.info("*** Networkstatus: Got event %d with %d", e, p);
}

void loop() {
    // update "brick status toggle" to visually show that unit isn't bricked
    if (millis() - brickToggleTimestamp > 1*1000) {
      brickToggleTimestamp = millis();
      brickToggle = !brickToggle;
      updateScreen = true;
    }

    // disconnect after 2 minutes
    if (connectionStatus) {
        if (millis() - lastDisconnectTimestamp > 1*45*1000) {
            lastDisconnectTimestamp = millis();
            isConnecting = false;
            isManualDisconnect = true;
            manualDisconnectCount++;
            Particle.disconnect();
            updateScreen = true;
        }
    } else if (!isConnecting) {
        restartCellular();
        isConnecting = true;
        Particle.connect();
    }

    // --------------------------------------------------------- Check connection status
    if (lastConnectionStatus != connectionStatus) {
        if(connectionStatus) {
            updateScreen = true;
            Log.info("Successfully connected");
        } else {
            updateScreen = true;
            lastDisconnectTimestamp = millis();
            if (!isManualDisconnect) {
                droppedDisconnectCount++;
                Log.info("Disconnected. Number of disconnects: %u", droppedDisconnectCount);
            }
            isManualDisconnect = false;
        }
        lastConnectionStatus = connectionStatus;
    }

    if ((millis() - lastUSBCheck > 2*1000)){
        statusUSB = pmic.getSystemStatus();
        if (lastUSB != statusUSB) {
            updateScreen = true;
            batteryStatus = fuel.getSoC();
            batteryStatus = map(batteryStatus, 0, 84, 0, 100);
            lastUSB = statusUSB;
        }
        lastUSBCheck = millis();
    }

    // --------------------------------------------------------- Update battery SoC every 20 minutes
    if ((millis() - checkBattery > 10*60*1000)) {
        updateScreen = true;
        batteryStatus = fuel.getSoC();
        batteryStatus = map(batteryStatus, 0, 84, 0, 100);
        checkBattery = millis();
    }

    if (updateScreen) {
        printDisplay();
        updateScreen = false;
    }
}

// --------------------------------------------------------- Print onto OLED
void printDisplay(){
    oled.clearDisplay();
    oled.setTextSize(1);
    oled.setTextColor(WHITE);
    oled.setFont(TEXT);
    oled.drawLine(0, 17, 128, 17, WHITE);

    // Print accumulated disconnect/handshakes status
    oled.setCursor(0,5);
    oled.print(ID);
    oled.setCursor(0,20);
    oled.print("Handshakes:");
    oled.setCursor(70,20);
    oled.print(handshakes);
    oled.setCursor(0,30);
    oled.print("Dropped:");
    oled.setCursor(70,30);
    oled.print(droppedDisconnectCount);
    oled.setCursor(0,40);
    oled.print("Manual:");
    oled.setCursor(70,40);
    oled.print(manualDisconnectCount);
    oled.setCursor(0,50);
    oled.print("Panic:");
    oled.setCursor(70,50);
    oled.print(panicCount);

    if (brickToggle) {
      oled.setCursor(108,20);
    } else {
      oled.setCursor(120,20);
    }
    oled.print("?");

    // Print battery status
    if((statusUSB == 4) || (statusUSB ==  36) || (statusUSB ==  100) || (statusUSB ==  116)) {
        oled.drawBitmap(51, -1, charging, 24, 24, WHITE);
    } else {
        if (batteryStatus > 75) {
            oled.drawBitmap(51, -1, batteri_4of4, 24, 24, WHITE);
        } else if (batteryStatus >= 50) {
            oled.drawBitmap(51, -1, batteri_3of4, 24, 24, WHITE);
        } else if (batteryStatus >= 25) {
            oled.drawBitmap(51, -1, batteri_2of4, 24, 24, WHITE);
        } else if (batteryStatus >= 10) {
            oled.drawBitmap(51, -1, batteri_1of4, 24, 24, WHITE);
        } else if (batteryStatus < 10) {
            oled.drawBitmap(51, -1, batteri_0of4, 24, 24, WHITE);
        }
    }

    oled.setCursor(63,4);
    if (batteryStatus > 100) {
        batteryStatus = 100;
    }
    oled.print(batteryStatus);
    oled.print("%");

    // Print connection status
    if (!connectionStatus) {
        oled.drawBitmap(102, 0, gprs_off, 24,17, WHITE);
    } else {
        oled.drawBitmap(102, 0, gprs_4of4, 24, 17, WHITE);
    }

    oled.display();
}

The units run in an environment where the Cellular coverage is very unreliable (disconnects occurs sporadically), which is a plausible setup where the units will be operating.

At this stage we have no choice but to recall all 2G Electrons since we cannot even guarantee their continuous operation in a basic setup, which is very unfortunate. We haven’t yet received the 3G Electrons, but are expecting them shortly and will straight away run them through our test suites.

As earlier, any help on how to fix or debug these issues properly will be greatly appreciated.

will · May 17, 2017, 6:10pm

Hey there! I did a little digging and it sounds like you're working with our customer support team in parallel.

These freezes, with solid LED, is not specific states in the firmware/hardware (as other community threads also conclude). Instead, this seems to typically be caused by the batteries not having the capability to handle the power surges that may occur during normal operation – the most demanding of which occur when connecting the Electron to Cellular/Cloud . This seems to be very much in line with what we've experienced.

Do you still believe that the battery is the root cause here? How are your batteries connected to the product?

ftideman · May 18, 2017, 2:24pm

Hey Will,

Me and @robinandersson are working on this together, so I’ll chip in here.

We had this complete freeze occurring both with the battery you provide, at 2000mAh, and the ones we stocked at 1500mAh. The ones we stock lacks a JST male connector, so we solder them onto Li+ and GND, and the Particle ones we just connect using the connector. This should point us in the direction that the battery is not the issue here, unless there is a hardware-related issue with the specific Electron that was running the 2000mAh battery.

About a week back to started to systematically run the same code on multiple devices simultaneously. Our test setup consists of 14 Electrons, with eleven of them being powered by a 1500mAh battery and the rest by the 2000mAh one. Since then we’ve logged seven complete crashes on the Electron, with one of them running the bigger battery. But as we started logging data so recently, we still think we lack the quantity necessary to draw any proper conclusions on whether the battery is the root cause or not.

will · May 18, 2017, 4:53pm

@ftideman sounds good, and thank you for providing updates. I’m going to make sure @mohit and @bdub from the Particle team are aware of this thread in the event that the data you collect is suggestive of a hardware or firmware issue.

will · May 18, 2017, 5:23pm

One more follow-up – I see you have multithreading turned on…are you seeing the same hangups when you run our out-of-box “Tinker” firmware with multithreading active?

I have not heard any consistent reports of this the Electron 2G freezing, so my guess is that it’s not a hardware issue but probably an issue with either your user application or our system firmware.

ftideman · May 19, 2017, 11:02am

@will we’ll set up our Electrons running the Tinker firmware to run this weekend to see if the same hangup occurs. I’m not sure what you mean by “with multithreading active” in this context though, since we can’t seem to find any option to activate multithreading in the Tinker app. Do you mean that it’s enabled by default, or do we need to do something else than simply flash the Tinker firmware OTA through the app?

ScruffR · May 19, 2017, 12:37pm

The Tinker app source is available on Web IDE and if you add SYSTEM_THREAD(ENABLED) to the top of the code you’ll have multithreaded Tinker to flash OTA from Web IDE or download the binary and flash via USB.

ftideman · May 19, 2017, 12:45pm

Thanks @ScruffR. We guessed that’s what he meant so we just did that.

We’ll report back when we have any news.

Topic		Replies	Views
Electron Connection Down General	21	1690	July 11, 2018
Electron Flashing Green - Will not connect until battery removed Troubleshooting	14	3288	January 27, 2019
Electron: how to freeze with solid cyan/green Troubleshooting	4	666	October 4, 2018
Electron loses cloud connection, won't automatically reconnect Troubleshooting	2	739	March 26, 2019
Electrons Disconnected Simultaneously Across the US Troubleshooting	11	1205	August 23, 2018

Electron sometimes freezes on reconnect - Led Solid Cyan

Related topics