Hello Readers,
I’ve got a Boron 2G/3G device hooked up to another device via Serial connection. The goal of this is to monitor and control this other device via serial (no I2C or SPI sadly) since it does not have 2G/3G capabilities.
I have a FSM and when running the Boron for a long period of time (4-24h) I find that the device locks up (i.e. solid Cyan) in its idle phase randomly (i.e. random time). The device never leaves the idle phase and during its idle phase it does the following:
- Ping the device via Serial1
-
- if the device response with data (i.e. data != “”), keep the LED breathing Cyan
-
- if the device does not response and times out (5s wait), blink the LED Red
- This state exits and ends the loop() really quick
- This state loops around on idle until a web call happens where the handler changes the next state. The other states will send different serial commands to the device but the dead lock happens in the idle state so I won’t go into details about the other states
One recent discovery is that monitoring the device via serial I see an error message. I’m rerunning some tests to see if this message appears consistently. Could this be the reason for a lock up?
0006072005 [comm.protocol] ERROR: Event loop error 1
0006072005 [system] WARN: Communication loop error, closing cloud socket
Some questions I have for readers
- Any tips for debugging a locked up device?
- Anyone know what “Event loop error 1” means?
- Anyone have a Boron running in the field for 1-2days + ?
- A simplified version of my code is posted below, if anyone can spot anything it would be much appreciated.
I’ve been debugging this issue for the last 2 weeks so any help or suggestions would be greatly appreciated =(.
I also tried to include a 5min application watch dog in attempt for the system to recover. If the system locks up, I expected the watchdog to kick in and reset the system. But this did not work as intended. I guess a side question is, does the watchdog also get locked up during this mode?
ApplicationWatchdog wd(300000, System.reset); // Watchdog set at 5mins
Set up
- Boron 2G/3G, DeviceOS 0.9.0, powered by 5V 2.4A
- 3rd party device connected to Serial 1 via…
-
- 3.3V for power
-
- GND pin
-
- P0.08 for RX
-
- P0.06 for TX
Stripped down version of my code
// Global Variables
unsigned int gBaudRate = 9600;
States gNextState, gCurrentState;
unsigned int gWebTimeout = 500;
unsigned int gDeviceCheckPeriod = 5000; // Orginal 1min
unsigned int gSerialTimeout = 5000;
unsigned int gMonitorPeriod = 10000;
unsigned int gSendDataPeriod = 31000;
int gRedPin = D8; // Pins for RGB LED
int gGreenPin = D6;
int gBluePin = D5;
// Declarations
SYSTEM_MODE(SEMI_AUTOMATIC); // Do not automatically connect to cloud
SerialLogHandler logHandler(LOG_LEVEL_ALL, { // Logging level for non-application messages
{ "app", LOG_LEVEL_ALL } // Default logging level for all application messages
});
WebHooks* web;
SerialController* serialController;
LED* led;
PMIC pmic;
ApplicationWatchdog wd(300000, System.reset); // Watchdog set at 5mins
// States Initialization
WaitingForAuth waitingForAuth(gSerialTimeout, gDeviceCheckPeriod);
...
void setup() {
gNextState = WAITING_FOR_AUTH;
gCurrentState = WAITING_FOR_AUTH;
web = new WebHooks(); // This registers all cloud components
serialController = new SerialController(gBaudRate);
led = new LED(gRedPin, gGreenPin, gBluePin);
// Disable charging
pmic.begin();
pmic.disableCharging();
pmic.disableWatchdog();
// Switch to Roger's IOT SIMs before connecting
Cellular.setActiveSim(EXTERNAL_SIM);
Cellular.setCredentials("m2minternet.apn");
Particle.keepAlive(30); // This is needed if we are using Roger's network
Particle.connect(); // Make sure connect is alway last since we want to register all cloud components first
}
void loop() {
switch (gCurrentState) {
case WAITING_FOR_AUTH:
gCurrentState = waitingForAuth.run(gWebTimeout, serialController, led);
break;
case STATE_1:
...
break;
case STATE_2:
...
break;
case State_3:
...
break;
default:
...
break;
States WaitingForAuth::run(unsigned int timeout, SerialController* serialController, LED* led) {
if (millis() > mDeviceCheckTime) {
// sendCMD just sends a serial command, waits for response and returns the String
String temp = serialController->sendCMD("Some serial command", mSerialTimeout);
mDeviceCheckTime = millis() + mDeviceCheckPeriod;
Log.trace("mDeviceCheckTime is: " + String(mDeviceCheckTime));
if (temp.equals("")) {
Log.error(memberName + "Device is not responding!");
led->blinkRed();
} else {
Log.trace("Device is alive:");
Log.trace("\t" + temp);
led->idleColor();
}
}
return gNextState;
}
LED::LED(int redPin, int greenPin, int bluePin) {
this->mBlinkOrange = new LEDStatus(RGB_COLOR_ORANGE, LED_PATTERN_BLINK, LED_PRIORITY_IMPORTANT);
this->mBlinkRed = new LEDStatus(RGB_COLOR_RED, LED_PATTERN_BLINK, LED_PRIORITY_IMPORTANT);
this->mBlinkGreen = new LEDStatus(RGB_COLOR_GREEN, LED_PATTERN_BLINK, LED_PRIORITY_IMPORTANT);
this->mSolidGreen = new LEDStatus(RGB_COLOR_GREEN, LED_PATTERN_SOLID, LED_PRIORITY_IMPORTANT);
this->mRedPin = redPin;
this->mGreenPin = greenPin;
this->mBluePin = bluePin;
pinMode(mRedPin, OUTPUT);
pinMode(mGreenPin, OUTPUT);
pinMode(mBluePin, OUTPUT);
RGB.onChange(ledHandler);
Log.info("LED: Initialized...");
}
void LED::blinkRed() {
Log.trace("Blinking red...");
if (!mBlinkRed->isActive()) mBlinkRed->setActive(true);
}
void LED::idleColor() {
Log.trace("Going back to system default...");
if (mBlinkRed->isActive()) mBlinkRed->setActive(false);
}
void LED::ledHandler(uint8_t r, uint8_t g, uint8_t b) {
Log.trace("Mirroring to external LED...");
analogWrite(mRedPin, r);
analogWrite(mGreenPin, g);
analogWrite(mBluePin, b);
}
. When debugging I intentionally left in all print statements in an attempt to see where the Boron device would freeze on, I guess its counter intuitive that removing a print statement would actually fix the issue. Hence I never found the problem.