I’ve got got around 35 Particle Electrons in the field and around 8 of them are stuck trying to connect back to Particle Cloud with the flashing green LED. What I’m finding is that pressing the reset button alone is not enough to resolve the issue and I have to remove the battery and replace it before it connects again.
My approach in the firmware is:
SYSTEM_MODE(MANUAL);
sync: // Every 6 hours
Cellular.on();
if (Particle.connected()) {
… publish data…
}
The Electrons are programmed to deep sleep with some retained variables. They wake from the WKP pin or from a wake timer every 6 hours to check in with Particle Cloud. They are reside in a Particle product so share the same firmware, they all use the same SIM carrier with a custom APN and for the last 2 weeks, they have been in the same location so it’s interesting that only some are affected.
Would be great if anyone knows a way to resolve this without physical access or can suggest changes to the firmware to prevent it happening in future as they have been installed into a trial product and getting access is difficult and will impact a product trial we’re doing next week.
Let me know if you need any additional information.
What system firmware are you running on these devices?
There was at some point an issue with the radio connections when code did switch off the radio module while it was still in the middle of a connection or communication attempt which caused it to not switch off correctly causing reconnection issues like the sort you describe.
AFAIK, there should be a fix in 0.7.0-rc.1 for that.
Device part of a product should auto-update their system version, though it will use up quite a bit of data. If at all possible, I’d recommend trying it on a local test device before deploying it to the fleet.
We really need a way to double check that the modem is OFF or ON Until then, sparingly try to access the cellular hal directly which does offer a return value.
#include "cellular_hal.h"
cellular_result_t result = -1;
result = cellular_off(NULL);
if (result) Serial.println("ERROR!\r\nFailure powering off the modem!");
else Serial.println("OFF!");
result = cellular_on(NULL);
if (result) Serial.println("ERROR!\r\nFailure powering on the modem!");
else Serial.println("ON!");
Also if that does not fix your issue, try a modem reset. This is also only something you should only escalate to, and not use every boot.
// call this as a last resort
void resetModem(void)
{
pinMode(RESET_UC, OUTPUT);
digitalWrite(RESET_UC, LOW);
delay(100);
digitalWrite(RESET_UC, HIGH);
}
While I would love for you to try the pre-release 0.7.0-rc.2 (out now), I would not recommend it for your trial customer testing. This is something you should only test in-house, or have the ability to go on-site and downgrade if necessary. The fix @ScruffR mentioned should really only be helpful to you if you were running multi-threaded mode, and it doesn’t sound like you are.
Also, it might be worth checking to see if you hit any data limits on your SIM cards.
Thanks for the information. I am in fact running multi-threaded. The reason for this is that I need to ensure that even when I’m turning on the modem and checking for a connection to particle cloud, the device isn’t blocked. After reading your message, I did a quick test to turn off multi-threading and it does indeed block when doing a connection which means the device can become unresponsive for a little while which is what I want to avoid.
Is there another way for me to approach the connecting every few hours but not blocking code? I reset 34 devices on Friday, delivered them to the field and confirmed they were all working, now after the weekend has passed, I’ve now got 4 devices that appear to be playing up and not checking in after the 6 hour interval. Some don’t check in at all, others come online in the console, but don’t publish then go offline.
I’m thinking that multi-threading could be causing my issues. My SIM cards have a 50MB monthly limit and I’m no where near that at the moment.
This problem still exists in 0.7.0. 1 times in 20 the DEEP SLEEP or SOFT SHUTDOWN does not results in a powered off modem. This makes particle unusable in long term remote deployments presently, until a robust solution is found.
Excuse the quick crude code but this is my attempt at creating a failsafe route to shutting the modem down, but it still results in random 1000-4000ua current states from time to time.
cellularOff is the main function.
bool cellularOff(){
delay(10000);
cellular_result_t result = -1;
//First lets make sure cellular is on and happy
if (!isCellularOn()) {
bool modemstate = 0; //assume ff
long modem_startup_startsecs = Time.now();
while(modemstate == 0 && (Time.now() - modem_startup_startsecs) < 30) {
result = cellular_on(NULL);
if (result) {
if (debugmode) Serial.println("ERROR!\r\nFailure powering on the modem!");
} else {
if (debugmode) Serial.println("Modem ON!");
modemstate = 1;
}
watchdog_pat();
}
}
// we can now assume cellular is on, if so lets turn it off
if (isCellularOn()) {
bool modemstate = 1; //assume on
long modem_shutdown_startsecs = Time.now();
while(modemstate == 1 && (Time.now() - modem_shutdown_startsecs) < 300) {
if (cellularOffChild() == true) {
modemstate = 0;
} else {
if (debugmode) Serial.printlnf("Modem power off command failed, trying again..");
}
delay(1000);
watchdog_pat();
}
if (modemstate == 1) {
if (debugmode) Serial.printlnf("Modem power off complete failure in 5 mins of trying");
cellularOn = true;
return false;
} else {
cellularOn = false;
}
} else {
//modem is still not on, what do we do?
if (debugmode) Serial.printlnf("Modem not responding to on command");
hardResetModem();
System.reset();
}
}
bool cellularOffChild(){
bool modemstate = 1; //assume on
long modem_shutdown_startsecs = Time.now();
while(modemstate == 1 && (Time.now() - modem_shutdown_startsecs) < 30) {
cellular_result_t result = -1;
result = cellular_off(NULL);
if (result) {
if (debugmode) Serial.printlnf("ERROR!\r\nFailure powering off the modem!");
} else {
if (debugmode) Serial.printlnf("Modem is OFF!");
modemstate = 0;
return true;
}
watchdog_pat();
delay(1000);
}
if (modemstate == 1) {
if (debugmode) Serial.printlnf("Modem power off timeout reached - Hard reset modem and try again");
hardResetModem();
return false;
}
}
bool isCellularOn() {
int commandresult = Cellular.command(1000, "AT\r");
if (commandresult == RESP_OK) {
if(debugmode) Serial.printlnf("AT ok");
return true;
} else {
if(debugmode) Serial.println("AT fail");
return false;
}
}
@BDub Another failure mode seems to be that occasionally the modem turns off but something stops the sleep command from then sleeping the MCU, and it goes into a limbo state pulling 40ma (slow pulsing green led and code lockup). I have to rely on external watchdog timer to reset it, at which point it enters proper sleep.
Until then, sparingly try to access the cellular hal directly which does offer a return value.