MAJOR BUG in 1.0.1-rc.1 and 1.0.1

electron
Tags: #<Tag:0x00007f1ca13d8f28>

#1

There is a MAJOR bug in the 1.0.1-rc.1 and 1.0.1 firmware. I have been tearing my hair out trying to figure out why the Electron will NOT reliably reconnect to the network after using ANY of the System.sleep() commands. I have tested on every version using the code below since 0.5.4 and it doesn’t show up until 1.0.1-rc.1 so I tried it on 1.0.1 and it does the same thing. I have tried multiple 3G Electrons in the EXACT same location and get the same issue.

Any version of code before 1.0.1.-rc.1:

  1. Let the Electron do something (in this case just delay and serial print then do a System.sleep().
  2. Electron does it’s thing, and goes to sleep.
  3. 30 seconds later it wakes up and connects to the network within a few seconds. Just like it’s supposed to.

Version 1.0.1-rc.1 or 1.0.1:

  1. Let the Electron do something (in this case just delay and serial print then do a System.sleep().
  2. Electron does it’s thing, and goes to sleep.
  3. 30 seconds later it wakes up and flashes green and never connects back to the network. If I power cycle it it connects within 1-2 minutes. It will sometimes connect back to the network after several minutes, but that is only about 10% of the time. I have let this run all night and only had 3 successful connections using this firmware or the 1.0.1.

Hopefully someone can figure out what’s happening.

Thanks!

void setup() {

  Serial.begin(9600);
}

void loop() {
    delay(10000);
  Serial.println("--------------");
  
    Serial.println("Going to sleep");
    delay(10000);
    
System.sleep(SLEEP_MODE_DEEP, 30);

}

#2

I don’t know if you found a bug, but you should replace both the delay(10000) with something like:
for (uint32_t ms = millis(); millis() - ms < 10000; Particle.process());
See if that helps :grinning:


#3

The delay() is just so I have time to do a particle serial monitor on the computer to time that the System.sleep() happened. delay() is a valid (though blocking) function. The issue is AFTER the sleep when the Electron wakes up. It will not run the loop until it is connected to the cloud (as it should in this mode) so the delays are just for testing.


#4

@Rftop I tried your code for both 1.0.0 and 1.0.1-rc.1 and the same issues occurred exactly the same. I rolled back to 1.0.0 and the non connection issues went away again.

Thanks for the snippet though. I will begin using it instead of delay, just for the Particle.process() addition. :wink:


#5

one thing i discovered a while back is when my electron does the extended green blinks if i tap reset a couple times in quick succession it usually reconnects. not always but about 8 out of 10 times. it’s of no use if your electron is remote but thought i’d mention it.


#6

Can you file that as a GitHub issue at github.com/particle-iot/device-os/issues ?


#7

Just a couple of questions, did the Electron cloud connect before you put it into deep sleep and are you using SYSTEM_THREAD(ENABLED); ?

I defer to @scruffr 's greater knowledge that this appears to be a bug.

IMO - you should check for Particle.connected() in setup before starting the loop and in particular before you call System.sleep(SLEEP_MODE_DEEP, 30); so that the device is in a know stable state and not in the middle of connecting. Also, the typical approach for the wait for serial setup is while(!Serial.available()) Particle.process(); immediately after Serial.begin();


#8

I have been trying to debug a simmiler problem with an electron flashing blue (listening) after waking up. I think this is related to 1.0.1 but haven’t rolled back yet to confirm.
Unfortunately it sounds like yours is a different issue but if there is one bug there could be more?


#9

Thanks for confirming my suspicions too about 1.0.1. It must have a lot of bugs, not only the one you raise. Connectivity to the network is very very unstable. We’ve been running a piece of firmware without issues in Electron for several months in 7.0 but it became too unstable once we tried it on 1.0.0 and 1.0.1.

For instance, we have a finite state machine to manage manually the connection to cloud through several states. It counts the number of cloud disconnects and after several of those (6 or more) makes a full_modem_reset (deep_sleep for 20 secs). We see the devices performing these resets too often now. Another test we made was putting the Connection_check from the electron_sample library. The effect is even worst as the connection_check resets the Electron after sensing no cloud events. Reverting to 7.0. R1.0.0 seems a bit more stable than 1.0.1 but I have not the chance to detect hard evidence to substantiate it.


#10

Is there any update on this. I too face similar issue on 1.0.0 and 1.0.1. Connectivity to cloud is very unstable.

I have to spend hours to update a Software on Electron with firmware 1.0.0 or 1.0.1.


#11

Hey folks – wanted to let folks know that the Particle team is aware of this thread and the reported issue. Thank you @spuckett for providing a code snippet that we can use to reproduce the issue.

For @fenriquez and @drpjshah – would welcome you to create threads to diagnose and debug issues that you’re seeing, if they are different from the one cited above.


#12

What model of Electron are you using (G350, U260, U270, etc.)?

And could you add this to your code?

SerialLogHandler logHandler(LOG_LEVEL_ALL);

Note: Make sure your device has been flashed with v1.0.1 firmware locally from the Particle CLI with particle update or via all 3 system binaries particle flash --usb system-part1-1.0.1-electron.bin from the github release if you prefer. This is required to ensure the binaries have been built using DEBUG_BUILD=y to get the AT commands in the logs. If your device updated via Safe Mode Healer because you flashed a v1.0.1 app, the resulting system firmware will not contain DEBUG_BUILD=y.

I tested the code in the first post, with that line added, on an Electron U260 (3G Americas), running Device OS 1.0.1, monitoring the USB serial port using:

particle serial monitor --follow

My Electron was able to immediately reconnect with no difficulties. I let it repeat 10 times and it only took less than 10 seconds to reconnect each time.

The TRACE log SerialLogHandler logHandler(LOG_LEVEL_TRACE); would be helpful in determining what step of the reconnection process is failing for you and help isolate the problem.

The ALL log SerialLogHandler logHandler(LOG_LEVEL_ALL); will provide full AT command details straight away though, so that is preferred.


#13

We are using Electron U260. We have updated firmware on couple of electrons in the field, and most of them (not all) gives connectivity issue when OTA update the software.

Today, I was facing issue in OTA update in the field. It was giving hard time in updating. Previously also we had to spend 2 hours just for one OTA update with version 1.0.0/1.0.1. Then I downgraded the firmware to 0.6.4. After this no issue is reported for connectivity.

So at present we will be downgrading the firmware version until the issue is resolved. As Will mentioned, we will wait until we hear back from particle team.


#14

I’m also not able to reproduce this issue on E402 (LTE) and U260 (3G) with the code provided (no alterations) in the first post running v1.0.1 Device OS. They happily sleep and wake even with low signal strength. My E402 is averaging RSSI of -88dB (3 bars) and U260 is averaging RSSI of -99dB (1 bar). I have a battery on my U260 and USB plugged in, E402 has no battery and only USB. I would test G350 (2G), but 2G coverage in my area is spotty so I’d have to go driving to test it :slight_smile:

If all of you having issue would provide more information that would be super helpful. In particular I would want to know:

  • Can you reproduce the issue with the above code example?
  • What Particle Device are you using?
  • What antenna are you using?
  • What SIM card are you using?
  • What Device OS version are you having trouble with?
  • What RSSI / signal strength average are you seeing?
  • If you can reproduce the issue, can you instrument the code with logging as recommended by Rick above (with full AT command logging) and PM me the log files? Please try to send a full example of booting, connecting, sleeping, waking, and not connecting.
  • :smile: I know it might seem like a lot to ask for all of this info, but I take it you all care deeply about the reliability of your Particle devices to post here about the issues you are having… and wish you didn’t have them! Reliability of our products are very important to us as well! Please help us help you :heart:

You can use particle serial monitor --follow to automatically reconnect to the device when it wakes. You can also use --port COMx or --port /dev/cu.usbmodem14xxx if you want to test multiple devices at once.

Have you started a separate thread for your issue, or contacted support for that issue yet? Can you reproduce connectivity issues with the above firmware?

I’m curious how you are counting the disconnects. There is a new method to do this now with 1.0.0 that wasn’t available in 0.7.0 and I wonder if that could be useful to you. Can you reproduce connectivity issues with the above firmware?


#15

Hi @BDub. I am currently running everything in 1.0.0 and 0.7.0 because of the initial unpleasant experiences with 1.0.1 so cannot give more updates. In the lab everything works ok, but when in the field everything messes up. As per your questions, we are running everything standard from Particle Electron (America): standard antenna, standard SIM.

The most problematic device has an RSSI of -95 (considered “fair”). No much we can do here as the devices are connecting industrial machines which are fixed to one place (i.e we cannot move the Electron to a place with better reception)

As per your question on “counting the cloud disconnects”, we do that by having a variable counting the number of times we sense Particle.connected returning FALSE. We are running in manual mode and have a simple code for this:

if (millis() - last_process >= espera_process){ //"espera_process = 3 seconds" 
      last_process = millis();
      if (Particle.connected()) Particle.process();
      else {
         disconnect_count++;
         Particle.disconnect();
         Particle.process();
          for(uint32_t ms = millis(); millis() - ms < 500; Particle.process()); //wait a bit for Particle.disconnect to perform before trying to connect again
         Particle.connect();
         waitFor(Particle.connected, 30000);
       }
    }   

I have a ticket open with Particle support. It seems that part of our issue is related to the carrier board we use for the electron. We’re solving that now before doing more tests on the software part.

Thanks again!


#16

Hi all! Is anyone here able to provide device logs so we can see what is going on with your device? Thanks!