Electron: Panic (Red SOS) on Particle.connect

Troubleshooting
1

Hi,
I wrote an small application to gather sensor data and send it directly to my server, without connecting to the particle cloud (if not specified by my server).
Unfortunately since a few months I get various "SOS" errors,.. Until now it was always pretty disturbing but now it is getting permanent and I don't have any ideas how to find the root cause of this error,.

  • executing Particle.connect(); in SYSTEM_MODE(SEMI_AUTOMATIC);
  • calling callular.connect when not execute cellular.on before
  • ....

Example:

Serial.print("Turn on cellular device,..");
delay(1000);
Cellular.on();
Serial.println("done");
delay(1000);

Serial.print("Setting credentials,..");
delay(1000);
cellular_credentials_set("mass.at", "", "", NULL);
Serial.println("done");
delay(1000);

Serial output:

Serial.print("Connecting,..");
delay(1000);
Cellular.connect();
Serial.println("done");
delay(1000);

Turn on cellular device,..done
Setting credentials,..done
Connecting,..

==> SOS

I tried that sketch on 2 different electrons, 2 different SIM cards, 0.6.2 on the first, 0.7.0-rc.3 on the second Electron. I compiled the code in the cloud using CLI and flashed thru USB (--serial) and even in recovery mode!

That said, the electron, the SIM and the credentials must be OK (at least somehow). There should be still plenty of heap memory free (approx 100k) just before entering cellular.connect.

And idea how I can find the cause of this error?

2

ok, I disabled 2 externa libs (smsNotify and cellLocate) and it seems to work for now,… both talk to the modem directly and may cause unexpected states,… I’ll keep you posted!

3

OK, I still got a lot of SOS-errors, but I found out that,…
1.) one error was caused by an overflow (?) when printing Serial.printf("integer: " + integer); instead of Serial.printf(“integer: %d”, integer)
2.) some more errors still happen when calling Particle.connect() in UDP mode.

When changing mode to TCP (and recreating keys) everything looks OK so far,…

Anyone ever had a similar issue or can give me a hint how to debug this issue??

regards

1 Like
4

Let me ping someone that might be able to help, @rickkas7 are you able to assist?

Kyle

5

Hi,

sorry for not responding earlier. Unfortunately it was not because the issue(s) disappeared but because I had many more issues getting electron REAL stable. Somehow they work, but I cannot leave them alone some miles away,…

What is working right now:

  • Connecting to network with 3rd party SIM (emnify)
  • Connecting to Particle Cloud (most of the time)

What does not work:

  • Firmware upgrade thru the cloud (almost 99% fail)
  • Stable particle.connect - still I get some occasional Red SOS

The issue is not limited to one device and not limited to one SIM provider. Right now I have 3 devices with emnify, 2 more with an local provider, one more with another one.

These are the startup settings I use:

STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));
STARTUP(System.enableFeature(FEATURE_RESET_INFO));
STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));
STARTUP(System.enableFeature(SEMI_AUTOMATIC));

In the void loop() I shut down the system completely to save power, except if my own server tells not to do, eg to connect to particle cloud.

For that I had also a Particle.process() in the loop. The loop itself was slowed down using delay(200);
The ApplicationWatchdog is right now set to 120s, as sometimes the modem takes very long to connect. If a reset occurs, a counter is beeing increased by one, if this counter is odd during loop, the device resets the modem and goes into deep_sleep for 10 seconds.

The rest is just a arduino sketch that runs nice. Usually I have around 100 kbytes free.

6

Just to complete previous attempts to resolve the issue:

I changed the cloud communication protocol from UDP to TCP (and back). Didn’t really change anything, suffer still a lot of various issues preventing the stuff to become a commercial product.

7

Hi @joky

I would try to debug your issues by running known good firmware such as Tinker for Electron, the app that is installed at the factory. Since you have third-party sim cards, you will need to compile Tinker yourself adding the correct startup commands for your sims.

If Tinker has connection issues you need to look at your provider settings. On the other hand, if that works OK, you need to look at your code.

8

is there any possibility to output any debugging-information to the (USB) Serial port?
Usually the Electron hang or SOS immediately after calling cellular.connect or particle.connect,…

In some cases the electron even hang when reset and booted into rescue mode,…

9

OK, it is possible to send some debugging information to console:

SerialLogHandler logHandler(LOG_LEVEL_ALL);

It seems, that there are several issues with the SIM card not completely detected (couldn’t get ICCID) or some errors while connecting to the mobile network,…

That said, I have still no idea how to recover from that state. I tried to reset anything, if the connection fails several times:

// Software-Reset
Cellular.command(500, "AT+CFUN=16\r\n");
delay(2000);

// Hardware-Reset
Serial.println("Performing cellular Hardware-Reset");
pinMode(RESET_UC, OUTPUT);
digitalWrite(RESET_UC, LOW);
delay(200);
digitalWrite(RESET_UC, HIGH);
Serial.println("netReset() done, deep-sleep for some seconds");

delay(2000);
System.sleep(SLEEP_MODE_DEEP, 30);
1 Like
10

@joky, have you checked the physical SIM connections on the Electron? You may want to gently bend up the contact fingers on the SIM socket.

11

all done already,… no success.
Could it be that mobile network operators has a connection rate limit?

 4.851 AT send      10 "AT+CREG?\r\n"
 4.892 AT read  +   14 "\r\n+CREG: 0,0\r\n"
 4.992 AT send      11 "AT+CREG=2\r\n"
 5.032 AT send      10 "AT+CREG?\r\n"
 5.072 AT read  +   14 "\r\n+CREG: 2,0\r\n"
20.132 AT send      10 "AT+CREG?\r\n"
20.183 AT read  +   14 "\r\n+CREG: 2,0\r\n"
35.243 AT send      10 "AT+CREG?\r\n"
35.284 AT read  +   14 "\r\n+CREG: 2,0\r\n"
50.344 AT send      10 "AT+CREG?\r\n"
50.344 AT read  +   12 "\r\n+CREG: 0\r\n"
50.425 AT read  +   14 "\r\n+CREG: 2,0\r\n"
16.221 AT send      10 "AT+CREG?\r\n"
16.262 AT read  +   14 "\r\n+CREG: 0,0\r\n"
16.362 AT send      11 "AT+CREG=2\r\n"
16.402 AT send      10 "AT+CREG?\r\n"
16.442 AT read  +   14 "\r\n+CREG: 2,0\r\n"
31.502 AT send      10 "AT+CREG?\r\n"
31.542 AT read  +   14 "\r\n+CREG: 2,0\r\n"
16.200 AT send      10 "AT+CREG?\r\n"
16.241 AT read  +   14 "\r\n+CREG: 2,0\r\n"
16.341 AT send      11 "AT+CREG=2\r\n"
16.381 AT send      10 "AT+CREG?\r\n"
16.421 AT read  +   14 "\r\n+CREG: 2,0\r\n"
31.481 AT send      10 "AT+CREG?\r\n"
31.522 AT read  +   14 "\r\n+CREG: 2,0\r\n"
46.582 AT send      10 "AT+CREG?\r\n"
46.582 AT read  +   12 "\r\n+CREG: 3\r\n"
46.603 AT read  +   12 "\r\n+CREG: 0\r\n"
46.623 AT read  +   14 "\r\n+CREG: 2,0\r\n"
12

ok, I got this specific error better under control by limiting the timespan between two connects to 3 minutes. But I cannot yet confirm it completely.

Furthermore, an old “friend” appeared again. I get occasional SOS - even when loading the electron in rescue mode. I also found another issue, that once connected to the cloud, breathing cyan stopps within seconds without any possibility to recover by ApplicationWatchdog,…

Is it rocket-science what I’m doin?

  • triggering a sensor delivering data over serial/TTL
  • querying one onewire sensor
  • submitting data over HTTP
  • keeping some values in EEPROM and/or retained RAM
  • checking values every 3 minutres, goin online once every half hour (or when value changed dramatically)

Is there anyone, who has knowledge of particle electron and can do code review & bug hunting for $$$?