Electrons Lose Connectivity Intermittently

Troubleshooting
1

Hi All,

Wondering if anyone has had an issue like the one outlined below. For some background, we use Particle Electrons for an agriculture application. We’ve got 500 or so Electrons in the field for this use, spread between Washington, Oregon and California. For the most part we use Particle’s sim cards.

We’ve deployed 22 of our devices with Electrons to a farm outside the LA area. On this farm all devices have good service, 70-85 RSSI dB. Each device checks in every 15 minutes.

6 of the 22 devices go offline intermittently, sometimes for 30 minutes, sometimes for multiple hours. They are the same 6 devices most of the time. This behavior does not occur at any of our other 30 or so locations.

The only clue I have is the devices which experience this seem to be jumping onto a tower with really bad service (111 RSSI dB) . I can see this happening as sometimes the devices don’t go offline when they go onto this tower. Here is example data from two devices https://pastebin.com/BL6WPSJY and https://pastebin.com/qbW7B5rK.

My guess is that the following is happening:

  • Device is on tower with good service
  • For some reason device is moved to tower with bad service
  • Device can connect to this tower via modem, but service is bad enough that it cannot make a data connection
  • Device gets stuck on this tower for a few hours, then goes back to good tower
  • Process repeats

Has anyone else see an issue like this? What did you do to resolve? I’m thinking through how we might force our devices not to connect to a tower with bad service when there are other towers with better service.

2

Hi
We have had similar issues in Florida.
We have still not got to the end of what’s happening but we have found the the sequence we power up the device fixes the issues.
You will need to use both power supplies
Step one plug in the Lipo in the jst port and then power through the mini usb port. You must do it in this order.
I’m not sure if this will help but it worked 100% for us. Let me know how you get on

4

This would be a recommended practice as unfortunately, most micro USB is not enough to power everything on the Electron–namely, the cellular modem on the device. The LiPo battery can, however, and when the device is both powered by USB and the battery–the battery will sometimes kick in to provide a little extra juice when USB is not enough. When the power draw is low enough (such as the cellular modem being idle), the micro USB is typically enough to power the Electron and charge the battery a bit. Ideally, this way the battery can be stocked up for when it is needed to kick in and assist with extra power consumption when necessary.

Doing so in reverse could cause some hiccups such as connectivity issues, as you seem to have experienced.

5

It sounds like you might have done this already, but if you haven’t yet, I would recommend you try performing some additional debugging using rickkas7/electron-clouddebug to do some additional debugging in these sparser locations and share the logs here to see what may be going on.

I do believe you are correct based on what you provided, but these logs are usually fairly comprehensive and may show us a bit more depth in what might be going on here.

6

These devices are about 6 hours away from me, so it is not easy to do the cloud debug. I can say though, that I’ve used @rickkas7’s tools extensively to make sure our firmware reconnects quickly to the cloud when kicked off.

I can put some of the debugging logging into our firmware and have it log to the particle console. What would be valuable information to have? I’m thinking:

  • Operator whose tower our devices are connecting to when online and when offline
  • Cell signal strength when “offline”

Anything other data points?

Also, our other MVNO had much more data for us when troubleshooting this issue than I can seem to find from Particle’s backend. With the other MVNO, we could see current tower our device was connected to as well as logs regarding which carrier our device was connected to for the current billing period. Is this data not available to Particle for some reason? Or is it something you can access and then provide to us based on ICCIDs?

7

I’ll ping @rickkas7 to see if he has any insight on what info you might want to dig into.

It also may not hurt to experiment with third party SIMs in these areas. It’s possible that the towers you have coverage to may simply be spotty or have something that is intermittently blocking their connection or degrading the signal. If a third party carrier has better coverage, it may be able to handle or eliminate these variables.

8

@mstanley,

I’m curious what you think a 3rd party SIM might accomplish? As far as I know, Particle’s SIMs utilize two networks in the US: T-Mobile and AT&T. These are the only two networks that the Electron can go on in the US. Our other MVNO that we’ve used has the same setup, AT&T and T-Mobile.

It would be nice if we could pick a network to use in this situation. For example, it would be great if we could tell a SIM it was only allowed to be on AT&T, as this would eliminate one possible cause of this issue. Would it be possible for you to do this for us on a test basis for a few ICCID’s?

9

The value is if the third party SIM uses other carriers and had access to new towers looking over the areas, potentially. There’d likely not be as much value in using a third party SIM with one of our existing providers for the Particle SIM, but I can’t be certain. The MVNO we use does use T-mobile & ATT, but it isn’t to say that if you had an ATT SIM directly that it may not have tower agreements with other providers in the US that you may not otherwise get from our MVNO.

I am not the one who directly negotiates our agreements, so I could be off the mark here. Take the above with a grain of salt. :smile_cat:

I don’t believe that we are able to prioritize a particular carrier from our end. I took a look and didn’t see anything that would suggest that.

EDIT: It looks like Dave has proved me wrong below. Check out his info to get the full scoop.

10

I am pretty sure you can select the carrier that you want to use. Let me do some digging. IIRC it’s via AT commands.

11

I have this in my notes:

force a connection to AT&T (“mobile network code” = 310410):

cellular_on(NULL);
cellular_command(NULL, NULL, 100000, "AT+UPSDA=0,4\r\n");
cellular_command(NULL, NULL, 100000, "AT+COPS=2\r\n");
cellular_command(NULL, NULL, 1000, "AT+COPS=4,2,\"310410\"\r\n");
Particle.connect();

(310260 is tmobile)

12

What would be the right command to get the mobile network code from the modem? AT+UDOPN=0? Or AT+CGED=3?

I see there are quite a few different mobile network codes for AT&T in the US, per: http://www.mcc-mnc.com/. I’d like to log out which one my devices are using before forcing them onto one.

13

I’ve put firmware on one of the devices that issues a UCELLINFO command. It returns the tower ID, LAC, MCC and MNC.

So far it has returned information on four towers. Two towers are AT&T (3G & 2G) towers which are 20 miles away from the device and two towers are T-Mobile (2G) towers which are 28 miles away from the device. All in Bakersfield, CA. Location data is from the Google Maps API, which I understand is not 100% accurate.

I’m wondering if I’ve run into the following situation:

  • Our device has line of sight to these towers, so has a good RSSI to them. RSSI being received signal strength
  • The towers are far enough away that our device cannot reliably transmit to them
  • So, RSSI in this scenario is not a good indicator of strength

Is that possible? Has anyone else experienced an issue like this?

14

Hey @hwestbrook — what about Rick's CellularHelper library? This will give you a lot of the info you're looking for.

You mentioned:

For example, it would be great if we could tell a SIM it was only allowed to be on AT&T, as this would eliminate one possible cause of this issue.

There's a selectOperator() method that you can try to use to prefer AT&T in this case.

15

I’ve collected more data related to what towers our devices are and aren’t connecting to. Using google maps to determine where the towers are, I can see that they are connecting to towers between 12 and 30 miles away from the devices. The device with the stable connection connects to a AT&T 3G tower 12-15 miles away and the other two test devices with unstable connections bounce around between AT&T and T-Mobile towers and between 2G and 3G with towers 15-30 miles away.

What’s weird about all this is that all towers that I’ve recorded have similar RSSI levels, ~75 dBm when on 2G and ~85 dBm when on 3G, no matter the distance.

I’m wondering if the modem is getting confused by these similar RSSI levels and picking whatever tower has the lowest dBm reading, no matter if it is far or close, 2G or 3G. I can’t seem to find an explanation in the manual as to what algorithm the modem uses for picking a tower. In this scenario I’d like the modem to pick the tower based on more than just RSSI, as it seems unreliable.

I think forcing our device onto AT&T is the next step on this. I see in that cellular helper library that you can select operator, but can you also select 3G vs 2G? I’d like to force the devices onto 3G in order to overcome this potential issue with defaulting to lower RSSI dBm levels.

Overall, the troubling thing for me on all this is that it seems that I cannot trust the RSSI dBm levels as a test of whether our device will work in a location. In the past we’ve told our installers that if a device has 95dBm or lower our device will work reliably. Now we are going to have to tell them something like “if a device has 95dBm or lower and it works for a week without missing checkins, it is installed correctly”. It makes installation much more difficult.

16

For anyone that comes across a similar issue in the future here are my findings, with a lot of help from Particle support:

  • RSSI is usually, but not always, a good predictor of whether your cellular application will work
    • Anything below -75 dB is considered risky
  • Quality (aka BER) is also important, with a quality level of less than 10 being a very bad connection

So what’s root cause and how to fix?

  • We’ve been mistakenly relying on RSSI as our primary indicator for people installing our equipment
    • It worked OK, until it didn’t at this location
  • Our devices that are going offline in this location have very bad quality readings, < 10, even though they have OK RSSI levels of -75 to -85 dB
  • We will be physically moving devices around to get better signals

Other stuff

  • Locking onto a certain network or network technology (3G vs 2G) with cellular commands didn’t help
  • Turning off power saving mode with cellular commands didn’t help