Status Update for cellular connectivity issues | 8/21/19

Hi everyone!

To those of you keeping an eye on our cellular connectivity issues as per our previous community post at Instructions for collecting and reporting cellular connectivity issues, many thanks to all who contributed logs and information of their devices having difficulties.

Particle’s engineering teams have all been hard at work assessing the issues our customers have been facing while building with Particle. Over this past month, our teams have been working on various fixes and improvements to help resolve some of these issues to help unblock our customers. All of these fixes have been bundled up and released as per our latest Device OS release, v1.3.1-rc.1. A comprehensive list of bug fixes can be found in the release here. Most notably is issue #1862 which addresses a memory leak found in Borons in poor connectivity areas that can cause a device to end up in a blinking green state until restarted.

While we have addressed a number of issues that were identified, we recognize that the nature of these issues are complex and have a number of root causes. As such, we would like to continue our feedback loop with customers who were experiencing issues by having them run tests with the newest Device OS release to both see if the identified issues are resolved as well as capturing logs for any potentially outstanding issues still present so that we may address them as timely as possible.

Instructions for capturing and submitting Logs

The most helpful information that you can provide to help us chase down intermittent connectivity issues are logs from the device from when the connectivity disruption occurs. This means that the device will need to be connected to a computer that can capture logs until the connectivity event occurs.

We’ll provide clear instructions for preparing your device to capture these logs, and what supplemental information we need in order to debug efficiently.

1. Written description

To start, we’d love to learn more about what happens to the device when the issue occurs. In your own words, what’s your impression of the following:

  • Have you been successful in getting your device connected to the Particle Cloud previously?

  • How often does the connectivity issue occur?

  • Does it occur on all your devices? If so, which ones?

  • What version of Device OS is your Particle device running before testing?

  • Are you still seeing the issue on v1.3.1-rc.1?

  • Are you using a Particle SIM or 3rd party SIM?

  • What other types of technologies is your device integrating with? For instance, is I2C used?

  • What does your environment look like?

    • Is the device antenna close to any metal or electrically noisy objects?

    • What is the signal strength in the area for your device? (not your cellphone)

    • Is the device stationary or mobile?

    • What power source are you using? (lipo battery, USB only, short USB cord, what type of USB hub?)

2. Device logs

The next step is to isolate as many variables as possible and capture logs from when the connectivity incident occurs. This includes:

  • Ensuring the device is running the most recent version of Device OS

  • Ensuring the device is running a standardized “cloud-debug” user application that captures logs that our team can use for debugging

Download everything

To start, we need to update your device to the latest version of Device OS, which is presently v1.3.1-rc.1 .

  • Download the correct set of binaries for your device. You can also find them below at:

  • Electron / E Series

  • Boron

  • The full list of binaries for all Particle devices is available at

https://github.com/particle-iot/device-os/releases/tag/v1.3.1-rc.1

  • Download the Cloud debug application which will help us collect standardized logs for your device.

  • If you are using the Electron or E Series , download the cloud debug firmware here

  • It will download with the filename firmware.bin

  • please rename to electron-clouddebug-v1.3.1-rc.1.bin for compatibility with following instructions.

  • If you are using the Boron , download the cloud debug firmware for Boron Device OS v1.3.1-rc.1 here .

Flash Device OS and clouddebug app

  • Use the Particle CLI upgrade your Device OS and flash the cloud-debug firmware to your device

  • Put your device into DFU mode (flashing yellow)

  • particle flash --usb <device>-system-part1@1.3.1-rc.1.bin

  • particle flash --usb <device>-clouddebug-v1.3.1-rc.1.bin

  • Put your device into listening mode (blinking blue)

  • particle flash --serial <device>-bootloader@1.3.1-rc.1.bin

  • Open up a serial terminal by issuing the following CLI command:

  • particle serial monitor --follow

You should see log output start to flow through your terminal that looks something like this:


neighbor 0 rat=GSM mcc=310, mnc=11094, lac=80592df ci=a56f band=GSM 850 bsic=18 arfcn=eb rxlev=37

neighbor 1 rat=GSM mcc=310, mnc=11094, lac=100 ci=a5f2 band=GSM 850 bsic=25 arfcn=b4 rxlev=22

15.443 AT send 20 "AT+UPING=\"8.8.8.8\"\r\n"

15.443 AT read + 14 "\r\n+CIEV: 2,2\r\n"

15.484 AT read OK 6 "\r\nOK\r\n"

ping addr 8.8.8.8=1

15.484 AT send 31 "AT+UDNSRN=0,\"device.spark.io\"\r\n"

17.204 AT read + 67 "\r\n+UUPING: 1,32,\"google-public-dns-a.google.com\",\"8.8.8.8\",55,812\r\n"

17.865 AT read + 67 "\r\n+UUPING: 2,32,\"google-public-dns-a.google.com\",\"8.8.8.8\",55,651\r\n"

17.936 AT read + 27 "\r\n+UDNSRN: \"52.91.48.237\"\r\n"

17.946 AT read OK 6 "\r\nOK\r\n"1

Capture logs

  • Be sure to capture logs for at least 10 minutes, including when the connectivity event occurs.

  • Copy the logs from your serial terminal into a .txt file (not .rtf, Microsoft Word, or Pages) using Notepad (Windows) or TextEdit (Mac)

  • If using Mac, you can pipe your log output directly to a file like so:

particle serial monitor --follow | tee -a log1.txt

3. Submit via support.particle.io

  • Submit the information above to our support team using the support portal at support.particle.io

  • Be sure to include both the 1) written information as well as the 2) connectivity logs.

Done!

Thank you for helping to improve the Particle ecosystem! If you have questions about these instructions, please feel free to post in the thread below.

4 Likes

@mstanley can you confirm what version of ublox firmware particle is testing the OS 1.3.1-RC1 with? L0.0.00.00.05.08 or still L0.0.00.00.05.06?

Hi @ric_hard. Particle currently does not update the ublox firmware as a part of its device OS releases. The current firmware that is on the module is the firmware that Device OS v1.3.1-rc.1 is expected to run with.

Internal testing is done by various engineering teams. The firmwares of the ublox modules will be the same as the one that came on the unit at the time of manufacturing unless the engineer has explicitly updated their own device’s ublox module. As such, each engineer that tests this will be testing against one of the various ublox firmware versions that came on their devices.

Is there a specific firmware version you are curious about?

@mstanley
Ublox firmware L0.0.00.00.05.06 (factory on Borons)

Ublox firmware L0.0.00.00.05.08 is the new firmware as there was an issue with 05.06 to do with the Quallcomm chip set. I know particle are making some jigs for flashing the modem firmware so my question was what you were testing with.

I have 40 Boron’s and during commissioning them I will be updating the ublox firmware.

This originally came about because of the constant drop outs it it was suspected that part if not all of the problem was due to the Quallcomm issue, hence why I was curious as to if OS 1.3.1-RC1 was tested for the new Ublox firmware.

Thank you @mstanley and Particle! I cannot wait to test and see if #1862 solves the repeatable issue in my case. It is not possible for me to collect the logs given the factors of my remote site, but hopefully I will have a successful result to share which will obviate the need for any such connection error logs.

1 Like

The most likely candidate to have explored the various firmware updates would be @BDub. If he’s available, he can provide insight on testing differences across ublox firmware versions and likely answer your questions better than I’m able.

@mstanley I want to let you know I am deploying tonight (2.5 hour trip) to remote site to effectuate this test of 1.3.1-rc1. I have not been currently using a Boron at this site due to these issues, but I am optimistic this will resolve what was happening (see eventual-infinite-green-flashing thread).

Once you fix this, I will trash my Pycom GPy and use Boron.

The reason I am optimistc the memory leak was the issue is because I can confirm this issue only happens where there are semi-regular occasional dropouts. I have had another Boron in the field since 7/14 and still connected no problem, only 3 or 4 total disconnects (stable signal). I have another Boron in the field for almost a month with not even any disconnects.

The fact that where perfect cell signal exists, it never happens, is a big indication the memory leak was it and 1.3.1-rc1 will fix it.

My test case is exactly, therefore, on-point to 1.3.1-rc1 (e.g., I never had the never-connect, boot-up-to-infinite-green-flash issue, which according to your statement above is the more elusive).

It will need to be at least two solid months of stable connection before I can truly declare victory (remember, my initial 0.9.0 Boron stayed connected at this site - the bad signal one - for a month-and-a-half in the winter before entering the infinite green light state, requiring 2.5 hour drive and back for power reset).

But given all the indications above, Occam’s Razor tells me the memory leak issue is the simplest and most beautiful explanation for what was going on.

THANK YOU again for your work, and I will be back with real world test result.

3 Likes

@ric_hard Since 1.1.1, 1.2.1 and 1.3.0-rc.1, Device OS supports operation with all u-blox firmware releases for the SARA_R410M_02B. Versions prior to those will not support 05.08,A02.04 u-blox firmware, and will have less bug fixes. Therefore it is recommended to test the latest versions of Device OS firmware and apply them to your devices.

1 Like