Boron LTE Fails to Connect

If I flash the pre-built cloud debug tool at this point, the device goes into an error state, flashing red. I’m unable to get it in listen mode in this condition so I can no longer run the serial inspect. In order to recover, I have to flash the system part again for 1.1.0-rc.2. I assume the pre-built tool overwrites the system code with something that is incompatible with the bootloader. The pre-built tool is significantly larger than when I build it.

After recovering and flashing a version of the cloud debug tool built for 1.1.0-rc.2, I see the same log output.

deviceID=<redacted>
manufacturer=u-blox
model=SARA-R410M-02B
firmware version=L0.0.00.00.05.06 [Feb 03 2018 13:00:41]
ordering code=SARA-R410M-02B
IMEI=<redacted>
IMSI=u-blox
ICCID=<redacted>
0000021722 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000138924 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000138924 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000150579 [hal] ERROR: Failed to power off modem
0000150579 [hal] ERROR: Failed to power off modem
0000171682 [hal] ERROR: No response from NCP
0000171682 [hal] ERROR: No response from NCP
0000305800 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000305800 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000317405 [hal] ERROR: Failed to power off modem
0000317405 [hal] ERROR: Failed to power off modem

With this configuration, I have the following additional issues:

  • The device can be put in listening mode but serial inspect usually times out.
  • The device will not go into safe mode but will go into DFU mode. When I attempt to go into safe mode the device just resets and begins running user code.

To be honest I’m not sure why, but maybe that is compiled as monolithic and not compatible with the newer bootloader (which would be unusual). Maybe @rickkas7 can straighten us out.

When you flash modular system 1.1.0-rc.2 and tinker, I know you can’t see the logs but will the device connect at all?

This may likely be due to debugging interfering with the serial commands. This bug will be fixed in a future Device OS update.

Hmm, that is strange. When serial inspect doesn’t timeout, what does it look like in this mode? You might have to disable logging and recompile with that firmware to get serial inspect to work.

No. This device has not connected under any circumstances since this began.

Are you asking if it will go into safe mode? I have not seen anything that allows it to go into safe mode.

I’d say we’ve done our dew diligence here, thank you for attempting some soft fixes. I’ll let you and @mstanley take it from here on that replacement device.

Thanks for checking into this @BDub :slight_smile:

And thank you Tom for taking the time to go through these steps.

If you’ve not already, please go ahead and file a ticket for a replacement, referencing this topic. We’ll be sending you a return label as well, as we’d like to get this unit into the hands of engineering to dig into this further and see what went wrong here.

Thanks everyone!

EDIT: As an added note, I’ll be updating the topic title to specify the Boron specifically, as I cannot say for certain this issue exists in our other R410M-02B devices. This will be helpful for others searching on this issue.

@mstanley, I recently became aware of an issue with the R410M-02B that can render the device unusable. There is a FW update available from u-blox. Has Particle looked at the relevance of this issue to the Boron devices?

We are aware of the issue described here. I’m uncertain of its relevance to the issue you are experiencing, but is something engineering will certainly investigate. Particle already has existing communications open with ublox in regards to known bugs and discussions on means of handling firmware update for our in-use ublox modules.

Hi @mstanley,

I’ve got three devices with this problem. They blink green trying to connect to the cloud forever. They were running 1.1.0-rc.2 when encountering the issue. Updating them to 1.2.1 does not fix the issue.

clouddebug: press letter corresponding to the command
a - enter APN for 3rd-party SIM card
k - set keep-alive value
c - show carriers at this location
p - select MNO profile
t - run normal tests (occurs automatically after 10 seconds)
or tap the MODE button once to show carriers
starting tests...
turning cellular on...
0000015009 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
0000036158 [hal] ERROR: No response from NCP
deviceID=e00fce68f378b52473295f7d
manufacturer=
model=
firmware version=
ordering code=
IMEI=
IMSI=
ICCID=
0000117160 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000117164 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117164 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117168 [hal] TRACE: PPP netif -> 8
0000117169 [net.ifapi] INFO: Netif pp3 state UP
0000117169 [net.ifapi] INFO: Netif pp3 state UP
0000117171 [hal] TRACE: Powering modem on
0000117172 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117172 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117322 [hal] TRACE: Modem powered on
0000117323 [hal] TRACE: Setting UART voltage translator state 1
0000118325 [ncp.at] TRACE: > AT
0000119326 [ncp.at] TRACE: > AT
0000120327 [ncp.at] TRACE: > AT
0000121327 [ncp.at] TRACE: > AT
0000122328 [ncp.at] TRACE: > AT
0000123329 [ncp.at] TRACE: > AT
0000124330 [ncp.at] TRACE: > AT
0000125330 [ncp.at] TRACE: > AT
0000126330 [ncp.at] TRACE: > AT
0000127331 [ncp.at] TRACE: > AT
0000128332 [ncp.at] TRACE: > AT
0000129333 [ncp.at] TRACE: > AT
0000130334 [ncp.at] TRACE: > AT
0000131335 [ncp.at] TRACE: > AT
0000132335 [ncp.at] TRACE: > AT
0000133336 [ncp.at] TRACE: > AT
0000134337 [ncp.at] TRACE: > AT
0000135338 [ncp.at] TRACE: > AT
0000136338 [ncp.at] TRACE: > AT
0000137339 [ncp.at] TRACE: > AT
0000138340 [hal] ERROR: No response from NCP
0000138340 [hal] ERROR: No response from NCP
0000138342 [hal] TRACE: Setting UART voltage translator state 0
0000138344 [hal] TRACE: Hard resetting the modem
0000149345 [net.pppncp] TRACE: Failed to initialize ublox NCP client: -210
0000149347 [hal] TRACE: Setting UART voltage translator state 0
0000149349 [hal] TRACE: Modem already off
0000149450 [hal] TRACE: Powering modem on
0000149600 [hal] TRACE: Modem powered on
0000149601 [hal] TRACE: Setting UART voltage translator state 1
0000150603 [ncp.at] TRACE: > AT
0000151604 [ncp.at] TRACE: > AT
0000152605 [ncp.at] TRACE: > AT
0000153606 [ncp.at] TRACE: > AT
0000154607 [ncp.at] TRACE: > AT
0000155608 [ncp.at] TRACE: > AT
0000156609 [ncp.at] TRACE: > AT
0000157610 [ncp.at] TRACE: > AT

Serial Inspect output

Platform: 13 - Boron
Modules
  Bootloader module #0 - version 311, main location, 49152 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  System module #1 - version 1213, main location, 671744 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      Bootloader module #0 - version 311
  User module #1 - version 5, main location, 131072 bytes max size
    UUID: 362808E0DB24B4ED309227C9121675856A32E73D5D16D2C7922C49C9AEF353C2
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #1 - version 326

These three devices are in the field with 17 others for a R&D project we are running with Borons. All three experienced this issue after running completely out of battery. They were not powered by a LiPo battery, but were powered by a 12V SLA battery run through a regulator.

All three of these devices had this problem after running the 12V SLA completely out of power (down to ~6V). I think this may have something to do with that? Maybe a modem brownout?

Hi Heath,

Those logs are fantastic, thanks so much for providing those! I have ensured that engineering will be taking a look at these.

We are still investigating this issue on our end. We have a device OS release coming sometime next week to mitigate some of the Boron LTE fail cases. As that comes out, we will be posting a new community topic for the release and instructions on how to capture additional debug logging so that we can continue to investigate connectivity concerns.

Not sure if this helps, but our carrier board can sit either a Boron or Electron and we have about 20 of each in the field. None of the Electrons have failed this way (or any other way).

Hi Heath.

Certainly insightful. At the moment, our upcoming release candidate for v1.3.1 addresses a Boron specific issue. If you continue to notice this issue post v1.3.1-rc, it is helpful to know that this issue is still exclusively impacting Borons.

Hi folks, we have released device OS v1.3.1-rc.1 to tackle some of the issues causing difficulty to all the users here. Please refer to Status Update for cellular connectivity issues | 8/21/19 for instructions on how to provide feedback and debug logs. Thanks!

@hwestbrook @trpropst

@hwestbrook - Did OSv1.3.1-rc.1 fix your issue? I had a device on my test bench that has very similar looking cloud debug logs “Trace->AT” that started up flashing green. The RC didn’t fix it.

I discovered this week that I had 6 devices in the field also permanently flashing green. I’ve only taken one out of the weatherproof case (I’ve had issues with case screws stripping out, so I try to avoid it), and it also had a similar looking clouddebug log. THis is an old thread I understand. I’m delayed, because my devices are mostly only used in summer, I assumed the devices were offline because the operator had shut off the main power, but I was surprised to see they were all flashing green even after power cycles. Out of 15 devices, 6 are experiencing this behavior.

@jonpasc,

It did recover a few of them, and didn’t others. I didn’t have time to dive much deeper into the issue, and Particle replaced a few of them for me.

The main takeaway for these LTE boron’s without the updated ublox firmware is to avoid using them in situations where the power is not guaranteed.

Did the 1.3.1-rc1 and 1.4.4 update that ublox modem firmware to your knowledge? I’m trying to get to a build where I can set and forget. I had one device fail on solar and 6 of a different type of setup on a power source that’s either on or off for months at a time. The cell reception is spotty.

To my knowledge, Particle does not have the ability to update the ublox firmware through device-os. If you search elsewhere in the forum you’ll find people have made some custom PCBs for performing this.

Thanks for pointing that out. I found Ublox modem firmware update with very specific directions. Fortunately, I’m in an off season until March, so I can sit on this a while longer and see if Particle finds a solution. I’ve been 0 for 7 in bringing devices that started flashing green to life with an upgrade to 1.4.4 using “particle update” in the CLI. Unfortunately, i’m not sure the root cause is addressed and it’s just a matter of time before my other deices fall into the same state.

@jonpasc
Were you able to bring them back online and install the latest firmware?

Reason is that I have shipped some units out of the country and my customer mentioned that the two LTE borons have been blinking green (looking for cellular connection) since December, in the console I have reported that last connection happened on Dec 1st.
But when I check the information these sensors send to my server, they show as if they were sending info last week, but now that I am trying to troubleshoot them -it is a pain in the neck- because they just stay blinking green, even after resetting them.

Both of these Borons are running under the firmware 1.2.1

I was not able to bring them back to life even after plugging them into a computer and running particle update which should be the best update method. However, it sounds like your’s are still sending data? Mine weren’t doing a thing. The best troubleshooting is to download and flash the cloud debug from rickkas7. Unfortunately, if it’s the ublox module, you have to flash the debugger over usb and read the output over serial to confirm that the ublox module is indeed the culprit. https://github.com/rickkas7/boron-clouddebug

I’m in the middle of debugging an issue where my code had some of the same debug output that I was reading over serial. I discovered that when my device got stuck in a disconnected state, just opening a serial connection over USB revived it. I was able to repeat this several times, letting the device sit in the error state for hours and seeing it connect immediately once I opened my serial console.

Just FYI in case you have trouble reproducing the problem when watching debug. I have not yet figure out why this was happening.