Boron LTE Fails to Connect

Hi Heath,

Those logs are fantastic, thanks so much for providing those! I have ensured that engineering will be taking a look at these.

We are still investigating this issue on our end. We have a device OS release coming sometime next week to mitigate some of the Boron LTE fail cases. As that comes out, we will be posting a new community topic for the release and instructions on how to capture additional debug logging so that we can continue to investigate connectivity concerns.

Not sure if this helps, but our carrier board can sit either a Boron or Electron and we have about 20 of each in the field. None of the Electrons have failed this way (or any other way).

Hi Heath.

Certainly insightful. At the moment, our upcoming release candidate for v1.3.1 addresses a Boron specific issue. If you continue to notice this issue post v1.3.1-rc, it is helpful to know that this issue is still exclusively impacting Borons.

Hi folks, we have released device OS v1.3.1-rc.1 to tackle some of the issues causing difficulty to all the users here. Please refer to Status Update for cellular connectivity issues | 8/21/19 for instructions on how to provide feedback and debug logs. Thanks!

@hwestbrook @trpropst

@hwestbrook - Did OSv1.3.1-rc.1 fix your issue? I had a device on my test bench that has very similar looking cloud debug logs “Trace->AT” that started up flashing green. The RC didn’t fix it.

I discovered this week that I had 6 devices in the field also permanently flashing green. I’ve only taken one out of the weatherproof case (I’ve had issues with case screws stripping out, so I try to avoid it), and it also had a similar looking clouddebug log. THis is an old thread I understand. I’m delayed, because my devices are mostly only used in summer, I assumed the devices were offline because the operator had shut off the main power, but I was surprised to see they were all flashing green even after power cycles. Out of 15 devices, 6 are experiencing this behavior.

@jonpasc,

It did recover a few of them, and didn’t others. I didn’t have time to dive much deeper into the issue, and Particle replaced a few of them for me.

The main takeaway for these LTE boron’s without the updated ublox firmware is to avoid using them in situations where the power is not guaranteed.

Did the 1.3.1-rc1 and 1.4.4 update that ublox modem firmware to your knowledge? I’m trying to get to a build where I can set and forget. I had one device fail on solar and 6 of a different type of setup on a power source that’s either on or off for months at a time. The cell reception is spotty.

To my knowledge, Particle does not have the ability to update the ublox firmware through device-os. If you search elsewhere in the forum you’ll find people have made some custom PCBs for performing this.

Thanks for pointing that out. I found Ublox modem firmware update with very specific directions. Fortunately, I’m in an off season until March, so I can sit on this a while longer and see if Particle finds a solution. I’ve been 0 for 7 in bringing devices that started flashing green to life with an upgrade to 1.4.4 using “particle update” in the CLI. Unfortunately, i’m not sure the root cause is addressed and it’s just a matter of time before my other deices fall into the same state.

@jonpasc
Were you able to bring them back online and install the latest firmware?

Reason is that I have shipped some units out of the country and my customer mentioned that the two LTE borons have been blinking green (looking for cellular connection) since December, in the console I have reported that last connection happened on Dec 1st.
But when I check the information these sensors send to my server, they show as if they were sending info last week, but now that I am trying to troubleshoot them -it is a pain in the neck- because they just stay blinking green, even after resetting them.

Both of these Borons are running under the firmware 1.2.1

I was not able to bring them back to life even after plugging them into a computer and running particle update which should be the best update method. However, it sounds like your’s are still sending data? Mine weren’t doing a thing. The best troubleshooting is to download and flash the cloud debug from rickkas7. Unfortunately, if it’s the ublox module, you have to flash the debugger over usb and read the output over serial to confirm that the ublox module is indeed the culprit. https://github.com/rickkas7/boron-clouddebug

I'm in the middle of debugging an issue where my code had some of the same debug output that I was reading over serial. I discovered that when my device got stuck in a disconnected state, just opening a serial connection over USB revived it. I was able to repeat this several times, letting the device sit in the error state for hours and seeing it connect immediately once I opened my serial console.

Just FYI in case you have trouble reproducing the problem when watching debug. I have not yet figure out why this was happening.

How do you confirm that it is the ublox module from the log of the clouddebugger?

My boron went down following a power down event and since then (2+ months) flashing green.

manufacturer=
model=
firmware version=
ordering code=
IMEI=
IMSI=
ICCID=
0000710976 [app] INFO: enabling trace logging
0000710977 [ncp.at] TRACE: > AT+CGDCONT?
0000720979 [ncp.at] TRACE: > AT+CSQ
attempting to connect to the cellular network...
0000730982 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000730982 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000730987 [hal] TRACE: PPP netif -> 8
0000730988 [net.ifapi] INFO: Netif pp3 state UP
0000730988 [net.ifapi] INFO: Netif pp3 state UP
0000730992 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000730992 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000730992 [hal] TRACE: Powering modem on
0000731148 [hal] TRACE: Modem powered on
0000731150 [hal] TRACE: Setting UART voltage translator state 1
0000732151 [ncp.at] TRACE: > AT
0000733151 [ncp.at] TRACE: > AT
0000734152 [ncp.at] TRACE: > AT
0000735152 [ncp.at] TRACE: > AT
0000736153 [ncp.at] TRACE: > AT
0000737154 [ncp.at] TRACE: > AT
0000738154 [ncp.at] TRACE: > AT
0000739155 [ncp.at] TRACE: > AT
0000740156 [ncp.at] TRACE: > AT
0000740766 [sys.power] TRACE: re-enabling charging
0000740796 [sys.power] TRACE: Battery state UNKNOWN -> CHARGED
0000741157 [ncp.at] TRACE: > AT
0000741801 [sys.power] TRACE: Battery state CHARGED -> DISCONNECTED
0000742157 [ncp.at] TRACE: > AT
0000743158 [ncp.at] TRACE: > AT
0000744158 [ncp.at] TRACE: > AT
0000745159 [ncp.at] TRACE: > AT
0000746160 [ncp.at] TRACE: > AT
0000747161 [ncp.at] TRACE: > AT
0000748162 [ncp.at] TRACE: > AT
0000749163 [ncp.at] TRACE: > AT
0000750163 [ncp.at] TRACE: > AT
0000751163 [ncp.at] TRACE: > AT
0000752163 [hal] ERROR: No response from NCP
0000752163 [hal] ERROR: No response from NCP
0000752165 [hal] TRACE: Setting UART voltage translator state 0
0000752168 [hal] TRACE: Hard resetting the modem
0000763168 [net.pppncp] TRACE: Failed to initialize ublox NCP client: -210
0000763170 [hal] TRACE: Setting UART voltage translator state 0
0000763172 [hal] TRACE: Modem already off
0000763273 [hal] TRACE: Powering modem on
0000763424 [hal] TRACE: Modem powered on
0000763426 [hal] TRACE: Setting UART voltage translator state 1
0000764428 [ncp.at] TRACE: > AT
0000765429 [ncp.at] TRACE: > AT
0000766430 [ncp.at] TRACE: > AT
0000767431 [ncp.at] TRACE: > AT
0000768432 [ncp.at] TRACE: > AT
0000769433 [ncp.at] TRACE: > AT
0000770434 [ncp.at] TRACE: > AT
0000771435 [ncp.at] TRACE: > AT
0000772435 [ncp.at] TRACE: > AT

I was told that the above log indicates a bricked modem - only fix is replacement.
FYI - for those who might see the same.

It would be very helpful to know what causes this condition. I understand improper firmware flashing or trying to upgrade modem firmware can cause it, but in your case those things presumably did not happen. Particle, are there power or operating conditions that are known to cause the modems to fail? I’ve seen this same thing once or twice as well on Boron LTE.

@evk02 - recommend that you contact particle support and start a ticket. They’ll be able to help you. I’m no expert, I just know that they’ll ask you to run the logs. You’ll have to see what they think it means. @picsil - I don’t think it’s reasonably possible to upgrade the modem firmware without modifying the board. There was one post about drilling the board and making a jig to manually update, but it was not a tinker weekend project. You’d have to be serious about wanting to upgrade the firmware.

@jonpasc, @picsil,
I had started another thread earlier on this specific problem including all the logs, see Boron LTE - Permanently flashing green after running out of battery

@DJ_Volt has replied over there that this is a known, although rare scenario and the only remedy is to replace the board.

And no, there was no firmware flashing or modem upgrade. The device was in the field connected to a solar panel and battery. There was a long period of cloudy conditions, so it lost power. The next time I visited the device I saw the symptoms described above, permanently flashing green and generating the logs copied.

Following @DJ_Volt 's comment in that other thread, I have raised a ticket for the board replacement. I did not get more detailed explanation about causes of this condition in the ticket, although I didn’t ask for it. If you are interested, maybe @DJ_Volt can provide that.

I strongly suspect these issues are caused by the uBlox bug I referenced above.

You are probably right. I was suspecting that it might be the same issue - which is why I have posted my issue in this thread too.

Wow! I didn’t read the information since my last post. Unbelievable!

I just got back from Mexico, where I had the systems deployed initially - but after reading @evk02 situation is exactly the same that happened to these borons 2G/3G I have now -flashing green ( I have the same report with the “Failed to initialize ublox NCP client: -210” ).

I even try to get them back following the next procedure Boron Debug - rickkas .

This whole experience with Particle has been bitter sweet, but more bitter than sweet.
Pretty much all the prototypes or testing products built, somehow or somewhere down the road they break or something weird happens to them and the possible final users start loosing interest - so far it has been my experience, which has taken us to try and stick with other technologies.