We are aware of the issue described here. I'm uncertain of its relevance to the issue you are experiencing, but is something engineering will certainly investigate. Particle already has existing communications open with ublox in regards to known bugs and discussions on means of handling firmware update for our in-use ublox modules.
Hi @mstanley,
I’ve got three devices with this problem. They blink green trying to connect to the cloud forever. They were running 1.1.0-rc.2 when encountering the issue. Updating them to 1.2.1 does not fix the issue.
clouddebug: press letter corresponding to the command
a - enter APN for 3rd-party SIM card
k - set keep-alive value
c - show carriers at this location
p - select MNO profile
t - run normal tests (occurs automatically after 10 seconds)
or tap the MODE button once to show carriers
starting tests...
turning cellular on...
0000015009 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
0000036158 [hal] ERROR: No response from NCP
deviceID=e00fce68f378b52473295f7d
manufacturer=
model=
firmware version=
ordering code=
IMEI=
IMSI=
ICCID=
0000117160 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000117164 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117164 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117168 [hal] TRACE: PPP netif -> 8
0000117169 [net.ifapi] INFO: Netif pp3 state UP
0000117169 [net.ifapi] INFO: Netif pp3 state UP
0000117171 [hal] TRACE: Powering modem on
0000117172 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117172 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117322 [hal] TRACE: Modem powered on
0000117323 [hal] TRACE: Setting UART voltage translator state 1
0000118325 [ncp.at] TRACE: > AT
0000119326 [ncp.at] TRACE: > AT
0000120327 [ncp.at] TRACE: > AT
0000121327 [ncp.at] TRACE: > AT
0000122328 [ncp.at] TRACE: > AT
0000123329 [ncp.at] TRACE: > AT
0000124330 [ncp.at] TRACE: > AT
0000125330 [ncp.at] TRACE: > AT
0000126330 [ncp.at] TRACE: > AT
0000127331 [ncp.at] TRACE: > AT
0000128332 [ncp.at] TRACE: > AT
0000129333 [ncp.at] TRACE: > AT
0000130334 [ncp.at] TRACE: > AT
0000131335 [ncp.at] TRACE: > AT
0000132335 [ncp.at] TRACE: > AT
0000133336 [ncp.at] TRACE: > AT
0000134337 [ncp.at] TRACE: > AT
0000135338 [ncp.at] TRACE: > AT
0000136338 [ncp.at] TRACE: > AT
0000137339 [ncp.at] TRACE: > AT
0000138340 [hal] ERROR: No response from NCP
0000138340 [hal] ERROR: No response from NCP
0000138342 [hal] TRACE: Setting UART voltage translator state 0
0000138344 [hal] TRACE: Hard resetting the modem
0000149345 [net.pppncp] TRACE: Failed to initialize ublox NCP client: -210
0000149347 [hal] TRACE: Setting UART voltage translator state 0
0000149349 [hal] TRACE: Modem already off
0000149450 [hal] TRACE: Powering modem on
0000149600 [hal] TRACE: Modem powered on
0000149601 [hal] TRACE: Setting UART voltage translator state 1
0000150603 [ncp.at] TRACE: > AT
0000151604 [ncp.at] TRACE: > AT
0000152605 [ncp.at] TRACE: > AT
0000153606 [ncp.at] TRACE: > AT
0000154607 [ncp.at] TRACE: > AT
0000155608 [ncp.at] TRACE: > AT
0000156609 [ncp.at] TRACE: > AT
0000157610 [ncp.at] TRACE: > AT
Serial Inspect output
Platform: 13 - Boron
Modules
Bootloader module #0 - version 311, main location, 49152 bytes max size
Integrity: PASS
Address Range: PASS
Platform: PASS
Dependencies: PASS
System module #1 - version 1213, main location, 671744 bytes max size
Integrity: PASS
Address Range: PASS
Platform: PASS
Dependencies: PASS
Bootloader module #0 - version 311
User module #1 - version 5, main location, 131072 bytes max size
UUID: 362808E0DB24B4ED309227C9121675856A32E73D5D16D2C7922C49C9AEF353C2
Integrity: PASS
Address Range: PASS
Platform: PASS
Dependencies: PASS
System module #1 - version 326
These three devices are in the field with 17 others for a R&D project we are running with Borons. All three experienced this issue after running completely out of battery. They were not powered by a LiPo battery, but were powered by a 12V SLA battery run through a regulator.
All three of these devices had this problem after running the 12V SLA completely out of power (down to ~6V). I think this may have something to do with that? Maybe a modem brownout?
Hi Heath,
Those logs are fantastic, thanks so much for providing those! I have ensured that engineering will be taking a look at these.
We are still investigating this issue on our end. We have a device OS release coming sometime next week to mitigate some of the Boron LTE fail cases. As that comes out, we will be posting a new community topic for the release and instructions on how to capture additional debug logging so that we can continue to investigate connectivity concerns.
Not sure if this helps, but our carrier board can sit either a Boron or Electron and we have about 20 of each in the field. None of the Electrons have failed this way (or any other way).
Hi Heath.
Certainly insightful. At the moment, our upcoming release candidate for v1.3.1 addresses a Boron specific issue. If you continue to notice this issue post v1.3.1-rc, it is helpful to know that this issue is still exclusively impacting Borons.
Hi folks, we have released device OS v1.3.1-rc.1 to tackle some of the issues causing difficulty to all the users here. Please refer to Status Update for cellular connectivity issues | 8/21/19 for instructions on how to provide feedback and debug logs. Thanks!
@hwestbrook - Did OSv1.3.1-rc.1 fix your issue? I had a device on my test bench that has very similar looking cloud debug logs “Trace->AT” that started up flashing green. The RC didn’t fix it.
I discovered this week that I had 6 devices in the field also permanently flashing green. I’ve only taken one out of the weatherproof case (I’ve had issues with case screws stripping out, so I try to avoid it), and it also had a similar looking clouddebug log. THis is an old thread I understand. I’m delayed, because my devices are mostly only used in summer, I assumed the devices were offline because the operator had shut off the main power, but I was surprised to see they were all flashing green even after power cycles. Out of 15 devices, 6 are experiencing this behavior.
It did recover a few of them, and didn’t others. I didn’t have time to dive much deeper into the issue, and Particle replaced a few of them for me.
The main takeaway for these LTE boron’s without the updated ublox firmware is to avoid using them in situations where the power is not guaranteed.
Did the 1.3.1-rc1 and 1.4.4 update that ublox modem firmware to your knowledge? I’m trying to get to a build where I can set and forget. I had one device fail on solar and 6 of a different type of setup on a power source that’s either on or off for months at a time. The cell reception is spotty.
To my knowledge, Particle does not have the ability to update the ublox firmware through device-os. If you search elsewhere in the forum you’ll find people have made some custom PCBs for performing this.
Thanks for pointing that out. I found Ublox modem firmware update with very specific directions. Fortunately, I’m in an off season until March, so I can sit on this a while longer and see if Particle finds a solution. I’ve been 0 for 7 in bringing devices that started flashing green to life with an upgrade to 1.4.4 using “particle update” in the CLI. Unfortunately, i’m not sure the root cause is addressed and it’s just a matter of time before my other deices fall into the same state.
@jonpasc
Were you able to bring them back online and install the latest firmware?
Reason is that I have shipped some units out of the country and my customer mentioned that the two LTE borons have been blinking green (looking for cellular connection) since December, in the console I have reported that last connection happened on Dec 1st.
But when I check the information these sensors send to my server, they show as if they were sending info last week, but now that I am trying to troubleshoot them -it is a pain in the neck- because they just stay blinking green, even after resetting them.
Both of these Borons are running under the firmware 1.2.1
I was not able to bring them back to life even after plugging them into a computer and running particle update which should be the best update method. However, it sounds like your’s are still sending data? Mine weren’t doing a thing. The best troubleshooting is to download and flash the cloud debug from rickkas7. Unfortunately, if it’s the ublox module, you have to flash the debugger over usb and read the output over serial to confirm that the ublox module is indeed the culprit. https://github.com/rickkas7/boron-clouddebug
I'm in the middle of debugging an issue where my code had some of the same debug output that I was reading over serial. I discovered that when my device got stuck in a disconnected state, just opening a serial connection over USB revived it. I was able to repeat this several times, letting the device sit in the error state for hours and seeing it connect immediately once I opened my serial console.
Just FYI in case you have trouble reproducing the problem when watching debug. I have not yet figure out why this was happening.
How do you confirm that it is the ublox module from the log of the clouddebugger?
My boron went down following a power down event and since then (2+ months) flashing green.
manufacturer=
model=
firmware version=
ordering code=
IMEI=
IMSI=
ICCID=
0000710976 [app] INFO: enabling trace logging
0000710977 [ncp.at] TRACE: > AT+CGDCONT?
0000720979 [ncp.at] TRACE: > AT+CSQ
attempting to connect to the cellular network...
0000730982 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000730982 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000730987 [hal] TRACE: PPP netif -> 8
0000730988 [net.ifapi] INFO: Netif pp3 state UP
0000730988 [net.ifapi] INFO: Netif pp3 state UP
0000730992 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000730992 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000730992 [hal] TRACE: Powering modem on
0000731148 [hal] TRACE: Modem powered on
0000731150 [hal] TRACE: Setting UART voltage translator state 1
0000732151 [ncp.at] TRACE: > AT
0000733151 [ncp.at] TRACE: > AT
0000734152 [ncp.at] TRACE: > AT
0000735152 [ncp.at] TRACE: > AT
0000736153 [ncp.at] TRACE: > AT
0000737154 [ncp.at] TRACE: > AT
0000738154 [ncp.at] TRACE: > AT
0000739155 [ncp.at] TRACE: > AT
0000740156 [ncp.at] TRACE: > AT
0000740766 [sys.power] TRACE: re-enabling charging
0000740796 [sys.power] TRACE: Battery state UNKNOWN -> CHARGED
0000741157 [ncp.at] TRACE: > AT
0000741801 [sys.power] TRACE: Battery state CHARGED -> DISCONNECTED
0000742157 [ncp.at] TRACE: > AT
0000743158 [ncp.at] TRACE: > AT
0000744158 [ncp.at] TRACE: > AT
0000745159 [ncp.at] TRACE: > AT
0000746160 [ncp.at] TRACE: > AT
0000747161 [ncp.at] TRACE: > AT
0000748162 [ncp.at] TRACE: > AT
0000749163 [ncp.at] TRACE: > AT
0000750163 [ncp.at] TRACE: > AT
0000751163 [ncp.at] TRACE: > AT
0000752163 [hal] ERROR: No response from NCP
0000752163 [hal] ERROR: No response from NCP
0000752165 [hal] TRACE: Setting UART voltage translator state 0
0000752168 [hal] TRACE: Hard resetting the modem
0000763168 [net.pppncp] TRACE: Failed to initialize ublox NCP client: -210
0000763170 [hal] TRACE: Setting UART voltage translator state 0
0000763172 [hal] TRACE: Modem already off
0000763273 [hal] TRACE: Powering modem on
0000763424 [hal] TRACE: Modem powered on
0000763426 [hal] TRACE: Setting UART voltage translator state 1
0000764428 [ncp.at] TRACE: > AT
0000765429 [ncp.at] TRACE: > AT
0000766430 [ncp.at] TRACE: > AT
0000767431 [ncp.at] TRACE: > AT
0000768432 [ncp.at] TRACE: > AT
0000769433 [ncp.at] TRACE: > AT
0000770434 [ncp.at] TRACE: > AT
0000771435 [ncp.at] TRACE: > AT
0000772435 [ncp.at] TRACE: > AT
I was told that the above log indicates a bricked modem - only fix is replacement.
FYI - for those who might see the same.
It would be very helpful to know what causes this condition. I understand improper firmware flashing or trying to upgrade modem firmware can cause it, but in your case those things presumably did not happen. Particle, are there power or operating conditions that are known to cause the modems to fail? I’ve seen this same thing once or twice as well on Boron LTE.
@evk02 - recommend that you contact particle support and start a ticket. They’ll be able to help you. I’m no expert, I just know that they’ll ask you to run the logs. You’ll have to see what they think it means. @picsil - I don’t think it’s reasonably possible to upgrade the modem firmware without modifying the board. There was one post about drilling the board and making a jig to manually update, but it was not a tinker weekend project. You’d have to be serious about wanting to upgrade the firmware.
@jonpasc, @picsil,
I had started another thread earlier on this specific problem including all the logs, see Boron LTE - Permanently flashing green after running out of battery
@DJ_Volt has replied over there that this is a known, although rare scenario and the only remedy is to replace the board.
And no, there was no firmware flashing or modem upgrade. The device was in the field connected to a solar panel and battery. There was a long period of cloudy conditions, so it lost power. The next time I visited the device I saw the symptoms described above, permanently flashing green and generating the logs copied.
Following @DJ_Volt 's comment in that other thread, I have raised a ticket for the board replacement. I did not get more detailed explanation about causes of this condition in the ticket, although I didn’t ask for it. If you are interested, maybe @DJ_Volt can provide that.
I strongly suspect these issues are caused by the uBlox bug I referenced above.