Boron LTE connectivity issue - Green LED Blinking

We’ve deployed our products with Particle Boron (BRN402) and we’re observing that some Borons just suddenly stop connecting. We’ve had this issue with more than a few Borons this year. Cloud debug logs below from the latest Boron that has this issue:

[2022-11-16 21:41:58.630] 0000033386 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.630] 0000034386 [ncp.client] TRACE: Modem is not responsive @ 115200 baudrate
[2022-11-16 21:41:58.637] 0000034386 [ncp.client] ERROR: No response from NCP
[2022-11-16 21:41:58.638] 0000034387 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:41:58.638] 0000034387 [ncp.client] TRACE: Hard resetting the modem
[2022-11-16 21:41:58.638] 0000034387 [ncp.client] TRACE: Modem waiting up to 30s to power off with PWR_UC...
[2022-11-16 21:41:58.638] 0000075487 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:41:58.638] 0000075487 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_DOWN
[2022-11-16 21:41:58.638] 0000075487 [system.nm] TRACE: Interface 4 power state changed: 1
[2022-11-16 21:41:58.638] 0000075488 [ncp.client] TRACE: Deinit modem serial.
[2022-11-16 21:41:58.638] 0000075489 [net.pppncp] ERROR: Failed to initialize cellular NCP client: -210
[2022-11-16 21:41:58.638] 0000075489 [app] INFO: modem is not yet responding
[2022-11-16 21:41:58.638] 0000075490 [app] INFO: Still trying to connect to cellular 00:52
[2022-11-16 21:41:58.638] 0000075490 [app] INFO: Power source: USB Host
[2022-11-16 21:41:58.638] 0000075492 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:41:58.638] 0000075506 [app] INFO: modem is not yet responding
[2022-11-16 21:41:58.638] 0000075589 [ncp.client] TRACE: Powering modem on, ncpId: 0x44
[2022-11-16 21:41:58.638] 0000075589 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:41:58.638] 0000075590 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_POWERING_UP
[2022-11-16 21:41:58.647] 0000075590 [system.nm] TRACE: Interface 4 power state changed: 4
[2022-11-16 21:41:58.647] 0000075740 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:41:58.647] 0000075740 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_UP
[2022-11-16 21:41:58.647] 0000075740 [system.nm] TRACE: Interface 4 power state changed: 2
[2022-11-16 21:41:58.647] 0000075741 [ncp.client] TRACE: Modem powered on
[2022-11-16 21:41:58.647] 0000075741 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:41:58.647] 0000075841 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:41:58.647] 0000075941 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:41:58.647] > 0000076942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000077942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000078942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000079942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000080942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000081942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000082942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000083942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000084942 [ncp.at] TRACE: > AT
[2022-11-16 21:41:58.647] 0000085489 [app] INFO: Still trying to connect to cellular 01:02
[2022-11-16 21:41:58.647] 0000085489 [app] INFO: Power source: USB Host
[2022-11-16 21:41:58.647] 0000085492 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:42:22.116] Particle Cloud Debugging Tool
[2022-11-16 21:42:22.116] Enter 'help' command to list available commands
[2022-11-16 21:42:22.116] > 0000015477 [app] INFO: Starting Tests!
[2022-11-16 21:42:36.726] 0000015477 [app] INFO: Platform: Boron
[2022-11-16 21:42:36.726] 0000015477 [app] INFO: Binary compiled for: 4.0.0
[2022-11-16 21:42:36.726] 0000015478 [app] INFO: Cloud Debug Release 4.0.5
[2022-11-16 21:42:36.726] 0000015478 [app] INFO: System version: 4.0.0
[2022-11-16 21:42:36.727] 0000015479 [app] INFO: Device ID: e00fce68361717c9c8ba8d5f
[2022-11-16 21:42:36.729] 0000015479 [app] INFO: Power source: USB Host
[2022-11-16 21:42:36.729] 0000015482 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:42:36.730] 0000015482 [app] INFO: This device could have Ethernet (is 3rd generation)
[2022-11-16 21:42:36.731] 0000015483 [app] INFO: FEATURE_ETHERNET_DETECTION not enabled, so Ethernet will not be used (even i
[2022-11-16 21:42:44.739] 0000023492 [app] INFO: deviceID: e00fce68361717c9c8ba8d5f
[2022-11-16 21:42:44.739] 0000023493 [app] INFO: PMIC inputVoltageLimit: 3880 mV
[2022-11-16 21:42:44.741] 0000023494 [app] INFO: PMIC inputCurrentLimit: 500 mA
[2022-11-16 21:42:44.744] 0000023495 [app] INFO: PMIC minimumSystemVoltage: 3500 mV
[2022-11-16 21:42:44.744] 0000023496 [app] INFO: PMIC chargeCurrentValue: 896 mA
[2022-11-16 21:42:44.745] 0000023496 [app] INFO: PMIC chargeVoltageValue: 4112 mV
[2022-11-16 21:42:44.747] 0000023497 [app] INFO: modem is not yet responding
[2022-11-16 21:42:44.747] 0000023498 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
[2022-11-16 21:42:44.748] 0000023498 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
[2022-11-16 21:42:44.748] 0000023500 [ncp.client] TRACE: Powering modem on, ncpId: 0x44
[2022-11-16 21:42:44.750] 0000023500 [ncp.client] TRACE: Modem already on
[2022-11-16 21:42:44.751] 0000023500 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:42:44.753] 0000023501 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
[2022-11-16 21:42:44.755] 0000023512 [app] INFO: Still trying to connect to cellular 00:00
[2022-11-16 21:42:44.758] 0000023512 [app] INFO: Power source: USB Host
[2022-11-16 21:42:44.758] 0000023515 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:42:44.842] 0000023600 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:42:44.945] 0000023700 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:42:44.945] 0000023701 [ncp.client] TRACE: Muxer is not currently running
[2022-11-16 21:42:44.947] 0000023702 [ncp.client] TRACE: Initialized muxer @ 460800 baudrate
[2022-11-16 21:42:44.947] 0000023702 [mux] INFO: Starting GSM07.10 muxer
[2022-11-16 21:42:44.949] 0000023703 [mux] INFO: GSM07.10 muxer thread started
[2022-11-16 21:42:45.246] 0000024002 [mux] INFO: Stopping GSM07.10 muxer
[2022-11-16 21:42:45.248] 0000024003 [mux] INFO: Gracefully stopping GSM07.10 muxer
[2022-11-16 21:42:45.248] 0000024003 [mux] INFO: Closing all muxed channels
[2022-11-16 21:42:45.248] 0000024003 [mux] INFO: Muxed channel 1 already closed
[2022-11-16 21:42:45.250] 0000024004 [mux] INFO: Muxed channel 2 already closed
[2022-11-16 21:42:45.250] 0000024004 [mux] INFO: Muxed channel 3 already closed
[2022-11-16 21:42:45.250] 0000024004 [mux] INFO: Muxed channel 4 already closed
[2022-11-16 21:42:45.898] 0000024656 [mux] INFO: Sending CLD (multiplexer close down)
[2022-11-16 21:42:46.801] 0000025559 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
[2022-11-16 21:42:46.853] 0000025609 [mux] INFO: GSM07.10 muxer thread exiting
[2022-11-16 21:42:46.853] 0000025610 [mux] INFO: GSM07.10 muxer stopped
[2022-11-16 21:42:46.855] 0000025610 [ncp.client] TRACE: Initialized muxer @ 115200 baudrate
[2022-11-16 21:42:46.855] 0000025611 [mux] INFO: Starting GSM07.10 muxer
[2022-11-16 21:42:46.857] 0000025611 [mux] INFO: GSM07.10 muxer thread started
[2022-11-16 21:42:47.155] 0000025911 [mux] INFO: Stopping GSM07.10 muxer
[2022-11-16 21:42:47.158] 0000025912 [mux] INFO: Gracefully stopping GSM07.10 muxer
[2022-11-16 21:42:47.158] 0000025912 [mux] INFO: Closing all muxed channels
[2022-11-16 21:42:47.160] 0000025912 [mux] INFO: Muxed channel 1 already closed
[2022-11-16 21:42:47.160] 0000025913 [mux] INFO: Muxed channel 2 already closed
[2022-11-16 21:42:47.160] 0000025913 [mux] INFO: Muxed channel 3 already closed
[2022-11-16 21:42:47.160] 0000025913 [mux] INFO: Muxed channel 4 already closed
[2022-11-16 21:42:47.808] 0000026565 [mux] INFO: Sending CLD (multiplexer close down)
[2022-11-16 21:42:48.711] 0000027468 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
[2022-11-16 21:42:48.760] 0000027518 [mux] INFO: GSM07.10 muxer thread exiting
[2022-11-16 21:42:48.762] 0000027519 [mux] INFO: GSM07.10 muxer stopped
[2022-11-16 21:42:48.763] 0000027520 [ncp.at] TRACE: > AT
[2022-11-16 21:42:49.761] 0000028520 [ncp.at] TRACE: > AT
[2022-11-16 21:42:50.764] 0000029521 [ncp.client] TRACE: Modem is not responsive @ 460800 baudrate
[2022-11-16 21:42:50.764] 0000029522 [ncp.at] TRACE: > AT
[2022-11-16 21:42:51.767] 0000030522 [ncp.at] TRACE: > AT
[2022-11-16 21:42:52.764] 0000031523 [ncp.at] TRACE: > AT
[2022-11-16 21:42:53.765] 0000032524 [ncp.at] TRACE: > AT
[2022-11-16 21:43:36.873] 0000033525 [ncp.at] TRACE: > AT
[2022-11-16 21:43:36.874] 0000034526 [ncp.client] TRACE: Modem is not responsive @ 115200 baudrate
[2022-11-16 21:43:36.874] 0000034527 [ncp.client] ERROR: No response from NCP
[2022-11-16 21:43:36.876] 0000034527 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:43:36.878] 0000034527 [ncp.client] TRACE: Hard resetting the modem
[2022-11-16 21:43:36.878] 0000034528 [ncp.client] TRACE: Modem waiting up to 30s to power off with PWR_UC...
[2022-11-16 21:43:36.879] 0000075627 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:43:36.879] 0000075628 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_DOWN
[2022-11-16 21:43:36.881] 0000075628 [system.nm] TRACE: Interface 4 power state changed: 1
[2022-11-16 21:43:36.881] 0000075628 [ncp.client] TRACE: Deinit modem serial.
[2022-11-16 21:43:36.887] 0000075629 [net.pppncp] ERROR: Failed to initialize cellular NCP client: -210
[2022-11-16 21:43:36.887] 0000075630 [app] INFO: modem is not yet responding
[2022-11-16 21:43:36.888] 0000075630 [app] INFO: Still trying to connect to cellular 00:52
[2022-11-16 21:43:36.888] 0000075630 [app] INFO: Power source: USB Host
[2022-11-16 21:43:36.888] 0000075633 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:43:36.888] 0000075646 [app] INFO: modem is not yet responding
[2022-11-16 21:43:36.972] 0000075729 [ncp.client] TRACE: Powering modem on, ncpId: 0x44
[2022-11-16 21:43:36.972] 0000075730 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:43:36.974] 0000075730 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_POWERING_UP
[2022-11-16 21:43:36.974] 0000075730 [system.nm] TRACE: Interface 4 power state changed: 4
[2022-11-16 21:43:37.123] 0000075880 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:43:37.123] 0000075881 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_UP
[2022-11-16 21:43:37.124] 0000075881 [system.nm] TRACE: Interface 4 power state changed: 2
[2022-11-16 21:43:37.125] 0000075881 [ncp.client] TRACE: Modem powered on
[2022-11-16 21:43:37.125] 0000075882 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:43:37.222] 0000075981 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:43:37.323] 0000076081 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:43:38.323] 0000077082 [ncp.at] TRACE: > AT
[2022-11-16 21:43:39.325] 0000078083 [ncp.at] TRACE: > AT
[2022-11-16 21:43:40.325] 0000079084 [ncp.at] TRACE: > AT
[2022-11-16 21:43:41.327] 0000080085 [ncp.at] TRACE: > AT
[2022-11-16 21:43:42.328] 0000081086 [ncp.at] TRACE: > AT
[2022-11-16 21:43:43.328] 0000082087 [ncp.at] TRACE: > AT
[2022-11-16 21:43:44.330] 0000083088 [ncp.at] TRACE: > AT
[2022-11-16 21:43:45.330] 0000084089 [ncp.at] TRACE: > AT
[2022-11-16 21:43:46.331] 0000085090 [ncp.at] TRACE: > AT
[2022-11-16 21:43:46.873] 0000085630 [app] INFO: Still trying to connect to cellular 01:02
[2022-11-16 21:43:46.873] 0000085630 [app] INFO: Power source: USB Host
[2022-11-16 21:43:46.874] 0000085633 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:44:34.444] 0000086091 [ncp.at] TRACE: > AT
[2022-11-16 21:44:34.444] 0000087093 [ncp.at] TRACE: > AT
[2022-11-16 21:44:34.444] 0000088093 [ncp.at] TRACE: > AT
[2022-11-16 21:44:34.444] 0000089094 [ncp.at] TRACE: > AT
[2022-11-16 21:44:34.447] 0000090095 [ncp.at] TRACE: > AT
[2022-11-16 21:44:34.447] 0000091096 [ncp.at] TRACE: > AT
[2022-11-16 21:44:34.447] 0000092097 [ncp.client] ERROR: No response from NCP
[2022-11-16 21:44:34.447] 0000092098 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:44:34.448] 0000092098 [ncp.client] TRACE: Hard resetting the modem
[2022-11-16 21:44:34.450] 0000092098 [ncp.client] TRACE: Modem waiting up to 30s to power off with PWR_UC...
[2022-11-16 21:44:34.451] 0000133198 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:44:34.451] 0000133199 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_DOWN
[2022-11-16 21:44:34.453] 0000133199 [system.nm] TRACE: Interface 4 power state changed: 1
[2022-11-16 21:44:34.453] 0000133199 [ncp.client] TRACE: Deinit modem serial.
[2022-11-16 21:44:34.455] 0000133200 [net.pppncp] ERROR: Failed to initialize cellular NCP client: -210
[2022-11-16 21:44:34.455] 0000133201 [app] INFO: modem is not yet responding
[2022-11-16 21:44:34.456] 0000133201 [app] INFO: Still trying to connect to cellular 01:49
[2022-11-16 21:44:34.457] 0000133201 [app] INFO: Power source: USB Host
[2022-11-16 21:44:34.458] 0000133204 [app] INFO: Battery state: charged, SoC: 99.89
[2022-11-16 21:44:34.459] 0000133219 [app] INFO: modem is not yet responding
[2022-11-16 21:44:34.541] 0000133300 [ncp.client] TRACE: Powering modem on, ncpId: 0x44
[2022-11-16 21:44:34.542] 0000133301 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:44:34.542] 0000133301 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_POWERING_UP
[2022-11-16 21:44:34.544] 0000133301 [system.nm] TRACE: Interface 4 power state changed: 4
[2022-11-16 21:44:34.691] 0000133451 [net.pppncp] TRACE: NCP event 3
[2022-11-16 21:44:34.693] 0000133452 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_UP
[2022-11-16 21:44:34.694] 0000133452 [system.nm] TRACE: Interface 4 power state changed: 2
[2022-11-16 21:44:34.695] 0000133452 [ncp.client] TRACE: Modem powered on
[2022-11-16 21:44:34.695] 0000133453 [ncp.client] TRACE: Setting UART voltage translator state 1
[2022-11-16 21:44:34.792] 0000133552 [ncp.client] TRACE: Setting UART voltage translator state 0
[2022-11-16 21:44:34.894] 0000133652 [ncp.client] TRACE: Setting UART voltage translator state 1

It looks like TAN004.

How are you powering the devices? And, more importantly, is there any circumstances where power can be removed from the device abruptly, or the device could run with very low battery power?

After reading TAN004, I do agree it seems to match the details. The devices don’t lose power abruptly but we do run them with low battery power sometimes until the battery is replaced.

As far as I understand according to TAN004, we should check Cellular.isOff before turning off power. Does going to DEEP_SLEEP count as turning off power? So we should check Cellular.isOff before deep sleeping?

Using sleep calls will handle the safe power down automatically. You only have to worry about it if you are using the EN pin on the Boron to remove power, or an external load switch to control power.

You should add code to setup to avoid connecting to the cloud if the battery level is very low. The problem is that when you start to connect to cellular and the cloud, the high current draw plus a low battery charge can cause a brownout, which could cause TAN004.

Understood. We don’t use EN pin to remove power so it’s likely caused by connecting to cellular/cloud with low battery. Is this the case for all cellular Particle products or just the ones with the SARA-R410M-02B module? We’re planning to move to B404X in the future so I’d like to know if there are any such considerations or recommendations for low battery use.

Also, would it be possible for us to get replacements for these Particle devices as the modem module seems to be permanently corrupted?

FYI we had this issue a lot, in our case we use the EN pin. Our issue was that calling cellular.off may actually take up to 60s and we had a 10s timeout.
Particle does warranty them although I have not used the warranty.