Cannot hard reset modem, it's not on. Modem is not responsive @ 921600 baudrate

Firmware Device OS
1

We manage a few dozen Borons that we have deployed, but this is a new one on me. Wondering if this is indicative of a Particle OS bug, or else a hardware issue with this particular Boron

It is running Particle OS 4.2.0

I scraped my application's debug log file using the following regex:

.*[0-9]{10} \[(?!app)

Here are the Particle OS error messages:

  E:\Downloads\Boron_Debug_OS_Messages.txt (3 hits)
	Line 3501: 	Line 22886: 0000018041 [ncp.client] ERROR: Failed to perform early initialization
	Line 3503: 	Line 22888: 0000018042 [ncp.client] ERROR: Cannot hard reset the modem, it's not on
	Line 3504: 	Line 22889: 0000018042 [net.pppncp] ERROR: Failed to initialize cellular NCP client: -210

which leads me to believe that perhaps this Boron has hardware damage? The modem is not on?

Here is a snippet of the Particle OS stream that smells particularly fishy:

Line 22433: 0000002845 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
Line 22434: 0000002848 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
Line 22435: 0000002849 [ncp.client] TRACE: Powering modem on, ncpId: 0x42
Line 22436: 0000002849 [ncp.client] TRACE: Modem already on
Line 22437: 0000002849 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 22438: =======0000002850 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
Line 22486: 0000002949 [ncp.client] TRACE: Setting UART voltage translator state 0
Line 22487: 0000003049 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 22488: 0000003050 [ncp.client] TRACE: Muxer is not currently running
Line 22489: 0000003051 [ncp.client] TRACE: Initialized muxer @ 921600 baudrate
Line 22490: 0000003051 [mux] INFO: Starting GSM07.10 muxer
Line 22491: 0000003052 [mux] INFO: GSM07.10 muxer thread started
Line 22540: 0000003351 [mux] INFO: Stopping GSM07.10 muxer
Line 22541: 0000003352 [mux] INFO: Gracefully stopping GSM07.10 muxer
Line 22542: 0000003352 [mux] INFO: Closing all muxed channels
Line 22543: 0000003353 [mux] INFO: Muxed channel 1 already closed
Line 22544: 0000003353 [mux] INFO: Muxed channel 2 already closed
Line 22545: 0000003354 [mux] INFO: Muxed channel 3 already closed
Line 22546: 0000003354 [mux] INFO: Muxed channel 4 already closed
Line 22594: |\-/0000004005 [mux] INFO: Sending CLD (multiplexer close down)
Line 22601: 0000004906 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
Line 22608: 0000004956 [mux] INFO: GSM07.10 muxer thread exiting
Line 22609: 0000004956 [mux] INFO: GSM07.10 muxer stopped
Line 22610: 0000004957 [ncp.at] TRACE: > AT
Line 22654: /|\-/|\0000005957 [ncp.at] TRACE: > AT
Line 22667: 0000006957 [ncp.client] TRACE: Modem is not responsive @ 921600 baudrate
Line 22668: 0000006958 [ncp.at] TRACE: > AT
Line 22669: 0000006960 [ncp.at] TRACE: < OK
Line 22670: 0000006960 [ncp.client] TRACE: NCP ready to accept AT commands
Line 22671: 0000006961 [ncp.at] TRACE: > AT
Line 22672: 0000006963 [ncp.at] TRACE: < OK
Line 22673: 0000006964 [ncp.at] TRACE: > ATI9
Line 22674: 0000006977 [ncp.at] TRACE: < 23.60,A01.01
Line 22675: 0000006978 [ncp.at] TRACE: < OK
Line 22676: 0000006978 [ncp.client] TRACE: App firmware: 101
Line 22677: 0000006979 [ncp.at] TRACE: > AT+UGPIOC?
Line 22678: 0000006983 [ncp.at] TRACE: < +UGPIOC:
Line 22679: 0000006983 [ncp.at] TRACE: < 16,255
Line 22680: 0000006984 [ncp.at] TRACE: < 23,255
Line 22681: 0000006985 [ncp.at] TRACE: < 24,4
Line 22682: 0000006985 [ncp.at] TRACE: < 25,5
Line 22683: 0000006986 [ncp.at] TRACE: < 34,12
Line 22684: 0000006986 [ncp.at] TRACE: < 35,12
Line 22685: 0000006987 [ncp.at] TRACE: < 36,12
Line 22686: 0000006988 [ncp.at] TRACE: < 37,12
Line 22687: 0000006988 [ncp.at] TRACE: < 42,7
Line 22688: 0000006989 [ncp.at] TRACE: < OK
Line 22689: 0000006989 [ncp.client] INFO: Using internal SIM card
Line 22690: 0000006990 [ncp.at] TRACE: > AT+CPIN?
Line 22691: 0000006994 [ncp.at] TRACE: < +CPIN: READY
Line 22692: 0000006994 [ncp.at] TRACE: < OK
Line 22693: 0000006995 [ncp.at] TRACE: > AT+CCID
Line 22694: 0000007014 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 22695: 0000007015 [ncp.at] TRACE: < OK
Line 22696: 0000007015 [ncp.at] TRACE: > AT+IFC?
Line 22697: 0000007019 [ncp.at] TRACE: < +IFC: 2,2
Line 22698: 0000007020 [ncp.at] TRACE: < OK
Line 22699: 0000007020 [ncp.at] TRACE: > AT+IFC=2,2
Line 22700: 0000007023 [ncp.at] TRACE: < OK
Line 22701: 0000007023 [ncp.at] TRACE: > AT
Line 22702: \-/0000012041 [ncp.at] TRACE: > AT
Line 22756: 0000013041 [ncp.at] TRACE: > AT
Line 22772: 0000014041 [ncp.at] TRACE: > AT
Line 22816: -/|\-/|\0000015041 [ncp.at] TRACE: > AT
Line 22829: 0000016041 [ncp.at] TRACE: > AT
Line 22873: |\-/|\-/0000017041 [ncp.at] TRACE: > AT
Line 22886: 0000018041 [ncp.client] ERROR: Failed to perform early initialization
Line 22887: 0000018042 [ncp.client] TRACE: Setting UART voltage translator state 0
Line 22888: 0000018042 [ncp.client] ERROR: Cannot hard reset the modem, it's not on
Line 22889: 0000018042 [net.pppncp] ERROR: Failed to initialize cellular NCP client: -210
Line 22933: -0000018142 [ncp.client] TRACE: Powering modem on, ncpId: 0x42
Line 22934: 0000018142 [net.pppncp] TRACE: NCP event 3
Line 22935: 0000018143 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_POWERING_UP
Line 22936: 0000018143 [system.nm] TRACE: Interface 4 power state changed: 4
Line 22937: 0000018153 [net.pppncp] TRACE: NCP event 3
Line 22938: 0000018153 [net.pppncp] TRACE: NCP power state changed: IF_POWER_STATE_UP
Line 22939: 0000018154 [system.nm] TRACE: Interface 4 power state changed: 2
Line 22940: 0000018154 [ncp.client] TRACE: Modem powered on
Line 22941: 0000018155 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 22942: /0000018255 [ncp.client] TRACE: Setting UART voltage translator state 0
Line 22943: 0000018355 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 22944: |\-/|\-/0000019356 [ncp.at] TRACE: > AT
Line 23000: |\0000020356 [ncp.at] TRACE: > AT
Line 23001: -/|\-/|\0000021356 [ncp.at] TRACE: > AT
Line 23057: -/0000022356 [ncp.at] TRACE: > AT
Line 23058: 0000022359 [ncp.at] TRACE: < OK
Line 23059: 0000022359 [ncp.client] TRACE: NCP ready to accept AT commands
Line 23060: 0000022360 [ncp.at] TRACE: > AT
Line 23061: 0000022362 [ncp.at] TRACE: < OK
Line 23062: 0000022362 [ncp.at] TRACE: > ATI9
Line 23063: 0000022381 [ncp.at] TRACE: < 23.60,A01.01
Line 23064: 0000022381 [ncp.at] TRACE: < OK
Line 23065: 0000022382 [ncp.client] TRACE: App firmware: 101
Line 23066: 0000022382 [ncp.at] TRACE: > AT+UGPIOC?
Line 23067: 0000022386 [ncp.at] TRACE: < +UGPIOC:
Line 23068: 0000022387 [ncp.at] TRACE: < 16,255
Line 23069: 0000022387 [ncp.at] TRACE: < 23,255
Line 23070: 0000022388 [ncp.at] TRACE: < 24,4
Line 23071: 0000022389 [ncp.at] TRACE: < 25,5
Line 23072: 0000022389 [ncp.at] TRACE: < 34,12
Line 23073: 0000022390 [ncp.at] TRACE: < 35,12
Line 23074: 0000022390 [ncp.at] TRACE: < 36,12
Line 23075: 0000022391 [ncp.at] TRACE: < 37,12
Line 23076: 0000022391 [ncp.at] TRACE: < 42,7
Line 23077: 0000022392 [ncp.at] TRACE: < OK
Line 23078: 0000022393 [ncp.client] INFO: Using internal SIM card
Line 23079: 0000022393 [ncp.at] TRACE: > AT+CPIN?
Line 23080: |0000022397 [ncp.at] TRACE: < ERROR
Line 23134: 0000023397 [ncp.at] TRACE: > AT+CPIN?
Line 23135: 0000023402 [ncp.at] TRACE: < +CPIN: READY
Line 23136: 0000023402 [ncp.at] TRACE: < OK
Line 23137: 0000023403 [ncp.at] TRACE: > AT+CCID
Line 23138: 0000023423 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 23139: 0000023423 [ncp.at] TRACE: < OK
Line 23140: 0000023423 [ncp.at] TRACE: > AT+IFC?
Line 23141: 0000023427 [ncp.at] TRACE: < +IFC: 2,2
Line 23142: 0000023428 [ncp.at] TRACE: < OK
Line 23143: 0000023428 [ncp.at] TRACE: > AT+IFC=2,2
Line 23144: 0000023431 [ncp.at] TRACE: < OK
Line 23145: 0000023432 [ncp.at] TRACE: > AT
Line 23204: \-/0000024432 [ncp.at] TRACE: > AT
Line 23205: 0000024435 [ncp.at] TRACE: < OK
Line 23206: 0000024435 [ncp.at] TRACE: > AT+IPR=921600
Line 23207: 0000024439 [ncp.at] TRACE: < OK
Line 23208: 0000024441 [ncp.at] TRACE: > AT
Line 23209: 0000024442 [ncp.at] TRACE: < 
Line 23210: 0000024443 [ncp.at] TRACE: < OK
Line 23211: |\-/|\-0000025441 [ncp.at] TRACE: > AT
Line 23212: 0000025443 [ncp.at] TRACE: < OK
Line 23213: 0000025443 [ncp.at] TRACE: > AT+COPS=3,2
Line 23214: /0000025446 [ncp.at] TRACE: < OK
Line 23215: 0000025446 [ncp.at] TRACE: > AT+CGEREP=1,0
Line 23216: 0000025448 [ncp.at] TRACE: < OK
Line 23217: 0000025448 [ncp.at] TRACE: > AT+UPSV?
Line 23218: 0000025451 [ncp.at] TRACE: < +UPSV: 0
Line 23219: 0000025451 [ncp.at] TRACE: < OK
Line 23220: 0000025451 [ncp.at] TRACE: > AT+CMUX=0,0,,1509,,,,,
Line 23221: |0000025580 [ncp.at] TRACE: < OK
Line 23222: 0000025580 [mux] INFO: Starting GSM07.10 muxer
Line 23223: 0000025581 [mux] INFO: Opening mux channel 0
Line 23224: 0000025581 [mux] INFO: GSM07.10 muxer thread started
Line 23225: 0000025583 [mux] INFO: Opening mux channel 1
Line 23226: 0000025586 [ncp.at] TRACE: > AT
Line 23227: 0000025589 [ncp.at] TRACE: < OK
Line 23228: 0000025589 [ncp.client] TRACE: NCP state changed: 1
Line 23229: 0000025590 [net.pppncp] TRACE: NCP event 1
Line 23230: 0000025590 [net.ppp.client] TRACE: PPP thread event ADM_UP data=0
Line 23231: 0000025591 [net.ppp.client] TRACE: State NONE -> READY
Line 23232: 0000025594 [ncp.at] TRACE: > AT
Line 23233: 0000025596 [ncp.at] TRACE: < OK
Line 23234: 0000025597 [ncp.at] TRACE: > AT+CFUN?
Line 23235: 0000025600 [ncp.at] TRACE: < +CFUN: 1,0
Line 23236: 0000025600 [ncp.at] TRACE: < OK
Line 23237: 0000025601 [ncp.at] TRACE: > AT+CCID
Line 23238: 0000025618 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 23239: 0000025618 [ncp.at] TRACE: < OK
Line 23240: 0000025619 [ncp.at] TRACE: > AT+CIMI
Line 23241: 0000025622 [ncp.at] TRACE: < 204047127292930
Line 23242: 0000025623 [ncp.at] TRACE: < OK
Line 23243: 0000025623 [ncp.at] TRACE: > AT+CGDCONT?
Line 23244: 0000025628 [ncp.at] TRACE: < +CGDCONT: 1,"IP","vfd1.korem2m.com","0.0.0.0",0,0
Line 23245: 0000025628 [ncp.at] TRACE: < OK
Line 23246: 0000025629 [ncp.at] TRACE: > AT+CFUN?
Line 23247: 0000025632 [ncp.at] TRACE: < +CFUN: 1,0
Line 23248: 0000025632 [ncp.at] TRACE: < OK
Line 23249: 0000025633 [ncp.at] TRACE: > AT+CREG=2
Line 23250: 0000025636 [ncp.at] TRACE: < OK
Line 23251: 0000025636 [ncp.at] TRACE: > AT+CGREG=2
Line 23252: 0000025639 [ncp.at] TRACE: < OK
Line 23253: 0000025640 [ncp.client] TRACE: NCP connection state changed: 1
Line 23254: 0000025640 [net.pppncp] TRACE: NCP event 2
Line 23255: 0000025640 [net.pppncp] TRACE: State changed event: 1
Line 23256: 0000025641 [net.ppp.client] TRACE: PPP thread event LOWER_DOWN data=0
Line 23257: 0000025642 [ncp.at] TRACE: > AT+COPS?
Line 23258: 0000025644 [ncp.at] TRACE: < +COPS: 0
Line 23259: 0000025645 [ncp.at] TRACE: < OK
Line 23260: 0000025646 [ncp.at] TRACE: > AT+CREG?
Line 23261: 0000025649 [ncp.at] TRACE: < +CREG: 2,0
Line 23262: 0000025649 [ncp.at] TRACE: < OK
Line 23263: 0000025650 [ncp.at] TRACE: > AT+CGREG?
Line 23264: 0000025652 [ncp.at] TRACE: < +CGREG: 2,0
Line 23265: 0000025653 [ncp.at] TRACE: < OK
Line 23344: /===================================0000002860 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
Line 23345: 0000002862 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
Line 23347: ===============DEBUG DUMP==0000002863 [ncp.client] TRACE: Powering modem on, ncpId: 0x42
Line 23348: 0000002864 [ncp.client] TRACE: Modem already on
Line 23349: 0000002864 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 23354: 0000002865 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
Line 23398: 0000002964 [ncp.client] TRACE: Setting UART voltage translator state 0
Line 23399: 0000003064 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 23400: 0000003065 [ncp.client] TRACE: Muxer is not currently running
Line 23401: 0000003066 [ncp.client] TRACE: Initialized muxer @ 921600 baudrate
Line 23402: 0000003066 [mux] INFO: Starting GSM07.10 muxer
Line 23403: 0000003067 [mux] INFO: GSM07.10 muxer thread started
Line 23464: 0000003366 [mux] INFO: Stopping GSM07.10 muxer
Line 23465: 0000003366 [mux] INFO: Gracefully stopping GSM07.10 muxer
Line 23466: 0000003367 [mux] INFO: Closing all muxed channels
Line 23467: 0000003367 [mux] INFO: Muxed channel 1 already closed
Line 23468: 0000003368 [mux] INFO: Muxed channel 2 already closed
Line 23469: 0000003368 [mux] INFO: Muxed channel 3 already closed
Line 23470: 0000003368 [mux] INFO: Muxed channel 4 already closed
Line 23520: |\-/0000004019 [mux] INFO: Sending CLD (multiplexer close down)
Line 23527: 0000004920 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
Line 23534: 0000004970 [mux] INFO: GSM07.10 muxer thread exiting
Line 23535: 0000004971 [mux] INFO: GSM07.10 muxer stopped
Line 23536: 0000004971 [ncp.at] TRACE: > AT
Line 23580: /|\-/|\0000005972 [ncp.at] TRACE: > AT
Line 23593: 0000006972 [ncp.client] TRACE: Modem is not responsive @ 921600 baudrate
Line 23594: 0000006973 [ncp.at] TRACE: > AT
Line 23595: 0000006974 [ncp.at] TRACE: < �AT
Line 23596: 0000006975 [ncp.at] TRACE: < OK
Line 23640: \-/|\-/0000007975 [ncp.at] TRACE: > AT
Line 23641: 0000007977 [ncp.at] TRACE: < OK
Line 23642: 0000007978 [ncp.client] TRACE: NCP ready to accept AT commands
Line 23643: 0000007978 [ncp.at] TRACE: > AT
Line 23644: 0000007980 [ncp.at] TRACE: < OK
Line 23645: 0000007980 [ncp.at] TRACE: > ATI9
Line 23646: 0000007994 [ncp.at] TRACE: < 23.60,A01.01
Line 23647: 0000007995 [ncp.at] TRACE: < OK
Line 23648: 0000007995 [ncp.client] TRACE: App firmware: 101
Line 23649: 0000007996 [ncp.at] TRACE: > AT+UGPIOC?
Line 23650: 0000008000 [ncp.at] TRACE: < +UGPIOC:
Line 23651: 0000008000 [ncp.at] TRACE: < 16,255
Line 23652: 0000008001 [ncp.at] TRACE: < 23,255
Line 23653: 0000008002 [ncp.at] TRACE: < 24,4
Line 23654: 0000008002 [ncp.at] TRACE: < 25,5
Line 23655: 0000008003 [ncp.at] TRACE: < 34,12
Line 23656: 0000008003 [ncp.at] TRACE: < 35,12
Line 23657: 0000008004 [ncp.at] TRACE: < 36,12
Line 23658: 0000008004 [ncp.at] TRACE: < 37,12
Line 23659: 0000008005 [ncp.at] TRACE: < 42,7
Line 23660: 0000008006 [ncp.at] TRACE: < OK
Line 23661: 0000008006 [ncp.client] INFO: Using internal SIM card
Line 23662: 0000008007 [ncp.at] TRACE: > AT+CPIN?
Line 23663: 0000008011 [ncp.at] TRACE: < +CPIN: READY
Line 23664: 0000008012 [ncp.at] TRACE: < OK
Line 23665: 0000008012 [ncp.at] TRACE: > AT+CCID
Line 23666: 0000008032 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 23667: 0000008032 [ncp.at] TRACE: < OK
Line 23668: 0000008033 [ncp.at] TRACE: > AT+IFC?
Line 23669: 0000008036 [ncp.at] TRACE: < +IFC: 2,2
Line 23670: 0000008037 [ncp.at] TRACE: < OK
Line 23671: 0000008037 [ncp.at] TRACE: > AT+IFC=2,2
Line 23672: 0000008040 [ncp.at] TRACE: < OK
Line 23673: 0000008041 [ncp.at] TRACE: > AT
Line 23697: /=0000002845 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
Line 23698: 0000002846 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
Line 23699: 0000002847 [ncp.client] TRACE: Powering modem on, ncpId: 0x42
Line 23700: 0000002847 [ncp.client] TRACE: Modem already on
Line 23701: 0000002847 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 23703: ========0000002849 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
Line 23733: 0000002948 [ncp.client] TRACE: Setting UART voltage translator state 0
Line 23752: 0000003048 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 23753: 0000003048 [ncp.client] TRACE: Muxer is not currently running
Line 23754: 0000003050 [ncp.client] TRACE: Initialized muxer @ 921600 baudrate
Line 23755: 0000003050 [mux] INFO: Starting GSM07.10 muxer
Line 23756: 0000003051 [mux] INFO: GSM07.10 muxer thread started
Line 23790: 0000003351 [mux] INFO: Stopping GSM07.10 muxer
Line 23791: 0000003351 [mux] INFO: Gracefully stopping GSM07.10 muxer
Line 23792: 0000003352 [mux] INFO: Closing all muxed channels
Line 23793: 0000003352 [mux] INFO: Muxed channel 1 already closed
Line 23794: 0000003352 [mux] INFO: Muxed channel 2 already closed
Line 23795: 0000003353 [mux] INFO: Muxed channel 3 already closed
Line 23796: 0000003354 [mux] INFO: Muxed channel 4 already closed
Line 23797: 0000004004 [mux] INFO: Sending CLD (multiplexer close down)
Line 23798: 0000004905 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
Line 23799: -/|\-0000006960 [ncp.client] TRACE: Modem is not responsive @ 921600 baudrate
Line 23800: 0000006961 [ncp.at] TRACE: > AT
Line 23801: 0000006963 [ncp.at] TRACE: < OK
Line 23802: 0000006963 [ncp.client] TRACE: NCP ready to accept AT commands
Line 23803: 0000006964 [ncp.at] TRACE: > AT
Line 23804: 0000006966 [ncp.at] TRACE: < OK
Line 23805: 0000006966 [ncp.at] TRACE: > ATI9
Line 23806: /0000006980 [ncp.at] TRACE: < 23.60,A01.01
Line 23807: 0000006981 [ncp.at] TRACE: < OK
Line 23808: 0000006981 [ncp.client] TRACE: App firmware: 101
Line 23809: 0000006981 [ncp.at] TRACE: > AT+UGPIOC?
Line 23810: 0000006985 [ncp.at] TRACE: < +UGPIOC:
Line 23811: 0000006986 [ncp.at] TRACE: < 16,255
Line 23812: 0000006987 [ncp.at] TRACE: < 23,255
Line 23813: 0000006987 [ncp.at] TRACE: < 24,4
Line 23814: 0000006988 [ncp.at] TRACE: < 25,5
Line 23815: 0000006988 [ncp.at] TRACE: < 34,12
Line 23816: 0000006989 [ncp.at] TRACE: < 35,12
Line 23817: 0000006989 [ncp.at] TRACE: < 36,12
Line 23818: 0000006990 [ncp.at] TRACE: < 37,12
Line 23819: 0000006991 [ncp.at] TRACE: < 42,7
Line 23820: 0000006991 [ncp.at] TRACE: < OK
Line 23821: 0000006992 [ncp.client] INFO: Using internal SIM card
Line 23822: 0000006992 [ncp.at] TRACE: > AT+CPIN?
Line 23823: 0000006996 [ncp.at] TRACE: < +CPIN: READY
Line 23824: 0000006997 [ncp.at] TRACE: < OK
Line 23825: 0000006997 [ncp.at] TRACE: > AT+CCID
Line 23826: 0000007018 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 23827: 0000007019 [ncp.at] TRACE: < OK
Line 23828: 0000007019 [ncp.at] TRACE: > AT+IFC?
Line 23829: 0000007022 [ncp.at] TRACE: < +IFC: 2,2
Line 23830: 0000007023 [ncp.at] TRACE: < OK
Line 23831: 0000007023 [ncp.at] TRACE: > AT+IFC=2,2
Line 23832: 0000007027 [ncp.at] TRACE: < OK
Line 23833: 0000007027 [ncp.at] TRACE: > AT
Line 23889: -/|\0000008027 [ncp.at] TRACE: > AT
Line 23890: 0000008029 [ncp.at] TRACE: < OK
Line 23891: 0000008030 [ncp.at] TRACE: > AT+IPR=921600
Line 23892: 0000008034 [ncp.at] TRACE: < OK
Line 23893: 0000008035 [ncp.at] TRACE: > AT
Line 23894: 0000008036 [ncp.at] TRACE: < 
Line 23895: 0000008037 [ncp.at] TRACE: < OK
Line 23896: -/|\-/|\0000009036 [ncp.at] TRACE: > AT
Line 23897: 0000009038 [ncp.at] TRACE: < OK
Line 23898: 0000009038 [ncp.at] TRACE: > AT+COPS=3,2
Line 23899: 0000009040 [ncp.at] TRACE: < OK
Line 23900: 0000009040 [ncp.at] TRACE: > AT+CGEREP=1,0
Line 23901: 0000009043 [ncp.at] TRACE: < OK
Line 23902: 0000009043 [ncp.at] TRACE: > AT+UPSV?
Line 23903: 0000009045 [ncp.at] TRACE: < +UPSV: 0
Line 23904: 0000009045 [ncp.at] TRACE: < OK
Line 23905: 0000009045 [ncp.at] TRACE: > AT+CMUX=0,0,,1509,,,,,
Line 23906: -0000009178 [ncp.at] TRACE: < OK
Line 23907: 0000009178 [mux] INFO: Starting GSM07.10 muxer
Line 23908: 0000009179 [mux] INFO: Opening mux channel 0
Line 23909: 0000009180 [mux] INFO: GSM07.10 muxer thread started
Line 23910: 0000009182 [mux] INFO: Opening mux channel 1
Line 23911: 0000009185 [ncp.at] TRACE: > AT
Line 23912: /0000009188 [ncp.at] TRACE: < OK
Line 23913: 0000009188 [ncp.client] TRACE: NCP state changed: 1
Line 23914: 0000009188 [net.pppncp] TRACE: NCP event 1
Line 23915: 0000009189 [net.ppp.client] TRACE: PPP thread event ADM_UP data=0
Line 23916: 0000009190 [net.ppp.client] TRACE: State NONE -> READY
Line 23917: 0000009193 [ncp.at] TRACE: > AT+CFUN?
Line 23918: 0000009196 [ncp.at] TRACE: < +CFUN: 1,0
Line 23919: 0000009196 [ncp.at] TRACE: < OK
Line 23920: 0000009197 [ncp.at] TRACE: > AT+CCID
Line 23921: 0000009214 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 23922: 0000009214 [ncp.at] TRACE: < OK
Line 23923: 0000009215 [ncp.at] TRACE: > AT+CIMI
Line 23924: 0000009218 [ncp.at] TRACE: < 204047127292930
Line 23925: 0000009218 [ncp.at] TRACE: < OK
Line 23926: 0000009220 [ncp.at] TRACE: > AT+CGDCONT?
Line 23927: 0000009223 [ncp.at] TRACE: < +CGDCONT: 1,"IP","vfd1.korem2m.com","0.0.0.0",0,0
Line 23928: 0000009224 [ncp.at] TRACE: < OK
Line 23929: 0000009225 [ncp.at] TRACE: > AT+CFUN?
Line 23930: 0000009228 [ncp.at] TRACE: < +CFUN: 1,0
Line 23931: 0000009228 [ncp.at] TRACE: < OK
Line 23932: 0000009229 [ncp.at] TRACE: > AT+CREG=2
Line 23933: 0000009231 [ncp.at] TRACE: < OK
Line 23934: 0000009232 [ncp.at] TRACE: > AT+CGREG=2
Line 23935: 0000009235 [ncp.at] TRACE: < OK
Line 23936: 0000009235 [ncp.client] TRACE: NCP connection state changed: 1
Line 23937: 0000009235 [net.pppncp] TRACE: NCP event 2
Line 23938: 0000009236 [net.pppncp] TRACE: State changed event: 1
Line 23939: 0000009236 [net.ppp.client] TRACE: PPP thread event LOWER_DOWN data=0
Line 23940: 0000009237 [ncp.at] TRACE: > AT+COPS?
Line 23941: 0000009240 [ncp.at] TRACE: < +COPS: 0
Line 23942: 0000009240 [ncp.at] TRACE: < OK
Line 23943: 0000009243 [ncp.at] TRACE: > AT+CREG?
Line 23944: 0000009245 [ncp.at] TRACE: < +CREG: 2,0
Line 23945: 0000009246 [ncp.at] TRACE: < OK
Line 23946: 0000009247 [ncp.at] TRACE: > AT+CGREG?
Line 23947: 0000009250 [ncp.at] TRACE: < +CGREG: 2,0
Line 23948: 0000009250 [ncp.at] TRACE: < OK
Line 24004: |\-0000003968 [mux] INFO: Sending CLD (multiplexer close down)
Line 24005: 0000004869 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
Line 24006: 0000004919 [mux] INFO: GSM07.10 muxer thread exiting
Line 24007: 0000004919 [mux] INFO: GSM07.10 muxer stopped
Line 24008: 0000004920 [ncp.at] TRACE: > AT
Line 24120: \-/0000005920 [ncp.at] TRACE: > AT
Line 24121: |\-/|\-/0000006920 [ncp.client] TRACE: Modem is not responsive @ 921600 baudrate
Line 24122: 0000006921 [ncp.at] TRACE: > AT
Line 24123: 0000006923 [ncp.at] TRACE: < OK
Line 24124: 0000006924 [ncp.client] TRACE: NCP ready to accept AT commands
Line 24125: 0000006924 [ncp.at] TRACE: > AT
Line 24126: 0000006926 [ncp.at] TRACE: < OK
Line 24127: 0000006927 [ncp.at] TRACE: > ATI9
Line 24128: 0000006941 [ncp.at] TRACE: < 23.60,A01.01
Line 24129: 0000006941 [ncp.at] TRACE: < OK
Line 24130: 0000006941 [ncp.client] TRACE: App firmware: 101
Line 24131: 0000006942 [ncp.at] TRACE: > AT+UGPIOC?
Line 24132: 0000006946 [ncp.at] TRACE: < +UGPIOC:
Line 24133: 0000006946 [ncp.at] TRACE: < 16,255
Line 24134: 0000006947 [ncp.at] TRACE: < 23,255
Line 24135: 0000006948 [ncp.at] TRACE: < 24,4
Line 24136: 0000006948 [ncp.at] TRACE: < 25,5
Line 24137: 0000006949 [ncp.at] TRACE: < 34,12
Line 24138: 0000006949 [ncp.at] TRACE: < 35,12
Line 24139: 0000006950 [ncp.at] TRACE: < 36,12
Line 24140: 0000006951 [ncp.at] TRACE: < 37,12
Line 24141: 0000006951 [ncp.at] TRACE: < 42,7
Line 24142: 0000006952 [ncp.at] TRACE: < OK
Line 24143: 0000006952 [ncp.client] INFO: Using internal SIM card
Line 24144: 0000006952 [ncp.at] TRACE: > AT+CPIN?
Line 24145: 0000006957 [ncp.at] TRACE: < +CPIN: READY
Line 24146: 0000006957 [ncp.at] TRACE: < OK
Line 24147: 0000006958 [ncp.at] TRACE: > AT+CCID
Line 24148: 0000006977 [ncp.at] TRACE: < +CCID: 89314404000821550960
Line 24149: 0000006978 [ncp.at] TRACE: < OK
Line 24150: 0000006978 [ncp.at] TRACE: > AT+IFC?
Line 24151: |0000006982 [ncp.at] TRACE: < +IFC: 2,2
Line 24152: 0000006982 [ncp.at] TRACE: < OK
Line 24153: 0000006983 [ncp.at] TRACE: > AT+IFC=2,2
Line 24154: 0000006986 [ncp.at] TRACE: < OK
Line 24155: 0000006986 [ncp.at] TRACE: > AT
Line 24211: -/|\-0000007987 [ncp.at] TRACE: > AT
Line 24212: 0000007989 [ncp.at] TRACE: < OK
Line 24213: 0000007990 [ncp.at] TRACE: > AT+IPR=921600
Line 24214: 0000007994 [ncp.at] TRACE: < OK
Line 24215: 0000007995 [ncp.at] TRACE: > AT
Line 24216: 0000007996 [ncp.at] TRACE: < 
Line 24217: 0000007997 [ncp.at] TRACE: < OK
Line 24245: 0000002845 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
Line 24253: 0000002852 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
Line 24254: 0000002855 [ncp.client] TRACE: Powering modem on, ncpId: 0x42
Line 24255: 0000002855 [ncp.client] TRACE: Modem already on
Line 24256: 0000002856 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 24259: 0000002857 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
Line 24295: 0000002956 [ncp.client] TRACE: Setting UART voltage translator state 0
Line 24296: 0000003056 [ncp.client] TRACE: Setting UART voltage translator state 1
Line 24297: 0000003057 [ncp.client] TRACE: Muxer is not currently running
Line 24298: 0000003058 [ncp.client] TRACE: Initialized muxer @ 921600 baudrate
Line 24299: 0000003058 [mux] INFO: Starting GSM07.10 muxer
Line 24300: 0000003059 [mux] INFO: GSM07.10 muxer thread started
Line 24350: 0000003358 [mux] INFO: Stopping GSM07.10 muxer
Line 24351: 0000003359 [mux] INFO: Gracefully stopping GSM07.10 muxer
Line 24352: 0000003359 [mux] INFO: Closing all muxed channels
Line 24353: 0000003359 [mux] INFO: Muxed channel 1 already closed
Line 24354: 0000003360 [mux] INFO: Muxed channel 2 already closed
Line 24355: 0000003360 [mux] INFO: Muxed channel 3 already closed
Line 24356: 0000003361 [mux] INFO: Muxed channel 4 already closed
Line 24403: |\-/0000004011 [mux] INFO: Sending CLD (multiplexer close down)
Line 24412: 0000004912 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
Line 24417: 0000004963 [mux] INFO: GSM07.10 muxer thread exiting
Line 24418: 0000004964 [mux] INFO: GSM07.10 muxer stopped
Line 24419: 0000004964 [ncp.at] TRACE: > AT
Line 24463: /|\-/|\0000005965 [ncp.at] TRACE: > AT
Line 24476: 0000006965 [ncp.client] TRACE: Modem is not responsive @ 921600 baudrate

What can you do with a drunken Boron?

2

Hi @jaza_tom!

I'm Alberto, from the TAM team.

There is a known issue with the SARA-R410 modem used on some of our products, including some of the Boron family. If the device is ungracefully powered off, the modem might be off at a bad moment and the internal flash might be corrupted and irrecoverable. That might be the source of your issue. This is especially true and risky when the device is constantly disconnected from power and without a backup battery. Newer products like B404X are unaffected as they use different modems.

We assembled this TAN with recommendations for preventing this.

3

I took a look at TAN004 and it doesn't appear that the error message specified is being emitted by the Boron in question:

ERROR: No response from NCP

and/or

modem is not yet responding

But it definitely seems like this might be something similar to what is happening with our mystery Boron.

In our application, sudden loss of power is definitely a possibility. I did not know there was a hardware risk for the Boron. Is there a way to sign up for notifications when a technical advisory notice gets posted?

Next steps for us

Backup power

It seems like we should be buying backup batteries for the devices that we have already deployed in the field...

We may have an option to keep the Boron powered when Vin is removed unexpectedly by having it plugged into an "always-on" USB cable. Would that work?

Then the general process is to detect that Vin is gone, and gracefully shut down the modem in application firmware, right?

Please advise.

What to do with Borons that succumb

What should we do with the Boron that has already been affected? Is there a recovery process? Is it a write-off. Is there a replacement process?

4

Update: this Boron appears to have come back online mysteriously....

5

Looking at the logs, the presence of +CGREG: 2,0 and +CGREG: 2,2 indicate the device was either not finding a tower, or finding a tower with no compatible carriers.

I see this device is located in my neck of the woods - it's possible the intensified loadshedding the last couple of days played a role.
It's signal is otherwise pretty great - so it must have been an issue with the tower it connects to.