Hi there!
Is anyone else impacted by today's (2/22/2024) AT&T (and others) outage? I am in the North Texas area and my devices are not connecting (green/blue flashing). The Particle dashboard says "all clear" but other outlets say otherwise.
We've seen recovery and devices have resumed normal connectivity. What are you seeing on Cloud Debug | Troubleshooting | Particle?
1 Like
CHRIS hahahah you rock!
1 Like
Currently I'm seeing:
Serial monitor opened successfully:
0000001317 [mux] INFO: Stopping GSM07.10 muxer
0000001317 [mux] INFO: Gracefully stopping GSM07.10 muxer
0000001317 [mux] INFO: Closing all muxed channels
0000001318 [mux] INFO: Muxed channel 1 already closed
0000001318 [mux] INFO: Muxed channel 2 already closed
0000001319 [mux] INFO: Muxed channel 3 already closed
0000001319 [mux] INFO: Muxed channel 4 already closed
0000001320 [mux] INFO: Sending CLD (multiplexer close down)
0000002220 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
0000002270 [mux] INFO: GSM07.10 muxer thread exiting
0000002270 [mux] INFO: GSM07.10 muxer stopped
0000002285 [ncp.client] INFO: Using internal SIM card
0000003395 [mux] INFO: Starting GSM07.10 muxer
0000003396 [mux] INFO: Opening mux channel 0
0000003397 [mux] INFO: GSM07.10 muxer thread started
0000003399 [mux] INFO: Opening mux channel 1
and it stops there. If you want me to continue and flash Cloud Doctor on one of the devices I can.
Alright, with the Cloud Doctor firmware installed it connects.
Serial monitor opened successfully:
0000001131 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000001132 [ncp.at] TRACE: < OK
0000001133 [ncp.at] TRACE: > AT+UMNOPROF?
0000001138 [ncp.at] TRACE: < +UMNOPROF: 100
0000001138 [ncp.at] TRACE: < OK
0000001139 [ncp.at] TRACE: > AT+UBANDMASK?
0000001144 [ncp.at] TRACE: < +UBANDMASK: 0,6170,0
0000001145 [ncp.at] TRACE: < OK
0000001145 [ncp.at] TRACE: > AT+COPS=3,2
0000001149 [ncp.at] TRACE: < OK
0000001150 [ncp.at] TRACE: > AT+CGEREP=1,0
0000001153 [ncp.at] TRACE: < OK
0000001154 [ncp.at] TRACE: > AT+URAT?
0000001157 [ncp.at] TRACE: < +URAT: 7
0000001158 [ncp.at] TRACE: < OK
0000001159 [ncp.at] TRACE: > AT+CPSMS?
0000001164 [ncp.at] TRACE: < +CPSMS: 0,,,"00010011","00000011"
0000001164 [ncp.at] TRACE: < OK
0000001165 [ncp.at] TRACE: > AT+CEDRXS?
0000001170 [ncp.at] TRACE: < +CEDRXS: 4,"0000"
0000001171 [ncp.at] TRACE: < OK
0000001171 [mux] INFO: Mux channel 1 already opened
0000001172 [ncp.at] TRACE: > AT
0000001176 [ncp.at] TRACE: < OK
0000001176 [ncp.client] TRACE: NCP state changed: 1
0000001177 [net.pppncp] TRACE: NCP event 1
0000001179 [app] INFO: cellular is on
0000001181 [ncp.at] TRACE: > AT+CCID
0000001206 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000001206 [ncp.at] TRACE: < OK
0000001208 [ncp.at] TRACE: > AT+CGMI
0000001212 [ncp.at] TRACE: < u-blox
0000001214 [ncp.at] TRACE: < OK
0000001216 [ncp.at] TRACE: > AT+CGMM
0000001219 [ncp.at] TRACE: < SARA-R510S
0000001221 [ncp.at] TRACE: < OK
0000001223 [ncp.at] TRACE: > AT+CGMR
0000001226 [ncp.at] TRACE: < 03.15
0000001227 [ncp.at] TRACE: < OK
0000001229 [ncp.at] TRACE: > AT+CGSN
0000001233 [ncp.at] TRACE: < 351457832089681
0000001233 [ncp.at] TRACE: < OK
> 0000010001 [app] INFO: Platform: Boron
0000010001 [app] INFO: Binary compiled for: 4.0.1
0000010002 [app] INFO: Troubleshooting firmware version 12
0000010003 [app] INFO: System version: 4.0.1
0000010004 [app] INFO: Device ID: e00fce68cc263c50da1df815
0000010005 [app] INFO: Connecting to the cloud
0000010089 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000010092 [net.ppp.client] TRACE: PPP thread event ADM_UP data=0
0000010092 [net.ppp.client] TRACE: State NONE -> READY
0000010095 [ncp.at] TRACE: > AT+CFUN?
0000010097 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000010099 [ncp.at] TRACE: < +CFUN: 1,7
0000010100 [ncp.at] TRACE: < OK
0000010101 [ncp.at] TRACE: > AT+CCID
0000010124 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000010125 [ncp.at] TRACE: < OK
0000010126 [ncp.at] TRACE: > AT+CGDCONT?
0000010132 [ncp.at] TRACE: < +CGDCONT: 1,"IP","super.mnc010.mcc234.gprs","100.70.20.148",0,0,0,2,0,0,0,0,0,0,0
0000010133 [ncp.at] TRACE: < OK
0000010134 [ncp.at] TRACE: > AT+CFUN?
0000010137 [ncp.at] TRACE: < +CFUN: 1,7
0000010138 [ncp.at] TRACE: < OK
0000010139 [ncp.at] TRACE: > AT+CEREG=2
0000010142 [ncp.at] TRACE: < OK
0000010142 [ncp.client] TRACE: NCP connection state changed: 1
0000010143 [net.pppncp] TRACE: NCP event 2
0000010144 [net.pppncp] TRACE: State changed event: 1
0000010145 [net.ppp.client] TRACE: PPP thread event LOWER_DOWN data=0
0000010145 [ncp.at] TRACE: > AT+COPS?
0000010150 [ncp.at] TRACE: < +COPS: 0,2,"310410",7
0000010150 [ncp.at] TRACE: < OK
0000010151 [ncp.at] TRACE: > AT+CEREG?
0000010156 [ncp.at] TRACE: < +CEREG: 2,5,"6a41","07f3a20f",7
0000010156 [ncp.at] TRACE: < OK
0000010157 [ncp.client] TRACE: NCP connection state changed: 2
0000010158 [mux] INFO: Openning mux channel 2
0000010491 [net.pppncp] TRACE: NCP event 100
0000010492 [net.pppncp] TRACE: New auth info
0000010495 [net.pppncp] TRACE: NCP event 2
0000010495 [net.pppncp] TRACE: State changed event: 2
0000010496 [ncp.at] TRACE: > AT+CIMI
0000010497 [net.ppp.client] TRACE: PPP thread event LOWER_UP data=0
0000010497 [net.ppp.client] TRACE: State READY -> CONNECT
0000010498 [net.ppp.client] TRACE: State CONNECT -> CONNECTING
0000010544 [ncp.at] TRACE: < 234104869428767
0000010545 [ncp.at] TRACE: < OK
0000010546 [ncp.at] TRACE: > AT+CGATT?
0000010549 [ncp.at] TRACE: < +CGATT: 1
0000010550 [ncp.at] TRACE: < OK
0000011551 [ncp.at] TRACE: > AT
0000011554 [ncp.at] TRACE: < OK
0000011555 [ncp.at] TRACE: > AT+UPORTFWD=100,1
0000011559 [ncp.at] TRACE: < +UPORTFWD: 100,1
0000011559 [ncp.at] TRACE: < OK
0000011560 [ncp.at] TRACE: > ATD*99***1#
0000011566 [ncp.at] TRACE: < CONNECT
0000011566 [net.ppp.client] TRACE: PPP phase -> Initialize
0000011568 [net.ppp.client] TRACE: PPP phase -> Establish
0000011568 [lwip.ppp] TRACE: sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x5852213f> <pcomp> <accomp>]
0000011571 [lwip.ppp] TRACE: rcvd [LCP ConfReq id=0x0 <asyncmap 0x0> <mru 1360> <auth pap> <accomp> <pcomp>]
0000011572 [lwip.ppp] TRACE: sent [LCP ConfAck id=0x0 <asyncmap 0x0> <mru 1360> <auth pap> <accomp> <pcomp>]
0000011583 [lwip.ppp] TRACE: rcvd [LCP ConfAck id=0x1 <asyncmap 0x0> <magic 0x5852213f> <pcomp> <accomp>]
0000011584 [lwip.ppp] TRACE: sent [LCP EchoReq id=0x0 magic=0x5852213f]
0000011585 [net.ppp.client] TRACE: PPP phase -> Authenticate
0000011586 [lwip.ppp] TRACE: sent [PAP AuthReq id=0x1 user="" password=""]
0000011595 [lwip.ppp] TRACE: rcvd [LCP EchoRep id=0x0 magic=0x0]
0000011598 [lwip.ppp] TRACE: rcvd [PAP AuthAck id=0x1 "Login successful."]
0000011598 [net.ppp.client] TRACE: PPP phase -> Network
0000011600 [lwip.ppp] TRACE: sent [IPCP ConfReq id=0x1 <addr 0.0.0.0> <msdns1 8.8.8.8> <msdns2 8.8.4.4>]
0000011602 [lwip.ppp] TRACE: rcvd [IPCP ConfReq id=0x0 <addr 192.168.201.2>]
0000011603 [lwip.ppp] TRACE: sent [IPCP ConfNak id=0x0 <addr 192.168.201.2>]
0000011611 [lwip.ppp] TRACE: rcvd [IPCP ConfNak id=0x1 <addr 192.168.201.1> <msdns1 8.8.4.4> <msdns2 8.8.8.8>]
0000011613 [lwip.ppp] TRACE: sent [IPCP ConfReq id=0x2 <addr 192.168.201.1> <msdns1 8.8.4.4> <msdns2 8.8.8.8>]
0000011615 [lwip.ppp] TRACE: rcvd [IPCP ConfReq id=0x1 <addr 192.168.201.2>]
0000011616 [lwip.ppp] TRACE: sent [IPCP ConfAck id=0x1 <addr 192.168.201.2>]
0000011625 [lwip.ppp] TRACE: rcvd [IPCP ConfAck id=0x2 <addr 192.168.201.1> <msdns1 8.8.4.4> <msdns2 8.8.8.8>]
0000011626 [system.nm] INFO: State changed: IFACE_UP -> IFACE_LINK_UP
0000011628 [net.ppp.client] TRACE: PPP thread event UP data=0
0000011628 [net.ppp.client] TRACE: State CONNECTING -> CONNECTED
0000011629 [net.pppncp] TRACE: Negotiated MTU: 1360
0000011628 [system.nm] INFO: State changed: IFACE_LINK_UP -> IP_CONFIGURED
0000011631 [net.ppp.client] TRACE: PPP phase -> Running
0000011633 [system] INFO: Cloud: connecting
0000011638 [system] WARN: Failed to load session data from persistent storage
0000012173 [system] INFO: Cloud socket connected
0000012173 [comm.protocol.handshake] INFO: Establish secure connection
0000012185 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=2
0000013021 [comm.protocol.handshake] INFO: Sending HELLO message
0000013176 [comm.protocol.handshake] INFO: Handshake completed
0000013176 [comm.protocol.handshake] TRACE: Updating cached session parameters
0000013178 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000013180 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000013566 [ncp.at] TRACE: > AT+CFUN?
0000013569 [ncp.at] TRACE: < +CFUN: 1,7
0000013570 [ncp.at] TRACE: < OK
0000013571 [ncp.at] TRACE: > AT+CCID
0000013615 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000013616 [ncp.at] TRACE: < OK
0000013616 [ncp.at] TRACE: > AT+CGSN
0000013621 [ncp.at] TRACE: < 351457832089681
0000013621 [ncp.at] TRACE: < OK
0000013622 [ncp.at] TRACE: > ATI9
0000013626 [ncp.at] TRACE: < 03.15,A00.01
0000013627 [ncp.at] TRACE: < OK
0000013650 [comm.protocol] INFO: Sending subscriptions
0000013831 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000013831 [comm.protocol] TRACE: message id 5 complete with code 0.00
0000013833 [comm.protocol] TRACE: rcv'd message type=13
0000013842 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000013842 [comm.protocol] TRACE: message id 2 complete with code 0.00
0000013844 [comm.protocol] TRACE: Updating system DESCRIBE checksum
0000013845 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000014240 [ncp.at] TRACE: > AT+CFUN?
0000014244 [ncp.at] TRACE: < +CFUN: 1,7
0000014246 [ncp.at] TRACE: < OK
0000014248 [ncp.at] TRACE: > AT+CCID
0000014274 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000014276 [ncp.at] TRACE: < OK
0000014278 [ncp.at] TRACE: > AT+CGSN
0000014282 [ncp.at] TRACE: < 351457832089681
0000014282 [ncp.at] TRACE: < OK
0000014284 [ncp.at] TRACE: > ATI9
0000014287 [ncp.at] TRACE: < 03.15,A00.01
0000014289 [ncp.at] TRACE: < OK
0000014289 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000014291 [comm.protocol] TRACE: rcv'd message type=13
0000014293 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000014295 [comm.protocol] TRACE: message id 3 complete with code 0.00
0000014295 [comm.protocol] TRACE: rcv'd message type=13
0000014397 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000014397 [comm.protocol] TRACE: message id 4 complete with code 0.00
0000014399 [comm.protocol] TRACE: rcv'd message type=13
0000014501 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000014501 [comm.protocol] TRACE: message id 7 complete with code 0.00
0000014503 [comm.protocol] TRACE: Updating application DESCRIBE checksum
0000014505 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000014507 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000014507 [comm.protocol] TRACE: rcv'd message type=13
0000014609 [comm.protocol] TRACE: Reply recieved: type=2, code=69
0000014609 [comm.protocol] TRACE: message id 6 complete with code 2.05
0000014611 [comm.protocol] INFO: Received TIME response: 1708622604
0000014613 [comm.protocol] TRACE: rcv'd message type=12
0000014713 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000014713 [comm.protocol] TRACE: message id 8 complete with code 0.00
0000014715 [comm.protocol] TRACE: rcv'd message type=13
0000014816 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000014816 [comm.protocol] TRACE: message id 9 complete with code 0.00
0000014818 [comm.protocol] TRACE: Updating subscriptions checksum
0000014820 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000014822 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000014822 [comm.protocol] TRACE: rcv'd message type=13
0000014924 [system] INFO: Cloud connected
0000014925 [ncp.at] TRACE: > AT+COPS=3,2
0000014929 [ncp.at] TRACE: < OK
0000014930 [ncp.at] TRACE: > AT+COPS?
0000014934 [ncp.at] TRACE: < +COPS: 0,2,"310410",7
0000014934 [ncp.at] TRACE: < OK
0000014935 [ncp.at] TRACE: > AT+CEREG?
0000014940 [ncp.at] TRACE: < +CEREG: 2,5,"6a41","07f3a20f",7
0000014941 [ncp.at] TRACE: < OK
0000014941 [app] INFO: Cellular Info: cid=133407247 lac=27201 mcc=310 mnc=410
0000014943 [ncp.at] TRACE: > AT+CFUN?
0000014946 [ncp.at] TRACE: < +CFUN: 1,7
0000014947 [ncp.at] TRACE: < OK
0000014948 [ncp.at] TRACE: > AT+CCID
0000014971 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000014972 [ncp.at] TRACE: < OK
0000014973 [ncp.at] TRACE: > AT+CGSN
0000014977 [ncp.at] TRACE: < 351457832089681
0000014977 [ncp.at] TRACE: < OK
0000014978 [ncp.at] TRACE: > ATI9
0000014981 [ncp.at] TRACE: < 03.15,A00.01
0000014982 [ncp.at] TRACE: < OK
0000014983 [ncp.at] TRACE: > AT+CGED=3
0000014986 [ncp.at] TRACE: < ERROR
0000015000 [ncp.at] TRACE: > AT+COPS=3,2
0000015003 [ncp.at] TRACE: < OK
0000015004 [ncp.at] TRACE: > AT+COPS?
0000015008 [ncp.at] TRACE: < +COPS: 0,2,"310410",7
0000015008 [ncp.at] TRACE: < OK
0000015009 [ncp.at] TRACE: > AT+CEREG?
0000015014 [ncp.at] TRACE: < +CEREG: 2,5,"6a41","07f3a20f",7
0000015015 [ncp.at] TRACE: < OK
0000015016 [ncp.at] TRACE: > AT+CFUN?
0000015020 [ncp.at] TRACE: < +CFUN: 1,7
0000015021 [ncp.at] TRACE: < OK
0000015022 [ncp.at] TRACE: > AT+CCID
0000015045 [ncp.at] TRACE: < +CCID: 89883070000037330709
0000015046 [ncp.at] TRACE: < OK
0000015047 [ncp.at] TRACE: > AT+CGSN
0000015051 [ncp.at] TRACE: < 351457832089681
0000015051 [ncp.at] TRACE: < OK
0000015052 [ncp.at] TRACE: > ATI9
0000015055 [ncp.at] TRACE: < 03.15,A00.01
0000015056 [ncp.at] TRACE: < OK
0000015057 [ncp.at] TRACE: > AT+CGED=3
0000015060 [ncp.at] TRACE: < ERROR
0000015060 [app] INFO: Cellular Info: cid=133407247 lac=27201 mcc=310 mnc=410
0000015122 [comm.protocol] TRACE: rcv'd message type=8
0000015192 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x04
0000015196 [ncp.at] TRACE: > AT+COPS=3,2
0000015200 [ncp.at] TRACE: < OK
0000015202 [ncp.at] TRACE: > AT+COPS?
0000015207 [ncp.at] TRACE: < +COPS: 0,2,"310410",7
0000015209 [ncp.at] TRACE: < OK
0000015209 [ncp.at] TRACE: > AT+UCGED?
0000015697 [ncp.at] TRACE: < +UCGED: 2
0000015698 [ncp.at] TRACE: < 6,4,310,410
0000015699 [ncp.at] TRACE: < 5110,12,50,50,6a41,7f3a20f,198,d5bdabc1,ff30,b1,50,16,0.93,3,255,255,49,4,-15,300,255,255,80
0000015700 [ncp.at] TRACE: < OK
0000015703 [ncp.at] TRACE: > AT+COPS=3,2
0000015705 [ncp.at] TRACE: < OK
0000015707 [ncp.at] TRACE: > AT+COPS?
0000015711 [ncp.at] TRACE: < +COPS: 0,2,"310410",7
0000015711 [ncp.at] TRACE: < OK
0000015713 [ncp.at] TRACE: > AT+CEREG?
0000015718 [ncp.at] TRACE: < +CEREG: 2,5,"6a41","07f3a20f",7
0000015720 [ncp.at] TRACE: < OK
0000015724 [comm.protocol] TRACE: rcv'd message type=1
0000015897 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000015897 [comm.protocol] TRACE: message id 10 complete with code 0.00
0000015899 [comm.protocol] TRACE: rcv'd message type=13
I'm going to go back and flash my firmware. This scares me a little, though, if I have to do this to get connectivity - that shouldn't be the case I'm sure!
Okay, something is not right - I went back and recompiled and flashed my firmware. It is not connecting.
I don't know if I was interpreting cloud doctors output correctly, but its definitely hung on trying to connect.
Serial monitor opened successfully:
0000001497 [app] INFO: loop
0000001900 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
0000001950 [mux] INFO: GSM07.10 muxer thread exiting
0000001950 [mux] INFO: GSM07.10 muxer stopped
0000001951 [mux] INFO: Starting GSM07.10 muxer
0000001952 [mux] INFO: GSM07.10 muxer thread started
0000002251 [mux] INFO: Stopping GSM07.10 muxer
0000002251 [mux] INFO: Gracefully stopping GSM07.10 muxer
0000002252 [mux] INFO: Closing all muxed channels
0000002252 [mux] INFO: Muxed channel 1 already closed
0000002252 [mux] INFO: Muxed channel 2 already closed
0000002253 [mux] INFO: Muxed channel 3 already closed
0000002253 [mux] INFO: Muxed channel 4 already closed
0000002254 [mux] INFO: Sending CLD (multiplexer close down)
0000002497 [app] INFO: loop
0000003155 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
0000003205 [mux] INFO: GSM07.10 muxer thread exiting
0000003205 [mux] INFO: GSM07.10 muxer stopped
0000003498 [app] INFO: loop
0000004498 [app] INFO: loop
0000005498 [app] INFO: loop
0000006499 [app] INFO: loop
0000007499 [app] INFO: loop
0000008499 [app] INFO: loop
0000009499 [app] INFO: loop
0000010207 [ncp.client] ERROR: No response from NCP
0000010407 [net.pppncp] ERROR: Failed to initialize cellular NCP client: -210
I am using a Boron 404X with 5.7.0.
12PM Central update - devices are beginning to connect on standard firmware. I think there's lingering issues with the AT&T network at least in North Texas. I'll report if things go south again.
5PM Central update - looks like the national "outage" for AT&T has ended, and my devices are connecting. Events like these always make me revisit SLAs and business continuity plans!
2 Likes
Glad to hear things are back to normal.
While it happens - it's fortunately rare that the carriers drop the ball like they did yesterday.