Electron can't reach Particle Device Cloud

Our Electron has been running successfully in the field for some time, then all of a sudden, it stopped connecting to the particle cloud. It will constantly blink cyan, then have a burst of three orange blinks. According to the documentation, this is “Connected to the internet, but could not reach the Particle Device Cloud. This sometimes is seen as yellow or red and indicates bad server keys.” I would like to see if someone can help me out identify the root of the problem?

Couple of things we have tried so far with no success:

  1. Added LIPO Battery for stable power to board
  2. Swapping the antenna out
  3. Re-flashing firmware as well as tinker
  4. Upgrading to most recent firmware (1.0.1 and 1.1.0)
  5. Resetting the server and device keys using:
    a. particle keys server (succeeded)
    b. particle keys doctor YOUR_DEVICE_ID (succeeded)
  6. particle device doctor (Resetting EEPROM failed, others passed)
  7. Ran rickkas7/electron-clouddebug and got the following output so someone can help my identify what is going on.
Serial monitor opened successfully:
clouddebug: press letter corresponding to the command
a - enter APN for 3rd-party SIM card
k - set keep-alive value
c - show carriers at this location
t - run normal tests (occurs automatically after 10 seconds)
or tap the MODE button once to show carriers
starting tests...
turning cellular on...
deviceID=38005d000d50483553353920
manufacturer=u-blox
model=SARA-U260
firmware version=23.20
ordering code=SARA-U260-00S-02
IMEI=352580085739683
IMSI=u-blox
ICCID=8934072500004338939
0000019760 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000019760 [system] INFO: Sim Ready
0000019760 [system] INFO: Sim Ready
0000019761 [system] INFO: ARM_WLAN_WD 1
0000019761 [system] INFO: ARM_WLAN_WD 1

[ Modem::register ] = = = = = = = = = = = = = =
    19.761 AT send      12 "AT+CGREG=2\r\n"
    19.766 AT read OK    6 "\r\nOK\r\n"
    19.766 AT send      11 "AT+CREG=2\r\n"
    19.771 AT read OK    6 "\r\nOK\r\n"
    19.771 AT send      10 "AT+CREG?\r\n"
    19.778 AT read  +   33 "\r\n+CREG: 2,5,\"9841\",\"27A2E1A\",6\r\n"
    19.779 AT read OK    6 "\r\nOK\r\n"
    19.779 AT send      11 "AT+CGREG?\r\n"
    19.787 AT read  +   39 "\r\n+CGREG: 2,5,\"9841\",\"27A2E1A\",6,\"01\"\r\n"
    19.789 AT read OK    6 "\r\nOK\r\n"
    19.789 AT send       4 "AT\r\n"
    19.792 AT read OK    6 "\r\nOK\r\n"
    19.792 AT send      10 "AT+COPS?\r\n"
    19.837 AT read  +   35 "\r\n+COPS: 0,0,\"0041005400260054\",2\r\n"
    19.839 AT read OK    6 "\r\nOK\r\n"
    19.839 AT send       8 "AT+CSQ\r\n"
    19.844 AT read  +   13 "\r\n+CSQ: 6,3\r\n"
    19.845 AT read OK    6 "\r\nOK\r\n"

[ Modem::join ] = = = = = = = = = = = = = = = =
    19.845 AT send      12 "AT+CGATT=1\r\n"
    19.850 AT read OK    6 "\r\nOK\r\n"
    19.850 AT send      14 "AT+UPSND=0,8\r\n"
    19.857 AT read  +   17 "\r\n+UPSND: 0,8,0\r\n"
    19.858 AT read OK    6 "\r\nOK\r\n"
    19.858 AT send      23 "AT+UPSD=0,7,\"0.0.0.0\"\r\n"
    19.865 AT read OK    6 "\r\nOK\r\n"
    19.865 AT send      36 "AT+UPSD=0,1,\"spark.telefonica.com\"\r\n"
    19.873 AT read OK    6 "\r\nOK\r\n"
    19.873 AT send      15 "AT+UPSD=0,6,0\r\n"
    19.879 AT read OK    6 "\r\nOK\r\n"
    19.879 AT send      14 "AT+UPSDA=0,3\r\n"
    22.813 AT read OK    6 "\r\nOK\r\n"
    22.813 AT send      14 "AT+UPSND=0,0\r\n"
    22.820 AT read  +   31 "\r\n+UPSND: 0,0,\"10.44.208.236\"\r\n"
    22.821 AT read OK    6 "\r\nOK\r\n"
    22.821 AT send      14 "AT+UPSND=0,1\r\n"
    22.828 AT read  +   30 "\r\n+UPSND: 0,1,\"80.58.61.250\"\r\n"
    22.829 AT read OK    6 "\r\nOK\r\n"
    22.829 AT send      14 "AT+UPSND=0,2\r\n"
    22.836 AT read  +   30 "\r\n+UPSND: 0,2,\"80.58.61.254\"\r\n"
    22.837 AT read OK    6 "\r\nOK\r\n"
0000022838 [system] INFO: ARM_WLAN_WD 2
0000022838 [system] INFO: ARM_WLAN_WD 2
0000022838 [system] INFO: CLR_WLAN_WD 1, DHCP success
0000022838 [system] INFO: CLR_WLAN_WD 1, DHCP success
connected to the cellular network in 0 milliseconds
connected to cellular network!
    22.839 AT send      12 "AT+UDOPN=9\r\n"
    22.880 AT read  +   32 "\r\n+UDOPN: 6,\"0041005400260054\"\r\n"
    22.881 AT read OK    6 "\r\nOK\r\n"
operator name=0041005400260054
    22.881 AT send       8 "AT+CSQ\r\n"
    22.886 AT read  +   13 "\r\n+CSQ: 6,4\r\n"
    22.887 AT read OK    6 "\r\nOK\r\n"
rssi=-101, qual=4
    22.887 AT send      11 "AT+CGED=5\r\n"
    22.895 AT read ERR  39 "\r\n+CME ERROR: operation not supported\r\n"
    22.895 AT send      11 "AT+CGED=3\r\n"
    22.904 AT read  +   22 "\r\n+CGED: RAT:\"UMTS\",\r\n"
    22.910 AT read UNK  62 "\r\nMCC:310, MNC:410, LAC:9841, CI:27a2e1a, DLF:4384, ULF:4159\r\n"
    22.912 AT read OK    6 "\r\nOK\r\n"
0000022913 [app] INFO: service rat=UMTS mcc=310, mnc=410, lac=9841 ci=27a2e1a band=UMTS 850 rssi=0 dlf=4384 ulf=4159
0000022913 [app] INFO: service rat=UMTS mcc=310, mnc=410, lac=9841 ci=27a2e1a band=UMTS 850 rssi=0 dlf=4384 ulf=4159
    22.913 AT send      20 "AT+UPING=\"8.8.8.8\"\r\n"
    22.919 AT read OK    6 "\r\nOK\r\n"
ping addr 8.8.8.8=1
    22.919 AT send      31 "AT+UDNSRN=0,\"device.spark.io\"\r\n"
device.spark.io=0.0.0.0
connecting to cloud
0000032922 [system] INFO: Cloud: connecting
0000032922 [system] INFO: Cloud: connecting
0000032924 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
0000032924 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
0000032925 [system] ERROR: Failed to load session data from persistent storage
0000032925 [system] ERROR: Failed to load session data from persistent storage
0000032926 [system] INFO: Discarding session data
0000032926 [system] INFO: Discarding session data
0000032927 [system] TRACE: Resolving 38005d000d50483553353920.udp.particle.io
    32.926 AT send      56 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r\n"
    32.941 AT read UNK  56 "\x00AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r"
    32.943 AT read ERR   9 "\r\nERROR\r\n"
    32.944 AT send      56 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r\n"
    32.959 AT read UNK  55 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r"
    32.961 AT read ERR   9 "\r\nERROR\r\n"
    32.962 AT send      56 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r\n"
    32.977 AT read UNK  55 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r"
    32.979 AT read ERR   9 "\r\nERROR\r\n"
0000032981 [system] ERROR: Unable to resolve IP for 38005d000d50483553353920.udp.particle.io (1)
0000032981 [system] ERROR: Unable to resolve IP for 38005d000d50483553353920.udp.particle.io (1)
0000032982 [system] ERROR: Failed to determine server address
0000032982 [system] ERROR: Failed to determine server address
0000032983 [system] WARN: Cloud socket connection failed: -230
0000032983 [system] WARN: Cloud socket connection failed: -230
0000032983 [system] TRACE: Internet test socket
    32.983 AT send       4 "AT\r\n"
    32.987 AT read UNK   3 "AT\r"
    32.988 AT read OK    6 "\r\nOK\r\n"
    32.988 AT send      15 "AT+USOCTL=0,1\r\n"
    32.995 AT read UNK  14 "AT+USOCTL=0,1\r"
    32.996 AT read ERR   9 "\r\nERROR\r\n"
    32.996 AT send      15 "AT+USOCTL=1,1\r\n"
    33.002 AT read UNK  14 "AT+USOCTL=1,1\r"
    33.003 AT read ERR   9 "\r\nERROR\r\n"
    33.003 AT send      15 "AT+USOCTL=2,1\r\n"
    33.009 AT read UNK  14 "AT+USOCTL=2,1\r"
    33.010 AT read ERR   9 "\r\nERROR\r\n"
    33.010 AT send      15 "AT+USOCTL=3,1\r\n"
    33.016 AT read UNK  14 "AT+USOCTL=3,1\r"
    33.017 AT read ERR   9 "\r\nERROR\r\n"
    33.017 AT send      15 "AT+USOCTL=4,1\r\n"
    33.023 AT read UNK  14 "AT+USOCTL=4,1\r"
    33.024 AT read ERR   9 "\r\nERROR\r\n"
    33.024 AT send      15 "AT+USOCTL=5,1\r\n"
    33.030 AT read UNK  14 "AT+USOCTL=5,1\r"
    33.031 AT read ERR   9 "\r\nERROR\r\n"
    33.031 AT send      15 "AT+USOCTL=6,1\r\n"
    33.037 AT read UNK  14 "AT+USOCTL=6,1\r"
    33.038 AT read ERR   9 "\r\nERROR\r\n"
socketSocket(TCP)
    33.038 AT send      12 "AT+USOCR=6\r\n"
    33.044 AT read UNK  11 "AT+USOCR=6\r"
    33.045 AT read ERR   9 "\r\nERROR\r\n"
0000033046 [system] TRACE: socketed testSocket=0
0000033046 [system] TRACE: Connect Attempt
0000033047 [system] TRACE: socket_connect()=fail
0000033047 [system] TRACE: Close
0000033047 [system] WARN: Internet available, Cloud not reachable!
0000033047 [system] WARN: Internet available, Cloud not reachable!
0000033048 [system] WARN: Handling cloud error: 3
0000033048 [system] WARN: Handling cloud error: 3
0000033983 [system] INFO: Cloud: connecting
0000033983 [system] INFO: Cloud: connecting
0000033984 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
0000033984 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
0000033985 [system] ERROR: Failed to load session data from persistent storage
0000033985 [system] ERROR: Failed to load session data from persistent storage
0000033986 [system] INFO: Discarding session data
0000033986 [system] INFO: Discarding session data
0000033987 [system] TRACE: Resolving 38005d000d50483553353920.udp.particle.io
    33.986 AT send      56 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r\n"
    34.003 AT read UNK  55 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r"
    34.005 AT read ERR   9 "\r\nERROR\r\n"
    34.006 AT send      56 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r\n"
    34.021 AT read UNK  55 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r"
    34.023 AT read ERR   9 "\r\nERROR\r\n"
    34.024 AT send      56 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r\n"
    34.039 AT read UNK  55 "AT+UDNSRN=0,\"38005d000d50483553353920.udp.particle.io\"\r"
    34.041 AT read ERR   9 "\r\nERROR\r\n"
0000034043 [system] ERROR: Unable to resolve IP for 38005d000d50483553353920.udp.particle.io (1)
0000034043 [system] ERROR: Unable to resolve IP for 38005d000d50483553353920.udp.particle.io (1)
0000034044 [system] ERROR: Failed to determine server address
0000034044 [system] ERROR: Failed to determine server address
0000034044 [system] WARN: Cloud socket connection failed: -230
0000034044 [system] WARN: Cloud socket connection failed: -230
0000034045 [system] WARN: Resetting WLAN due to 2 failed connect attempts
0000034045 [system] WARN: Resetting WLAN due to 2 failed connect attempts
0000034046 [system] TRACE: Internet test socket
    34.045 AT send       4 "AT\r\n"
    34.049 AT read UNK   3 "AT\r"
    34.050 AT read OK    6 "\r\nOK\r\n"
    34.050 AT send      15 "AT+USOCTL=0,1\r\n"
    34.056 AT read UNK  14 "AT+USOCTL=0,1\r"
    34.057 AT read ERR   9 "\r\nERROR\r\n"
    34.057 AT send      15 "AT+USOCTL=1,1\r\n"
    34.063 AT read UNK  14 "AT+USOCTL=1,1\r"
    34.064 AT read ERR   9 "\r\nERROR\r\n"
    34.064 AT send      15 "AT+USOCTL=2,1\r\n"
    34.070 AT read UNK  14 "AT+USOCTL=2,1\r"
    34.071 AT read ERR   9 "\r\nERROR\r\n"
    34.071 AT send      15 "AT+USOCTL=3,1\r\n"
    34.077 AT read UNK  14 "AT+USOCTL=3,1\r"
    34.078 AT read ERR   9 "\r\nERROR\r\n"
    34.078 AT send      15 "AT+USOCTL=4,1\r\n"
    34.084 AT read UNK  14 "AT+USOCTL=4,1\r"
    34.085 AT read ERR   9 "\r\nERROR\r\n"
    34.085 AT send      15 "AT+USOCTL=5,1\r\n"
    34.091 AT read UNK  14 "AT+USOCTL=5,1\r"
    34.092 AT read ERR   9 "\r\nERROR\r\n"
    34.092 AT send      15 "AT+USOCTL=6,1\r\n"
    34.098 AT read UNK  14 "AT+USOCTL=6,1\r"
    34.099 AT read ERR   9 "\r\nERROR\r\n"
socketSocket(TCP)
    34.099 AT send      12 "AT+USOCR=6\r\n"
    34.105 AT read UNK  11 "AT+USOCR=6\r"
    34.106 AT read ERR   9 "\r\nERROR\r\n"
0000034107 [system] TRACE: socketed testSocket=0
0000034107 [system] TRACE: Connect Attempt
0000034108 [system] TRACE: socket_connect()=fail
0000034108 [system] TRACE: Close
0000034108 [system] WARN: Internet available, Cloud not reachable!
0000034108 [system] WARN: Internet available, Cloud not reachable!
0000034109 [system] WARN: Handling cloud error: 3
0000034109 [system] WARN: Handling cloud error: 3
0000034109 [system] WARN: Resetting WLAN due to SPARK_WLAN_RESET
0000034109 [system] WARN: Resetting WLAN due to SPARK_WLAN_RESET

[ Modem::deactivate ] = = = = = = = = = = = = =
    34.109 AT send      14 "AT+UPSDA=0,4\r\n"
    34.116 AT read UNK  13 "AT+UPSDA=0,4\r"
    34.117 AT read ERR   9 "\r\nERROR\r\n"

[ Modem::deactivate ] = = = = = = = = = = = = =
    34.117 AT send      14 "AT+UPSDA=0,4\r\n"
    34.124 AT read UNK  13 "AT+UPSDA=0,4\r"
    34.125 AT read ERR   9 "\r\nERROR\r\n"

[ Modem::deactivate ] = = = = = = = = = = = = =
    34.125 AT send      14 "AT+UPSDA=0,4\r\n"
    34.132 AT read UNK  13 "AT+UPSDA=0,4\r"
    34.133 AT read ERR   9 "\r\nERROR\r\n"

[ Modem::powerOff ] = = = = = = = = = = = = = =
    34.133 AT send       4 "AT\r\n"
    34.137 AT read UNK   3 "AT\r"
    34.138 AT read OK    6 "\r\nOK\r\n"
    34.138 AT send      12 "AT+CPWROFF\r\n"
    70.805 AT read UNK  11 "AT+CPWROFF\r"
    70.806 AT read OK    6 "\r\nOK\r\n"
[ ElectronSerialPipe::end ] pipeTx=0 pipeRx=0
[ ElectronSerialPipe::begin ] pipeTx=0 pipeRx=0

[ Modem::powerOn ] = = = = = = = = = = = = = =
    72.116 AT send       4 "AT\r\n"
    73.427 AT send       4 "AT\r\n"
    74.738 AT send       4 "AT\r\n"
    74.741 AT read UNK   3 "AT\r"
    74.742 AT read OK    6 "\r\nOK\r\n"
    74.742 AT send       9 "AT+CGMM\r\n"
    74.748 AT read UNK   8 "AT+CGMM\r"
    74.749 AT read UNK  13 "\r\nSARA-U260\r\n"
    74.750 AT read OK    6 "\r\nOK\r\n"
    74.951 AT send       6 "ATE0\r\n"
    74.956 AT read UNK   5 "ATE0\r"
    74.957 AT read OK    6 "\r\nOK\r\n"
    74.957 AT send      11 "AT+CMEE=2\r\n"
    74.962 AT read OK    6 "\r\nOK\r\n"
    74.962 AT send      19 "AT+CMER=1,0,0,2,1\r\n"
    74.968 AT read OK    6 "\r\nOK\r\n"
    74.968 AT send      15 "AT+IPR=115200\r\n"
    74.973 AT read OK    6 "\r\nOK\r\n"
    75.073 AT send      10 "AT+CPIN?\r\n"
    75.079 AT read  +   16 "\r\n+CPIN: READY\r\n"
    75.080 AT read OK    6 "\r\nOK\r\n"

[ Modem::init ] = = = = = = = = = = = = = = =
    75.080 AT send       9 "AT+CGSN\r\n"
    75.099 AT read UNK  19 "\r\n352580085739683\r\n"
    75.100 AT read OK    6 "\r\nOK\r\n"
    75.100 AT send       9 "AT+CGMI\r\n"
    75.104 AT read UNK  10 "\r\nu-blox\r\n"
    75.105 AT read OK    6 "\r\nOK\r\n"
    75.105 AT send       9 "AT+CGMR\r\n"
    75.109 AT read UNK   9 "\r\n23.20\r\n"
    75.110 AT read OK    6 "\r\nOK\r\n"
    75.110 AT send       9 "AT+CCID\r\n"
    75.117 AT read  +   30 "\r\n+CCID: 8934072500004338939\r\n"
    75.118 AT read OK    6 "\r\nOK\r\n"
    75.118 AT send      11 "AT+UPSV=1\r\n"
    75.123 AT read OK    6 "\r\nOK\r\n"
    75.123 AT send      11 "AT+CMGF=1\r\n"
    75.128 AT read OK    6 "\r\nOK\r\n"
    75.128 AT send      13 "AT+CNMI=2,1\r\n"
    75.133 AT read OK    6 "\r\nOK\r\n"
    75.133 AT send       9 "AT+CIMI\r\n"
    75.139 AT read UNK  19 "\r\n214074303747384\r\n"
    75.140 AT read OK    6 "\r\nOK\r\n"
0000075141 [system] INFO: Sim Ready
0000075141 [system] INFO: Sim Ready
0000075142 [system] INFO: ARM_WLAN_WD 1
0000075142 [system] INFO: ARM_WLAN_WD 1

[ Modem::register ] = = = = = = = = = = = = = =
    75.141 AT send      12 "AT+CGREG=2\r\n"
    75.146 AT read OK    6 "\r\nOK\r\n"
    75.146 AT send      11 "AT+CREG=2\r\n"
    75.151 AT read OK    6 "\r\nOK\r\n"
    75.151 AT send      10 "AT+CREG?\r\n"
    75.157 AT read  +   14 "\r\n+CREG: 2,0\r\n"
    75.158 AT read OK    6 "\r\nOK\r\n"
    75.158 AT send      11 "AT+CGREG?\r\n"
    75.164 AT read  +   15 "\r\n+CGREG: 2,0\r\n"
    75.165 AT read OK    6 "\r\nOK\r\n"
    75.165 AT send      10 "AT+URAT?\r\n"
    75.171 AT read  +   14 "\r\n+URAT: 1,2\r\n"
    75.172 AT read OK    6 "\r\nOK\r\n"
    75.172 AT send      10 "AT+CREG?\r\n"
    75.178 AT read  +   14 "\r\n+CREG: 2,0\r\n"
    75.179 AT read OK    6 "\r\nOK\r\n"
    75.179 AT send      11 "AT+CGREG?\r\n"
    75.185 AT read  +   15 "\r\n+CGREG: 2,0\r\n"
    75.186 AT read OK    6 "\r\nOK\r\n"
    77.037 AT read  +   14 "\r\n+CIEV: 2,1\r\n"
    81.723 AT read  +   31 "\r\n+CREG: 5,\"9841\",\"27A2E1A\",6\r\n"
    81.727 AT read  +   37 "\r\n+CGREG: 5,\"9841\",\"27A2E1A\",6,\"01\"\r\n"
    81.729 AT read  +   14 "\r\n+CIEV: 3,1\r\n"
    81.730 AT read  +   14 "\r\n+CIEV: 7,1\r\n"
    81.731 AT send      10 "AT+CREG?\r\n"
    81.731 AT read  +   14 "\r\n+CIEV: 9,2\r\n"
CIEV matched: 9,2
0000081733 TRACE: GPRS WD Cleared, was 0
    81.739 AT read  +   33 "\r\n+CREG: 2,5,\"9841\",\"27A2E1A\",6\r\n"
    81.740 AT read OK    6 "\r\nOK\r\n"
    81.740 AT send      11 "AT+CGREG?\r\n"
    81.748 AT read  +   39 "\r\n+CGREG: 2,5,\"9841\",\"27A2E1A\",6,\"01\"\r\n"
    81.750 AT read OK    6 "\r\nOK\r\n"
    81.750 AT send       4 "AT\r\n"
    81.753 AT read OK    6 "\r\nOK\r\n"
    81.753 AT send      10 "AT+COPS?\r\n"
    81.798 AT read  +   35 "\r\n+COPS: 0,0,\"0041005400260054\",2\r\n"
    81.800 AT read OK    6 "\r\nOK\r\n"
    81.800 AT send       8 "AT+CSQ\r\n"
    81.805 AT read  +   13 "\r\n+CSQ: 8,3\r\n"
    81.806 AT read OK    6 "\r\nOK\r\n"

[ Modem::join ] = = = = = = = = = = = = = = = =
    81.806 AT send      12 "AT+CGATT=1\r\n"
    81.812 AT read OK    6 "\r\nOK\r\n"
    81.812 AT send      14 "AT+UPSND=0,8\r\n"
Serial connection closed.