I’m using a Particle M-SoM running Device OS 6.3.3. My device is in a location that does not have cell service, but does have WiFi available. When the device first started, the WiFi network was available and operational. After a day, I unplugged the WiFi network, but left the M-SoM running. Obviously, during the 3 days the WiFi network was unavailable, the M-SoM could not connect to the cloud. However, after 3 days I plugged the WiFi network back in, and the M-SoM successfully connected to WiFi again. However, it was unable to connect to the Particle cloud and all my HTTP server requests failed too. I plugged in a USB cable to the M-SoM and got the following logs multiple times:
0328885985 [system] INFO: Cloud: connecting
0328886008 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
0328886051 [system] WARN: Failed to load session data from persistent storage
0328886086 [system] INFO: Discarding session data
0328886141 [system] ERROR: Failed to determine server address
0328886164 [system] WARN: Cloud socket connection failed: -230
0328886228 [system] ERROR: Failed to determine server address
0328886253 [system] WARN: Internet test failed: -170 DNS
0328886282 [system] WARN: Handling cloud error: 2
I let the device try to connect for 15-20 minutes before power cycling it. Power cycling did fix the issue which is fine in my testing environment, but my concern is if I ship a device to customer who places it in the middle of nowhere with no one to power cycle it, I could be left high and dry. Is there any way to prevent this from happening? I’d prefer to not have to restart the device to fix this –either manually or in code. Thanks in advance for the help.
It would be helpful to get significantly more logs. The device should have switched to cellular during the time that it was not connected to Wi-Fi. The logs would indicate why that did not happen. In the bit of log included, it looks like DNS is not working over Wi-Fi, but it's not possible to tell why from the log.
Please see below log from a Particle M524 device, it is running device OS 6.3.3 and configured to use both WiFi and Cellular. This device periodically permanently loses its cloud connection every few days, power cycling the device restores the connection. I have a device with the same set up about 3m away that is on DeviceOS 5.9.0 that has no problems. I have other devices within 100m on the same network on DeviceOS 6.2.1 that similarly have no problems. The 6.3.3. firmware uses the PublishQueueExtRK whereas the other devices use PublishQueuePosixRK (thanks btw). The 6.3.3 device also has a BLE log handler which the other devices do not, this was used to obtain the attached logs.
Please note in the firmware logs there is a watchdog tickle, there is an external hardware watchdog on the device that is being tickled.
.006 0194809528 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:03.043 0194809530 [system] WARN: Failed to load session data from persistent storage
15:35:03.049 0194809530 [system] INFO: Discarding session data
15:35:03.090 0194809551 [system] ERROR: Failed to determine server address
15:35:03.091 0194809551 [system] WARN: Cloud socket connection failed: -230
15:35:03.135 0194809570 [system] ERROR: Failed to determine server address
15:35:03.136 0194809570 [system] WARN: Internet test failed: -170 DNS
15:35:03.182 0194809570 [system] WARN: Handling cloud error: 2
15:35:05.077 0194811086 [app] TRACE: Sampling indoor unit temperatures...
15:35:07.011 0194813570 [system] INFO: Cloud: connecting
15:35:07.012 0194813575 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:07.049 0194813576 [system] WARN: Failed to load session data from persistent storage
15:35:07.096 0194813577 [system] INFO: Discarding session data
15:35:07.096 0194813599 [system] ERROR: Failed to determine server address
15:35:07.140 0194813599 [system] WARN: Cloud socket connection failed: -230
15:35:07.141 0194813624 [system] ERROR: Failed to determine server address
15:35:07.235 0194813624 [system] WARN: Internet test failed: -170 DNS
15:35:07.239 0194813624 [system] WARN: Handling cloud error: 2
15:35:10.070 0194816086 [app] TRACE: Sampling indoor unit temperatures...
15:35:11.100 0194817624 [system] INFO: Cloud: connecting
15:35:11.101 0194817627 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:11.106 0194817627 [system] WARN: Failed to load session data from persistent storage
15:35:11.146 0194817629 [system] INFO: Discarding session data
15:35:11.191 0194817650 [system] ERROR: Failed to determine server address
15:35:11.194 0194817650 [system] WARN: Cloud socket connection failed: -230
15:35:11.235 0194817669 [system] ERROR: Failed to determine server address
15:35:11.236 0194817669 [system] WARN: Internet test failed: -170 DNS
15:35:11.282 0194817669 [system] WARN: Handling cloud error: 2
15:35:13.401 0194820001 [app] INFO: Feed the Dog :D
15:35:15.064 0194821086 [app] TRACE: Sampling indoor unit temperatures...
15:35:15.107 0194821669 [system] INFO: Cloud: connecting
15:35:15.107 0194821673 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:15.150 0194821674 [system] WARN: Failed to load session data from persistent storage
15:35:15.153 0194821674 [system] INFO: Discarding session data
15:35:15.194 0194821695 [system] ERROR: Failed to determine server address
15:35:15.196 0194821695 [system] WARN: Cloud socket connection failed: -230
15:35:15.240 0194821713 [system] ERROR: Failed to determine server address
15:35:15.241 0194821713 [system] WARN: Internet test failed: -170 DNS
15:35:15.287 0194821713 [system] WARN: Handling cloud error: 2
15:35:19.116 0194825713 [system] INFO: Cloud: connecting
15:35:19.155 0194825717 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:19.158 0194825718 [system] WARN: Failed to load session data from persistent storage
15:35:19.201 019^T ?^B^@^@^@^@ ^C^@^@^@^@^@] INFO: Discarding session data
15:35:19.202 0194825741 [system] ERROR: Failed to determine server address
15:35:19.339 0194825741 [system] WARN: Cloud socket connection failed: -230
15:35:19.342 0194825760 [system] ERROR: Failed to determine server address
15:35:19.381 0194825761 [system] WARN: Internet test failed: -170 DNS
15:35:19.382 0194825761 [system] WARN: Handling cloud error: 2
15:35:20.057 0194826087 [app] TRACE: Sampling indoor unit temperatures...
15:35:23.165 0194829762 [system] INFO: Cloud: connecting
15:35:23.205 0194829767 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:23.206 0194829768 [system] WARN: Failed to load session data from persistent storage
15:35:23.300 0194829769 [system] INFO: Discarding session data
15:35:23.301 0194829791 [system] ERROR: Failed to determine server address
15:35:23.342 0194829791 [system] WARN: Cloud socket connection failed: -230
15:35:23.344 0194829809 [system] ERROR: Failed to determine server address
15:35:23.386 0194829809 [system] WARN: Internet test failed: -170 DNS
15:35:23.387 0194829809 [system] WARN: Handling cloud error: 2
15:35:24.518 0194831070 [mux] INFO: Starting GSM07.10 muxer
15:35:24.520 0194831071 [mux] INFO: Opening mux channel 0
15:35:24.530 0194831071 [mux] INFO: GSM07.10 muxer thread started
15:35:25.103 0194831120 [app] TRACE: Sampling indoor unit temperatures...
15:35:25.104 0194831172 [mux] INFO: Opening mux channel 1
15:35:25.105 0194831205 [mux] INFO: Opening mux channel 2
15:35:25.864 0194832427 [net.ifapi] INFO: Netif pp3 link UP, profile=NONE
15:35:25.865 0194832430 [hal] INFO: DNS server list changed
15:35:25.866 0194832431 [hal] INFO: DNS server list changed
15:35:30.052 0194836087 [app] TRACE: Sampling indoor unit temperatures...
15:35:31.221 0194837809 [system] INFO: Cloud: connecting
15:35:31.223 0194837813 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:31.261 0194837814 [system] WARN: Failed to load session data from persistent storage
15:35:31.306 0194837815 [system] INFO: Discarding session data
15:35:31.307 0194837840 [system] ERROR: Failed to determine server address
15:35:31.307 0194837840 [system] WARN: Cloud socket connection failed: -230
15:35:31.351 0194837860 [system] ERROR: Failed to determine server address
15:35:31.396 0194837860 [system] WARN: Internet test failed: -170 DNS
15:35:31.398 0194837860 [system] WARN: Handling cloud error: 2
15:35:35.093 0194841087 [app] TRACE: Sampling indoor unit temperatures...
15:35:39.275 0194845860 [system] INFO: Cloud: connecting
15:35:39.278 0194845864 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:39.314 0194845865 [system] WARN: Failed to load session data from persistent storage
15:35:39.362 0194845866 [system] INFO: Discarding session data
15:35:39.363 0194845892 [system] ERROR: Failed to determine server address
15:35:39.406 0194845893 [system] WARN: Cloud socket connection failed: -230
15:35:39.407 0194845910 [system] ERROR: Failed to determine server address
15:35:39.451 0194845911 [system] WARN: Internet test failed: -170 DNS
15:35:39.453 0194845911 [system] WARN: Handling cloud error: 2
15:35:40.126 0194846087 [app] TRACE: Sampling indoor unit temperatures...
15:35:45.083 0194851087 [app] TRACE: Sampling indoor unit temperatures...
15:35:47.333 0194853911 [system] INFO: Cloud: connecting
15:35:47.370 0194853916 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:47.416 0194853916 [system] WARN: Failed to load session data from persistent storage
15:35:47.461 0194853917 [system] INFO: Discarding session data
15:35:47.462 0194853942 [system] ERROR: Failed to determine server address
15:35:47.506 0194853942 [system] WARN: Cloud socket connection failed: -230
15:35:47.508 0194853960 [system] ERROR: Failed to determine server address
15:35:47.550 0194853960 [system] WARN: Internet test failed: -170 DNS
15:35:47.596 0194853960 [system] WARN: Handling cloud error: 2
15:35:50.166 0194856087 [app] TRACE: Sampling indoor unit temperatures...
15:35:55.162 0194861087 [app] TRACE: Sampling indoor unit temperatures...
15:35:55.385 0194861960 [system] INFO: Cloud: connecting
15:35:55.394 0194861964 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:35:55.427 0194861965 [system] WARN: Failed to load session data from persistent storage
15:35:55.472 0194861966 [system] INFO: Discarding session data
15:35:55.472 0194861986 [system] ERROR: Failed to determine server address
15:35:55.517 0194861986 [system] WARN: Cloud socket connection failed: -230
15:35:55.562 0194862011 [system] ERROR: Failed to determine server address
15:35:55.563 0194862011 [system] WARN: Internet test failed: -170 DNS
15:35:55.608 0194862011 [system] WARN: Handling cloud error: 2
15:36:00.152 0194866087 [app] TRACE: Sampling indoor unit temperatures...
15:36:03.443 01948700^T ?^B^@^@^@^@ ^C^@^@INFO: Cloud: connecting
15:36:03.445 0194870016 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:36:03.480 0194870016 [system] WARN: Failed to load session data from persistent storage
15:36:03.526 0194870017 [system] INFO: Discarding session data
15:36:03.527 0194870037 [system] ERROR: Failed to determine server address
15:36:03.571 0194870037 [system] WARN: Cloud socket connection failed: -230
15:36:03.616 0194870062 [system] ERROR: Failed to determine server address
15:36:03.618 0194870062 [system] WARN: Internet test failed: -170 DNS
15:36:03.662 0194870062 [system] WARN: Handling cloud error: 2
15:36:05.106 0194871087 [app] TRACE: Sampling indoor unit temperatures...
15:36:10.102 0194876087 [app] TRACE: Sampling indoor unit temperatures...
15:36:13.478 0194880001 [app] INFO: Feed the Dog :D
15:36:15.054 0194881087 [app] TRACE: Sampling indoor unit temperatures...
15:36:19.508 0194886062 [system] INFO: Cloud: connecting
15:36:19.509 0194886065 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:36:19.591 0194886066 [system] WARN: Failed to load session data from persistent storage
15:36:20.092 0194886066 [system] INFO: Discarding session data
15:36:20.131 0194886089 [system] ERROR: Failed to determine server address
15:36:20.180 0194886089 [system] WARN: Cloud socket connection failed: -230
15:36:20.220 0194886106 [system] ERROR: Failed to determine server address
15:36:20.222 0194886107 [system] WARN: Internet test failed: -170 DNS
15:36:20.267 0194886107 [system] WARN: Handling cloud error: 2
15:36:20.267 0194886111 [app] TRACE: Sampling indoor unit temperatures...
15:36:25.124 0194891087 [app] TRACE: Sampling indoor unit temperatures...
15:36:30.079 0194896087 [app] TRACE: Sampling indoor unit temperatures...
15:36:35.070 0194901087 [app] TRACE: Sampling indoor unit temperatures...
15:36:35.611 0194902107 [system] INFO: Cloud: connecting
15:36:35.612 0194902110 [system] INFO: Read Server Address = type:1,domain:$id.udp.particle.io
15:36:35.701 0194902111 [system] WARN: Failed to load session data from persistent storage
15:36:35.703 0194902112 [system] INFO: Discarding session data
15:36:35.746 0194902133 [system] ERROR: Failed to determine server address
15:36:35.791 0194902134 [system] WARN: Cloud socket connection failed: -230
15:36:35.792 0194902152 [system] ERROR: Failed to determine server address
15:36:35.837 0194902152 [system] WARN: Internet test failed: -170 DNS
15:36:35.838 0194902152 [system] WARN: Handling cloud error: 2
15:36:36.973 Disconnected
I would enable LOG_LEVEL_TRACE so you get more information, particularly what automatic connection management is doing and which interface it's trying to use.
Also the information about what failed before trying to reconnect and failing would be useful.
What is the system free RAM at the time of being unable to reconnect? Do you have an out of memory handler defined? A memory leak causing an out of memory condition is one reason a device will fail to reconnect but will immediately reconnect after reset.