Boron Randomly Enters Rapid Cyan Blink Mode - Stuck

Hi,
I have a Boron LTE running over cellular (no wifi connection) on my test bed. I have observed over the past two weeks the device has gone into rapid cyan blinking mode. When this happens there is no output to the serial log. I have the device sending reboot notes to persisted storage and can confirm the device is not rebooting either. SYSTEM_THREAD(ENABLED) is true. The firmware does not put the device into any sleep mode. I have flashed the latest firmware (5.7.0 as of last week.) I have confirmed in the Particle console that I have not reached any limits (cellular or operations per month). I have other devices, one other boron in the field, that are not experiencing this problem. This same problem happened with this device last week and required a reboot to get the device back online, I did NOT change the keys or anything else a simple reboot worked.

I am at a loss of what to do next. The device is blinking rapid cyan only, no other colors. It was running fine for around a week, no changes or updates to the firmware or hardware. What other reasons might cause a device to enter this state for no apparent reason?

If you are able to do so, I'd add this line to your firmware (as a global, not in a function):

SerialLogHandler logHandler(LOG_LEVEL_TRACE);

Checking the USB serial logs with logging enabled and it's in this state may provide helpful messages.

Fast blinking cyan is connecting to the Particle cloud.

I hit the reset button and the device connected without issue and firmware is running fine now. That confirms the problem is not the public or private key. All of the online docs I reviewed on your site point at key issues or not having system_thread enabled to maintain the cloud connection as primary reasons for the fast cyan.

The cellular signal quality and strength were around 19% at the time the device lost connection w/the cloud and went into fast cyan. Are there known issues with poor cell signal causing connection failures and unexpected loops on reconnect?

What other reasons are there for a fast blink with no other error flashes?

It's definitely not keys. Poor cellular connectivity could prevent the final handshake with the cloud from completing, which could cause getting stuck at fast blinking cyan.

But the symptoms are a little unusual, which is why a trace level debug log would be ideal.

Any other ideas? I also confirmed there is no memory leak, it holds steady. I set trace up in background now the waiting game.

Seems to have happened again today - did not have trace enabled but did collect a number of system messages. Device is stuck breathing green and hardware watchdog is not resetting the device. Stuck on slow green after 10 mins appears to have rebooted. The device came online after the restart. Appears from error either DNS has failed or endpoint gateway is not online? Is this a known scenario related to poor cellular connectivity or other condition I can code around?

0066919231 [system] ERROR: Failed to determine server address
0066927231 [system] WARN: Cloud socket connection failed: -230
0066935231 [system.cm] ERROR: No addrinfo for xxx.v5.udp-mesh.particle.io#5684
0066943231 [system] WARN: Internet test failed: -230 network
0066943231 [system] WARN: Handling cloud error: 2
0066943334 [system] WARN: Failed to load session data from persistent storage
0066951231 [system] ERROR: Failed to determine server address
0066959231 [system] WARN: Cloud socket connection failed: -230
0066967231 [system.cm] ERROR: No addrinfo for XXX.v5.udp-mesh.particle.io#5684

"Failed to determine server address" is usually a connectivity issue, as the device was able to connect to tower, but was unable to complete a DNS transaction.

Less commonly, the SIM was misconfigured in the mobile carrier network and is not allowing IP on the data connection, so UDP (DNS) is not working. This is not the case if the device has ever connected to the network.