P2 stuck at (solid) cyan

Firmware Device OS
1

running connectivity tests over night (connect to ethernet, once it is connected, restart and do it again) with our p2 device i found it solid cyan this morning.

last log lines:

0000367807 [app] INFO: Connecting Ethernet
0000367810 [app] INFO: timeout set to 5000 ms
0000367813 [app] INFO: State: 'eth connecting' (previous: 'eth connect')
0000367819 [app] INFO: State: eth connecting: Ethernet: ready, WiFi: not ready, Cloud: not connected
0000367880 [app] INFO: Ethernet network connected in 0 s
0000367882 [app] INFO: Ethernet IP address: 192.168.1.121
0000369908 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000369909 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000369915 [comm.protocol.handshake] INFO: Sending HELLO message
0000369921 [comm.coap] TRACE: Sending CoAP message
0000369926 [comm.coap] TRACE: CON POST /h size=39 token= id=8952
0000375157 [comm.coap] TRACE: Retransmitting CoAP message; ID: 8952; attempt 1 of 3
0000375181 [comm.coap] TRACE: Sending CoAP message
0000375187 [comm.coap] TRACE: CON POST /h size=39 token= id=8952
0000384954 [comm.coap] TRACE: Retransmitting CoAP message; ID: 8952; attempt 2 of 3
0000384972 [comm.coap] TRACE: Sending CoAP message
0000385000 [comm.coap] TRACE: CON POST /h size=39 token= id=8952
0000401422 [comm.coap] TRACE: Retransmitting CoAP message; ID: 8952; attempt 3 of 3
0000401424 [comm.coap] TRACE: Sending CoAP message
0000401428 [comm.coap] TRACE: CON POST /h size=39 token= id=8952

the watchdog is set but did not restart. why is the rgb led solid cyan?

thanks
frank

2

Solid colors occur if the device deadlocks. The hardware MCU watchdog should be able to reset out of that state, however.

The deadlock can occur if interrupts are disabled or you have single threaded mode enabled and block on a mutex that can only be freed from another thread. Memory corruption from using a freed pointer, freeing a pointer twice, overwriting the end of a memory block, or overflowing the stack can also cause deadlock.

3

what could be the reason it did not? it was enabled

4

The most likely cause of the device not watchdog resetting after lockup is that the MCU locked up because of a power-related issue.

5

the board is powered by a more than decent power supply

the interesting bit: every time it hangs with the same last log lines

hang 1:

0003482707 [app] INFO: Connecting Ethernet
0003482709 [app] INFO: timeout set to 5000 ms
0003482713 [app] INFO: State: 'eth connecting' (previous: 'eth connect')
0003482833 [app] INFO: State: eth connecting: Ethernet: ready, WiFi: not ready, Cloud: not connected
0003483005 [app] INFO: Ethernet network connected in 0 s
0003483020 [app] INFO: Ethernet IP address: 192.168.1.121

hang 2:

0000370479 [app] INFO: Connecting Ethernet
0000370483 [app] INFO: timeout set to 5000 ms
0000370487 [app] INFO: State: 'eth connecting' (previous: 'eth connect')
0000370606 [app] INFO: State: eth connecting: Ethernet: ready, WiFloud: not connected
0000370610 [app] INFO: Ethernet network connected in 0 s
0000370614 [app] INFO: Ethernet IP address: 192.168.1.121

hang 3:

0006739713 [app] INFO: Connecting Ethernet
0006739715 [app] INFO: timeout set to 5000 ms
0006739717 [app] INFO: State: 'eth connecting' (previous: 'eth connect')
0006739835 [app] INFO: State: eth connecting: Ethernet: ready, WiFi: not ready, Cloud: not connected
0006739838 [app] INFO: Ethernet network connected in 0 s
0006739844 [app] INFO: Ethernet IP address: 192.168.1.121

if(Ethernet.ready()) {

   connectivity_time_linkup = millis();

   Log.info("Ethernet network connected in %d s",
      (int)((connectivity_time_linkup - connectivity_time_start)/1000));

   // LAST LOG LINE BEFORE HANG:
   Log.info("Ethernet IP address: %s", Ethernet.localIP().toString().c_str());   
                
   connectivity_state = CONNECTIVITY_ETHERNET_CLOUD_WAIT_CONNECTED;

   Particle.keepAlive(25s);
   Particle.connect();

   timeouttimer_set(TIMEOUT_CLOUD_CONNECT);

}