[LTE] Tracker SoM with Tracker Edge sometimes crashes when trying to publish to Cloud

Edge Systems
1

Hi,

I am using a Tracker SoM device running a custom version of Tracker Edge v18 with DeviceOS 4.0.0.

I see from time to time the following issue appearing:

0000320477 [app] INFO: publishing now...
0000321802 [hal] ERROR: Not found response
0000321822 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000321822 [hal] ERROR: Please restart slave and test again,error code:264
0000324897 [ncp.esp32.mux] INFO: Starting GSM07.10 muxer
0000324897 [ncp.esp32.client] ERROR: Failed to perform early initialization
0000324898 [net.esp32ncp] ERROR: Failed to initialize wifi NCP client: -210
0000325319 [hal] ERROR: Not found response
0000325339 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000325339 [hal] ERROR: Please restart slave and test again,error code:264

Sometimes just after this issue, the board reboots and the reset reason flag is 130: RESET_REASON_PANIC.

I've checked the forum but couldn't find any explanation for this.

Best regards,
A.P.

2

I can confirm that I see this error too and I wonder if I should worry about it.

Antoine: my device does NOT reset on PANIC so that may be something else.

3

The CMD0 error message indicates that the main MCU was not able to communicate with the ESP32 by SPI. I’ve never seen that happen. It shouldn’t be panicking even if it can’t communicate with the ESP32, however.

I’d use DFU device restore to roll the device back to 3.x and plain Tracker Edge to rule out a hardware issue.

4

Hi,

Thank you for your quick replies!

I can reproduce this issue just by moving the board, it will induce a publishing event.

Here are logs where I see the error but the board continues to live:

0000365774 [tracker_motion] INFO: Motion detected by tracker motion
0000366219 [app] INFO: publishing now...
0000367565 [hal] ERROR: Not found response
0000367585 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000367585 [hal] ERROR: Please restart slave and test again,error code:264
0000370660 [ncp.esp32.mux] INFO: Starting GSM07.10 muxer
0000370661 [ncp.esp32.client] ERROR: Failed to perform early initialization
0000370661 [net.esp32ncp] ERROR: Failed to initialize wifi NCP client: -210
0000371081 [hal] ERROR: Not found response
0000371101 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000371101 [hal] ERROR: Please restart slave and test again,error code:264
0000371664 [app] INFO: cloud sent: {"cmd":"loc","time":1679330803,"loc":{"lck":0,"cell":78.3,"batt":100.0,"speed":0.00,"counter":361810.00},"trig":["imu_m"],"loc_cb":true,"req_id":2}
0000372318 [app] INFO: cloud received: {"cmd":"ack","req_id":2,"src_cmd":"loc","status":0}
0000372325 [app] INFO: location cb publish 371 success!
0000372370 [app] INFO: cloud received: {"cmd":"ack","req_id":2,"src_cmd":"loc","status":0}
0000374176 [ncp.esp32.mux] INFO: Starting GSM07.10 muxer
0000374176 [ncp.esp32.client] ERROR: Failed to perform early initialization
0000374177 [net.esp32ncp] ERROR: Failed to initialize wifi NCP client: -210

And where it crashes:

0000569215 [tracker_motion] INFO: Motion detected by tracker motion
0000569566 [app] INFO: publishing now...
0000570912 [hal] ERROR: Not found response
0000570932 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000570932 [hal] ERROR: Please restart slave and test again,error code:264
0000574007 [ncp.esp32.mux] INFO: Starting GSM07.10 muxer
0000574007 [ncp.esp32.client] ERROR: Failed to perform early initialization
0000574008 [net.esp32ncp] ERROR: Failed to initialize wifi NCP client: -210
0000574428 [hal] ERROR: Not found response
0000574448 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000574448 [hal] ERROR: Please restart slave and test again,error code:264
0000000501 [system.nm] INFO: State changed: NONE -> DISABLED

@gusgonnet I confirm that I see each time the same error code: 130.

@rickkas7 to what precise version of DeviceOS do you propose me to try?

Thanks!
Antoine

5

Oh, there are two parts to it: The Tracker is having trouble communicating with the ESP32, then you invoke a publish at the same time. It might be the combination of those two events that leads to the panic.

Tracker Edge v18 with Device OS 3.3.0 would be good.

6

In the meantime I confirm that even with Tracker Edge v18 by default with DeviceOS4.0.0 I see the HAL errors:

0000188909 [app] INFO: publishing now...
0000188943 [system.nm] TRACE: Request to power on the interface
0000189264 [hal] ERROR: Not found response
0000189284 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000189284 [hal] ERROR: Please restart slave and test again,error code:264

but the publication goes fine and the board doesn't crash. I tried this a few times and never a reboot.

When I put back my custom version with loggings set to Trace I got the following (crash at first try):

0000110225 [tracker_motion] INFO: Motion detected by tracker motion
0000113044 [app] TRACE: evaluatePublish minNetwork
0000113047 [app] TRACE: evaluatePublish min
0000113047 [app] TRACE: waiting for stable GNSS lock for triggers
...
0000123069 [app] TRACE: evaluatePublish minNetwork
0000123069 [app] TRACE: evaluatePublish min
0000123076 [app] TRACE: publishing from triggers after waiting
0000123076 [app] INFO: publishing now...
0000124104 [system.nm] TRACE: Request to power on the interface
0000124424 [hal] ERROR: Not found response
0000124444 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000124444 [hal] ERROR: Please restart slave and test again,error code:264
0000126506 [ncp.esp32.at] TRACE: > AT
0000126508 [ncp.esp32.at] TRACE: < OK
0000127508 [ncp.esp32.client] TRACE: NCP ready to accept AT commands
0000127509 [ncp.esp32.at] TRACE: > AT+MVER
0000127511 [ncp.esp32.at] TRACE: < 7
0000127511 [ncp.esp32.at] TRACE: < OK
0000127512 [ncp.esp32.at] TRACE: > AT+GETMAC=0
0000127514 [ncp.esp32.at] TRACE: < +GETMAC: "30:ae:a4:ac:98:5c"
0000127515 [ncp.esp32.at] TRACE: < OK
0000127517 [ncp.esp32.at] TRACE: > AT+CMUX=0
0000127520 [ncp.esp32.at] TRACE: < OK
0000127520 [ncp.esp32.mux] INFO: Starting GSM07.10 muxer
0000127520 [ncp.esp32.client] ERROR: Failed to perform early initialization
0000127521 [system.nm] TRACE: Interface 5 power state changed: 1
0000127521 [net.esp32ncp] ERROR: Failed to initialize wifi NCP client: -210
0000127621 [system.nm] TRACE: Interface 5 power state changed: 4
0000127941 [hal] ERROR: Not found response
0000127961 [hal] ERROR: CMD0 response error, expect 0x1, response 0
0000127961 [hal] ERROR: Please restart slave and test again,error code:264
!0000000503 [system.nm] INFO: State changed: NONE -> DISABLED
0000000503 [system.nm] TRACE: Interface 5 power state: 1
0000000503 [system.nm] TRACE: Interface 4 power state: 2