Gen3 Filesystem Issue

Firmware Device OS
1

Hi all,

Hoping someone can shine some light on these logs from a dev-edition monitor one. Specifically, I've been working on developing our file-backed logging strategy. This includes char[] buffers being opened, written, appended to, and read through the LittleFS API per the docs. Notably, our app appears to have continued running throughout (albeit in a limited capacity) based on the timestamps and continued output via RS-485 and IO relay. While logging these errors, the status LED is bouncing between rapid cyan, rapid green, and an occasional pair of orange pulses corresponding with the 'internet test failed' log. Eventually it does self-recover, but after a while, without rebooting, it loses access to the session data in persistent storage and the whole process repeats. Is there anything I could be doing in the filesystem inadvertently that could explain this? I have used the DFU reset approach to wipe the 4MB flash section, but my understanding was that Device-OS had a separate 4MB section that the user firmware couldn't access.

Thanks!

0003833906 [system] WARN: Failed to load session data from persistent storage
0003849274 [system] ERROR: Failed to determine server address
0003849274 [system] WARN: Cloud socket connection failed: -230
0003854842 [system] WARN: Internet test failed: network
0003854842 [system] WARN: Handling cloud error: 2
0003854843 [system] INFO: Cloud: connecting
0003854849 [system] WARN: Failed to load session data from persistent storage
0003870274 [system] ERROR: Failed to determine server address
0003870274 [system] WARN: Cloud socket connection failed: -230
0003876950 [system] WARN: Internet test failed: network
0003876950 [system] WARN: Handling cloud error: 2
0003876952 [system] INFO: Cloud: connecting
0003876956 [system] WARN: Failed to load session data from persistent storage
0003892274 [system] ERROR: Failed to determine server address
0003892274 [system] WARN: Cloud socket connection failed: -230
0003894237 [system] WARN: Internet available, cloud not reachable
0003894237 [system] WARN: Handling cloud error: 3
0003894338 [system] INFO: Cloud: connecting
0003894341 [system] WARN: Failed to load session data from persistent storage
0003901120 [system] INFO: Cloud socket connected
0003901120 [comm.protocol.handshake] INFO: Establish secure connection
0003901126 [comm.dtls] INFO: (CMPL,RENEG,NO_SESS,ERR) restoreStatus=2
0003908318 [comm.protocol.handshake] INFO: Sending HELLO message
0003908663 [comm.protocol.handshake] INFO: Handshake completed
0003908663 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0003908665 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0003910760 [comm.protocol] INFO: Sending subscriptions
0003912774 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0003913856 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0003913958 [comm.protocol] INFO: Received TIME response: 1743630916
0003914160 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x02
0003914521 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0003914522 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0003915664 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x04
0003917055 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0003917057 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0003917370 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0003917372 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0003917474 [system] INFO: Cloud connected
2

This is unrelated to the file system.

I presume you are using Wi-Fi, and you are able to keep the connection to your access point, but the link between your network and the Internet is going down. Internet available, cloud not reachable is an indication of this.

However this message WARN: Failed to load session data from persistent storage is just a warning, not an error, and occurs all of the time. The session data is erased after some number of failed connections, in case the session data is corrupt, and a number of other common cases. You'll see this message frequently.

Also, I believe the session data is stored in retained memory, not the flash file system.

3

Ok good to know I'm not causing the problem w/ filesystem access code. We are not using WiFi (but also have not disabled it). I'll investigate the portion of our app that uses retained memory - thanks for the tip!

4

If you are using cellular, what country and which Particle device are you using?

5

USA, Monitor One dev edition SKU MON404E01C01KIT

6

@rickkas7 any updates here? This has continued to occur to some degree, and the devices in question are now also dropping subscriptions it appears. I am able to signal the devices from the Console, but any webhook-delivered messages for the devices are not being delivered to the device as evidenced by a debugger breakpoint set in the subscription callback never being hit. What else am I missing about how subscriptions work that would cause this behavior?

7

Subscriptions are best-effort, not guaranteed. Poor cellular connectivity can prevent subscriptions from being received.

Since you have other mysterious issues, something that is corrupting memory such as using a free block, overwriting a memory block, or using an deleted pointer, could cause the system to become unstable and practically anything could happen at that point.