E-Series resetting after publish

[supscientist]

Thanks for jumping in to help, @ScruffR.

More recent test results first, then I'll answer your questions. After leaving it charging via USB overnight, I just reflashed the same firmware and observed the same behavior. Then I flashed very slightly modified firmware (I commented out the two #if/#endif blocks as I'd had #define USING_CELL_NETWORK_UPLOAD 1 anyway so no change, theoretically, by commenting out the if logic. Then I retested the upload and it worked as expected once. Then without (re)flashing anything, I tried another upload and it failed as previously described.

I've observed this previously where I had the exact same problem as described above then flashed with a minor change, observed proper behavior for a bit, then the problem returns. So that seems like an interesting clue, maybe free memory related?

As for your questions and comments,

@ScruffR

When you say “goes to sleep” what’s the sleep command you use?

      Cellular.on();
      System.sleep(SLEEP_MODE_DEEP);

I have it in SYSTEM_MODE(MANUAL) to save power by turning Cellular.off() when I know it's out of range. But I know to turn Cellular.on() just before putting it into SLEEP_MODE_DEEP in order to properly, fully turn everything off.

@ScruffR

How long is the device running between two occurences of this issue?

It depends. I have two different ways to reproduce it. One is a normal operation of the device which is that it is turned on via a WKP trigger and it logs data till it gets a stop recording command (more on this in a sec) then it begins to try to transmit over cellular. The other is that I have it in its CLI and issue a "U" command for upload which puts it in the exact same state as it would have entered via the actual use case described above. So in the former (normal operation/field deployment) case, it can be hours or days between occurrences; in the latter (CLI/benchtop) case, it can be minutes. The WKP and "Stop Recording" are both tied to a water sensor. When the water sensor is tripped, it hits the WKP pin and begins recording. When the water sensor circuit is open (out of water), it begins trying to send data via cellular.

@ScruffR

Since this seems to be connected to some kind of panic reset of the device, have you got any means to catch the SOS blink pattern before reset?

The LED sequence I see (forgive me, I'm 90% sure this is correct but I'm red/green colorblind) is breathing cyan for a second or two (when cellular is turned on), then flashing green (seeking cell connection), breathing cyan for a couple seconds (transmitting that single message), then off. I have another red LED that is supposed to pulse at each successful transmission, or almost once per second (I'm limiting upload rate with while(((millis() - time_of_init_state_exit_ms) > 1000)) where time_of_init_state_exit_ms is reset to millis() after each successful transmission. But I see that red LED light up only once before the crash. I never see an SOS blink pattern in the RGB LED.

@ScruffR

I see you’re using String in your snippet, so I guess you may be using it frequently in the other thousands of lines too, which can cause issues over time, so I always advise against String and rather use good-ol’ C strings (if you use deep sleep, heap fragmentation should be less of a problem than with stop mode sleep).

Not seeing the rest of you code, some additional things to look out for is tripple checking buffer integrity and boundary checks for any indexes you use, pointer validation, free memory trend during the lead-up to the issue,

Noted. I'll take a look. Admittedly, I'll first have to look up what a lot of these suggestions mean (which is probably a good indicator that they may be related). Is there a straightforward way to monitor the free memory trend?

*** EDIT ***

@peekay123

System.freeMemory() API to determine the amount of available RAM.

I'll give that a shot.
*** End edit ***

Thanks again, I know this is a pretty involved conversation already. I really appreciate it!