I have four trackers running a custom firmware based on fw v11 (2.0.0-rc4) that crashed at different times this week. The cloud light is fixed on. The GPS light is off. They don’t send vitals anymore and don’t respond to cloud commands.
Of course it’s hard to ask (and provide) support on a custom firmware but the modifications are very limited and these trackers had been running fine since November.
What makes it even more surprising is that they are part of different groups of trackers assigned to different locations and some other trackers in the same groups with the same firmware are still working fine.
I have sent more details to support but wanted to ask here to see if anyone has an idea on the best way to recover them besides waiting for the battery to run out? If I had physical access I would open the box (void the warranty?) and disconnect/reconnect the battery but where they are I don’t have anyone technical enough to do this.
@sarfata, I had this happen with Edge Tracker v11 and v12 and DeviceOS 3.0.0-rc.2 but only when I exposed my TrackerOne to direct sunlight and the internal temperature went above 55C. The only way to recover is to open the TrackerOne case and remove the battery connector from the board. Pressing RESET did not work so all power had to be removed.
Logging a support ticket as you did is your best bet for now.
@sarfata, the TrackerOne publishes its internal temperature as part of its standard payload as “temp”. I have not tried to see what happens when the battery runs out but this assumes you have no charging source. I may wire a magnetic reed switch to the contacts on the reset switch and tape the reed to the cover so I can reset the unit by swiping a magnet over the “magic” spot from the outside.
BTW, I have been active since Elites came into being
@sarfata, not sure what is causing the problem. I handed over a serial log to the Particle Team to look at. Obviously, the battery still provides power since the LED stays on. The log file suggested that it might be an issue with the NCP which may cause a deadlock in DeviceOS. I am waiting for the analysis to provide more information.
BTW, in my case the final failure temperature was not published due to the publishing interval I set. However, I can attest that is it above 55C.
@eberseth Thanks Eric. I was able to reproduce the bug here by putting the device in a pelican case and leaving it out in the sun. It took about two hours:
When I tested this morning I had 2.0.0-rc4 on it. I just flashed one tracker with 3.0.0-rc2 and another one with the branch fix/3002-i2c-rtc. I will report back tomorrow when my source of heat is back on.
@eberseth the device with 3.0.0-rc2 crashed this morning. I flashed it with fix/3002-i2c-rtc after the crash. Both this one and the other one (which is on fix/3002-i2c-rtc since this morning) are both still working now and the sun is going down. It looks like the bug is fixed.
@sarfata Yes, it is a temperature related issue with some I2C timing. Some of the fixes don’t seem to make sense but we are accounting for clock drift between a couple of oscillators. @avtolstoy should take credit for figuring it out. It’ll be added to upcoming device OS releases.
I’m not sure if this is relevant, but I was having what seemed like random instances of my TrackerOne “locking up” as you described. It would start working fine again after I let the battery drain or even after a reflash of the firmware. But then, a day or two later, it would crash again.
My fix was to replace a 60-second delay in my main loop with a millis() strategy. I think somehow the delay was causing the problem. I’m not sure if it was or why it was, but everything has worked fine since I made that change. Hope this helps.
