Tracker One crashed with fixed cloud lights - how to recover?


I have four trackers running a custom firmware based on fw v11 (2.0.0-rc4) that crashed at different times this week. The cloud light is fixed on. The GPS light is off. They don’t send vitals anymore and don’t respond to cloud commands.

Of course it’s hard to ask (and provide) support on a custom firmware but the modifications are very limited and these trackers had been running fine since November.

What makes it even more surprising is that they are part of different groups of trackers assigned to different locations and some other trackers in the same groups with the same firmware are still working fine.

I have sent more details to support but wanted to ask here to see if anyone has an idea on the best way to recover them besides waiting for the battery to run out? If I had physical access I would open the box (void the warranty?) and disconnect/reconnect the battery but where they are I don’t have anyone technical enough to do this.

Has anyone else seen this happen recently?


@sarfata, I had this happen with Edge Tracker v11 and v12 and DeviceOS 3.0.0-rc.2 but only when I exposed my TrackerOne to direct sunlight and the internal temperature went above 55C. The only way to recover is to open the TrackerOne case and remove the battery connector from the board. Pressing RESET did not work so all power had to be removed.

Logging a support ticket as you did is your best bet for now.

Do you mean that waiting for the battery to run out is not going to work? Have you tried this?

How do you know the temperature went above 55ºC? Are you tracking internal temperature somehow?


ps: long time no see - glad to see you are still active in this community :wink:

@sarfata, the TrackerOne publishes its internal temperature as part of its standard payload as “temp”. I have not tried to see what happens when the battery runs out but this assumes you have no charging source. I may wire a magnetic reed switch to the contacts on the reset switch and tape the reed to the cover so I can reset the unit by swiping a magnet over the “magic” spot from the outside.

BTW, I have been active since Elites came into being :wink:

1 Like

Do you know what state/color the cloud light is fixed on?

Seems to be Cyan (the normal color).

On this picture the one on the left is not working, the one on the right is working:

@peekay123 I think you were right on what cause them to crash.

All trackers were reporting temperatures close to 50ºC when they crashed.

On this graph is the temperature of the three trackers that crashed within one hour (the two green and one light blue):

And on this graph is the other one which crashed a few days before:

Do we know why the trackers would crash at this temperature? The graphs shows that another tracker got warmer but did not crash.

The doc says we should be good up to 60ºC:

(from Tracker One | Datasheets | Particle)

@sarfata, not sure what is causing the problem. I handed over a serial log to the Particle Team to look at. Obviously, the battery still provides power since the LED stays on. The log file suggested that it might be an issue with the NCP which may cause a deadlock in DeviceOS. I am waiting for the analysis to provide more information.

BTW, in my case the final failure temperature was not published due to the publishing interval I set. However, I can attest that is it above 55C.

1 Like

I have several tracker devices in a temperature chamber and testing them now at high temp. I’ll let you know.

@eberseth Thanks Eric. I was able to reproduce the bug here by putting the device in a pelican case and leaving it out in the sun. It took about two hours:

After unplugging the battery for one minute the device is working again.

Let me know if there is anything I can do to help.


1 Like

@sarfata Can you try to build against this branch in device OS? Commits · particle-iot/device-os · GitHub

Thanks @eberseth.

When I tested this morning I had 2.0.0-rc4 on it. I just flashed one tracker with 3.0.0-rc2 and another one with the branch fix/3002-i2c-rtc. I will report back tomorrow when my source of heat is back on.

@eberseth the device with 3.0.0-rc2 crashed this morning. I flashed it with fix/3002-i2c-rtc after the crash. Both this one and the other one (which is on fix/3002-i2c-rtc since this morning) are both still working now and the sun is going down. It looks like the bug is fixed.

(tracker 3 - yellow line - crashed around 11am see blue bar)

Looking at the commits, I do not understand how it got fixed though. Do you think the temperature was causing some i2c clocking issues?

Anyway it is looking good. Let me know if there is anything else I can do to help. Thanks for the super quick fix time.



@sarfata, I only got mine up to 62.1C and it didn’t crash either. I’ll be looking at the fix to try and understand as well.

@sarfata Yes, it is a temperature related issue with some I2C timing. Some of the fixes don’t seem to make sense but we are accounting for clock drift between a couple of oscillators. @avtolstoy should take credit for figuring it out. It’ll be added to upcoming device OS releases.


I’m not sure if this is relevant, but I was having what seemed like random instances of my TrackerOne “locking up” as you described. It would start working fine again after I let the battery drain or even after a reflash of the firmware. But then, a day or two later, it would crash again.

My fix was to replace a 60-second delay in my main loop with a millis() strategy. I think somehow the delay was causing the problem. I’m not sure if it was or why it was, but everything has worked fine since I made that change. Hope this helps.