Debugging hard faults - accessing hard fault contextual information

[jaza_tom]

Hello all. I am trying to debug a hard fault that I am getting on my Boron application.

I have isolated the fault to a certain call from within my main loop through the use of the retained SRAM feature and by creating “checkpoints” which save the last checkpoint hit in the previous runtime. I can then check which checkpoint was the last one to get hit before the hard fault on a subsequent boot (via debug message).

See rough setup below:

I am wondering if that is as good as it gets in the Particle world in terms of tracing down the source of hard faults?

I am used to implementing a custom hard fault handler in the bare metal projects that I usually work on which allows me to data log/save the stack frame that gets pushed as per cortex-M specifications when the hard fault occurs.

This allows be to:

• See what value the the program counter was at when the hard fault occurred
• See what the core registers of the MCU were at the time of the hard fault

Rough setup shown below:

Is there anything like this in Particle that I can access?

How can I determine what was the last line of code to execute before the hard fault?

Is it really just a matter of iteratively adding checkpoints everywhere in my code until I can trace where/when the fault is happening?

[rickkas7]

In the past you could just put a breakpoint in the Device OS panic_ext function and look at the backtrace. This should still be possible, but when debugging with a modular Device OS build there are probably some steps involved in making sure the device has a debugging version of Device OS flashed and loading the symbols so gdb can set a breakpoint in the system code. I'm not sure how you do that, however.

[jaza_tom]

OK that sounds like it could be promising.

The last time I tried to get debugging working in Particle Workbench I could not get the device to boot up (just kept blinking yellow in DFU mode after flashing). I used Flash application and device OS for debug (local)

I am assuming that this may have something to do with the modular device OS and the trickiness you are implying.

Is there an up-to-date known working way to debug Particle OS devices using Particle Workbench and Device OS v 6.3.3?

If not, is there a plan to provide that guidance/functionality any time soon?

I’m guessing this is not high on the Particle to-do list since there is no Particle Debugger product being sold anymore?

[rickkas7]

I figured out a far easier way to set the breakpoint.

• Connect the device and debugger to your computer. Both USB connections must be made.
• Use Particle: Flash application & Device OS for debug (local)

This is my test firmware:

#define PARTICLE_USE_UNSTABLE_API
#include "Particle.h"

SYSTEM_MODE(AUTOMATIC);

SYSTEM_THREAD(ENABLED);

SerialLogHandler logHandler(LOG_LEVEL_INFO);

void MyPanicHook(const ePanicCode code, const void* extraInfo);
// void panic_set_hook(const PanicHook panicHookFunction, void* reserved);


int testFn(String cmd); 

void setup()
{
    panic_set_hook(MyPanicHook, nullptr);

    Particle.function("test", testFn);
}

void loop()
{
}

int testFn(String cmd) {
    Log.info("test %s", cmd.c_str());

    uint32_t *p = (uint32_t *)0x7fffffff;
    *p = 0xffffffff;

    return 0;
}

void MyPanicHook(const ePanicCode code, const void* extraInfo) 
{

}

Note the #define PARTICLE_USE_UNSTABLE_API at before Particle.h is included. This is necessary for the panic_set_hook to be available.

Set a breakpoint in MyPanicHook.

Use Run: Start Debugging.

Calling the function test on the device will cause a hard fault, which should halt at the breakpoint.

You can see that the breakpoint was hit. Looking at the stack trace you can see testFn in the call stack, which is where the fault actually occurred.

Once you've hit the breakpoint, clicking on testFn in the call stack seems to correctly flag just after the line of code that caused the fault.

Also the debugger seems to be able to display Device OS source when you click on those items in the call stack without requiring any special configuration. That seems to just work.

[rickkas7]

I figured out an even easier way that does not require modifying your source. It also is how you set a breakpoint in any Device OS function, even with a modular build. It's in the Workbench debugger documentation now.

[michschn-oww]

I’m interested in this too, its very complicated to debug a crash.

could the particle OS make the pointer available where it crashed, which we could then resolve with the symbols?

The panic_set_hook does work, but I cannot manage to get the debugger to work in my HW setup (due to the conflicts of the SWD/JTAG pins with D6 and D7), so setting the breakpoint does not work.

I’d be happy if I could get the crash address after reboot. This would at least help me know where where to start looking.

[rickkas7]

You could capture the current thread and stack frame information in retained memory and recover it on boot and send it to the cloud. Then you have a system that takes the map file and decodes it. That's what MemFault does. The caveat its that it's kind of expensive and does not currently work on RTL872x devices.

But the basic problem is that you can't capture the whole stack because it's too large, and in order to get the call stack of where the fault occurred you need to be able to decode stack frames to get the PC addresses and ignore the stack local data. It's definitely possible, but not trivial to do.