Muon Unknown crash

Firmware
1

BusFault Crash When Calling handleParseFailure()

I’m having an issue where simply calling a function causes a crash.

I have reset reason turned on and it prints this upon checking on reboot:

Reset Reason: 130, Data: 4

Which means:

RESET_REASON_PANIC, BusFault

Full Code Block for Reference

void ScaleDevice::handleReadResponse(CommandHandler &handler)
{
    Log.info("B1");

    if (handler.hasTimedOut())
    {
        Log.info("B2 timed out");

        FaultLight::setFault(faultLightCodes::connectionError);

        std::string timeoutMsg = handler.handleTimeout();
        deviceLogger(LOG_LEVEL_WARN, "%s", timeoutMsg);

        return;
    }

    if (!indicatorComm->available())
    {
        Log.info("B3 no data");
        return;
    }

    std::string response = indicatorComm->read();
    Log.info("B4 response='%s'", response.c_str());

    if (response.empty())
    {
        Log.info("B5 empty");
        return;
    }

    Log.info("B6 pre-parsing");

    Result preParsedStr = indicator->preParsing(response);

    Log.info("B7 preParse success=%d data='%s' err='%s'",
             preParsedStr.success,
             preParsedStr.data.c_str(),
             preParsedStr.errorMessage.c_str());

    if (!preParsedStr.success)
    {
        Log.info("B8 parse failed path");

        std::string parseFailMsg = handler.handleParseFailure();

        Log.info("B9");

        deviceLogger(LOG_LEVEL_WARN, "%s", parseFailMsg);
        FaultLight::setFault(faultLightCodes::badParseReponse);

        return;
    }

    if (preParsedStr.data.empty())
    {
        Log.info("B10 empty after parse");

        std::string parseFailMsg = handler.handleParseFailure();
        deviceLogger(LOG_LEVEL_WARN, "%s", parseFailMsg);

        return;
    }

    Log.info("B11 processing valid response");

    processValidResponse(handler, preParsedStr.data.c_str());

    Log.info("B12 done");
}

Where the Crash Happens

The issue arises when pre-parsing fails and execution enters this block:

if (!preParsedStr.success)
{
    Log.info("B8 parse failed path");

    std::string parseFailMsg = handler.handleParseFailure();

    Log.info("B9");

    deviceLogger(LOG_LEVEL_WARN, "%s", parseFailMsg);
    FaultLight::setFault(faultLightCodes::badParseReponse);

    return;
}

It always logs everything up to:

B8 parse failed path

However, when it enters handleParseFailure(), it never seems to even run the first line of code in that function. Even if I place a log statement as the first line, it never prints.

handleParseFailure() Implementation

std::string handleParseFailure()
{
    Log.info("HPF enter");

    if (!currentCommand)
    {
        Log.info("HPF no cmd");
        return "No Current Command";
    }

    Log.info("HPF1");

    std::string name = currentCommand->getCommandName();
    Log.info("HPF2 name=%s", name.c_str());

    std::string failResult = currentCommand->handleFailure();
    Log.info("HPF3 fail=%s", failResult.c_str());

    if (currentCommand->getState() == CommandState::FAILED)
    {
        Log.info("HPF4 failing");

        failCommand();

        Log.info("HPF5 failed");
    }
    else
    {
        Log.info("HPF4 retry");

        currentCommand->setState(CommandState::READY_TO_SEND);

        Log.info("HPF5 reset");
    }

    return "handled: " + name;
}

Observations

  • handler is valid at this point.
  • I can successfully call other functions on it right before handleParseFailure() (e.g., getting command queue size).
  • freeMemory() shows there is plenty of memory available.
  • If I move all the code from handleParseFailure() directly into handleReadResponse() and replace the function call with inline logic, everything works.
  • The crash only occurs when calling the function.

Current Theory

My only remaining lead is that this might be a heap memory issue — but I’ve tried everything I can think of.

2

Does this really work?

return "handled: " + name;

I don't think you can concatenate a const char * and a std::stringbut maybe you can. I'd change the line to just returning name and see if it makes a difference for a quick test.

Also, you should avoid using std::string. The C++ standard library function can throw an exception, but since exceptions are disabled in Device OS, the device will just reset.

3

Ive been using AI to work through this for a while now and that return statement was not always like that, used to just be returning a string with out concatenation and it still wasn't working then.

Ill change stuff to be char* and see if that changes anything, any idea why id be crashing on function call?

4 
std::string handleParseFailure()
{
Log.info("HPF1 currentCommand=%d", currentCommand != nullptr);

    if (currentCommand)
    {
        std::string errorMessage = "Failed to parse response for command '" +
                                   currentCommand->getCommandName() +
                                   "'. " +
                                   currentCommand->handleFailure();

        if (currentCommand->getState() == CommandState::FAILED)
        {
            failCommand();
        }
        else
        {
            currentCommand->setState(CommandState::READY_TO_SEND);
        }
        return errorMessage;
    }
    return "No Current Command";
}

This is what the function used to look like before I ran it through AI and it inserted its log statements.

This exact code used to work before I refactored some thing in my program and it worked fine. I expected there to be issues. Im just not even sure what this issue could be.

5

Ive been trying to use a debugger but I havent had much luck with that, it seems to error out on me when I try to set breakpoints

6

It's still using the + operator to concatenate a const char * and a std::string and I'm really not positive that works.

Also, all of the deviceLogger(LOG_LEVEL_WARN, "%s", parseFailMsg); are suspect. I think you need to be using parseFailMsg.c_str().

When the device crashes, you may not get the last debug messages by USB serial, so the last message you see logged may not exactly match the last line of code executed.

1 Like
7

Sure I can format the string concat in a different way, thought im not sure if thats the issue, the code execution never even seems to reach that in the first place. Do you think simply having that code in the function could cause it to crash when calling it?

8

Also my device logging function is set up to work with std::string or cstr

// Common logging function
template <typename... Args>
void deviceLogger(LogLevel level, const char* format, Args&&... args) const
{
    auto toCString = [](const auto& arg) -> auto {
        if constexpr (std::is_same_v<std::decay_t<decltype(arg)>, std::string>) {
            return arg.c_str();
        } else if constexpr (std::is_same_v<std::decay_t<decltype(arg)>, const char*>) {
            return arg ? arg : "<null>";
        } else {
            return arg;
        }
    };

    char buffer[256];
    snprintf(buffer, sizeof(buffer), format, toCString(std::forward<Args>(args))...);
    Log.log(level, "[%s][%s] %s", deviceType.c_str(), name.c_str(), buffer);
}
10 
void ScaleDevice::handleSendCommand(CommandHandler &handler)
{
    Log.info("A1");
    Result commandToSend = handler.getCommandToSend();
    Log.info("A2 success=%d data='%s'", commandToSend.success, commandToSend.data.c_str());

    if (!commandToSend.success || commandToSend.data.empty())
    {
        if (commandToSend.errorMessage == "No Current Command")
            return;
        Log.info("A3");
        std::string errorMessage = handler.handleSendFailure();
        Log.info("A4");
        deviceLogger(LOG_LEVEL_WARN, "%s: %s", commandToSend.errorMessage, errorMessage);
        return;
    }

    Log.info("A5");
    bool writeOk = indicatorComm->write(commandToSend.data.c_str());
    Log.info("A6 result=%d", writeOk);

    if (writeOk)
    {
        Log.info("A7");
        deviceLogger(LOG_LEVEL_INFO, "Command sent: %s", commandToSend.data);
        Log.info("A8");
        handler.markCommandSent();
        Log.info("A9");
    }
    else
    {
        Log.info("A10");
        FaultLight::setFault(faultLightCodes::connectionError);
        std::string failMsg = handler.handleSendFailure();
        deviceLogger(LOG_LEVEL_WARN, "%s", failMsg);
    }
}

It also seems to fail when calling handle send failure, which would lead to me thinking the handlers is screwed up but ive verified I can call other functions from handler just fine.

11

In that case, I would definitely look toward heap corruption.

In particular, make sure you aren't overflowing a member variable in handlers, but it also could be a different object that's overflowing into handlers.

12

0000011210 [app] INFO: Stack: 0x10041470, free mem: 2895648
0000011225 [app] INFO: A10
0000011232 [app] INFO: Stack: 0x10041470, free mem: 2895648

        Log.info("Stack: %p, free mem: %lu", __builtin_frame_address(0), System.freeMemory());

        Log.info("A10");
        Log.info("Stack: %p, free mem: %lu", __builtin_frame_address(0), System.freeMemory());

Im not sure what id be looking for here would you know if these numbers would point towards anything?

13

Is there anything in particular i should be looking for that i may have done to cause this?

14

The most common cause is a fixed-length array that you're overwriting the end of, but it could also be caused by using an invalid pointer that has been deleted.

1 Like
15

I have found success in simply making them return void. Not sure why that was the issue since even when i removed the concatenation and just make them return "", it still crashed. Ill just bite the bullet and take less accurate error logging. Hopefully that's it and it isn't actually heap corruption cause if that's the case ill probably run into the issue again later.

16

Another option to consider is instead of returning an object, pass it by reference:

void handleParseFailure(std::string &reason)
1 Like
17

thank you for the suggestion