Serial1 causing solid LED on P2

Firmware Device OS
1

I have recently ported a very well-tested code that has been working out in the field for several years on a P1 to a P2 and have encountered the following issues that I am struggling to solve. Any help would be highly appreciated.

  • Under Device OS 5.1.0, after a non-deterministic period of continuous operation (it can be anything from two days to several weeks), the P2 hangs with a solid LED and it is unrecoverable until you power cycle the device (it is not enough to press the reset button).

  • With the release of the Hardware Watchdog with Device OS 5.3.0, we thought it could be a temporary solution to mitigate the issue and buy us some time to discover the underlying problem. However, after upgrading to Device OS 5.3.0, the P2 hangs before even finishing the setup() function. In case it is of any interest, the same happens with Device OS 5.2.0. I have read in the changelog that this release introduced an experimental modification to how threads are handled in FreeRTOS, and I wonder if my problem could be related.

We have conducted extensive testing to try to track down the issue with Device OS 5.3.0 and this is the minimal non-working code that we have managed to find. There seems to be something wrong with Serial1, because we can run a fairly complex code with three threads perfectly, but as soon we call Serial1.begin(), the program freezes. Interestingly, if you add more code to the setup() function it sometimes starts up correctly, but we have been unable to find a pattern.

Without the SYSTEM_THREAD enabled:

SYSTEM_MODE(AUTOMATIC);

void setup() {
    Serial1.begin(9600, SERIAL_8N1);
}

void loop() {
}

With the SYSTEM_THREAD enabled the situation is a bit more difficult to diagnose. Depending on the code you add to the setup, it may or may not block, but it does hang often. If you remove the Serial1 initialization, this code works and if you don’t use WITH_LOCK, it works too, so I wonder if the issue has something to do with Serial1 using the Serial mutex inappropriately and creating a deadlock.

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(AUTOMATIC);

void setup() {
    Serial.begin(115200);
    WITH_LOCK(Serial) {
        Serial.printf("\n%s - Hello", (const char*)Time.timeStr());
    }
    Serial1.begin(9600, SERIAL_8N1);
}

void loop() {
}

Thank you in advance for your help,

Jaime

3

I run the above example which with threading enabled aginast 5.3.0 on a P2 and it’s hard to reproduce the issue. Can you reliably reproduce it?

2 Likes
4

Yes, it fails consistently. In case it has something to do, there is another device connected to the other end of the UART that may be transmitting while the module is starting, but that is not an issue for the P1 or for the P2 running Device OS 5.1.0.

5

So my P2 has been running the below app continuously very well for almost 74 hours:

#include "application.h"

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(AUTOMATIC);

void setup() {
    Serial.begin(115200);
    WITH_LOCK(Serial) {
        Serial.printf("\n%s - Hello", (const char*)Time.timeStr());
    }
    Serial1.begin(9600, SERIAL_8N1);
}

void loop() {
    Serial1.printlnf("Tick: %ld", millis());
    delay(1s);
}

Corresponding logs:

Tick: 265285564
Tick: 265286564
Tick: 265287565
Tick: 265288565
Tick: 265289566
Tick: 265290566
Tick: 265291567
Tick: 265292568
Tick: 265293568
Tick: 265294569
Tick: 265295569
Tick: 265296570
Tick: 265297570
Tick: 265298571
Tick: 265299571
Tick: 265300572
Tick: 265301572
Tick: 265302573

Could you try re-flashing the system-part1@5.3.0 and the app using another board? If you can still reproduce the issue reliably, it might be something wrong with the external flash, since we did experience some weired behaviors previously and we finally get rid of the trouble by erasing the whole external flash. Just make sure we're running the same user app.

Best regards

6

We have tried it on several P2 modules with the same result. We do have some additional information:

  • If there is nothing connected to the UART upon startup, everything initializes perfectly and the P2 almost never gets stuck.
  • However, if the device on the other end of the UART bus is transmitting while doing Serial1.begin(), the P2 freezes, no matter the frame that is being sent. This did not happen with Device OS <= 5.1.0.

Thanks again for your help,

Jaime

7

Yeah, I can reproduce the issue now. I’ll update the post when I have some outcomes.

8

Hi @jboalml There is the PR fixing the issue: [rtl872x] hal: uart may deadlock on initialization. by XuGuohui · Pull Request #2643 · particle-iot/device-os · GitHub

Best regards

9

Thank you. I have compiled the modifications locally and everything seems to work without issues now.

Regards,

Jaime

1 Like