Can't read serial bytes fast enough to stop buffer overflow @ 115,200

Firmware
1

Hi All,

I have an external device I am trying to get to work with a B5 SOM. It communicates via a UART at 115,200 or faster. I have many years experience working with it.

The device responds to a command from the controller and returns a page, 1024 bytes plus a few bytes of overhead. The controller must acknowledge the page and the device sends a continuation byte and a further page or a "no more data" byte. It can send up to 13 pages.

This works fine up to 3 pages but then the wheels fall off. The Particle stops receiving data. This always happens at the same point. blockOnOverrun is false. I am logging each handshake for each page and the timing is consistent with the data being processed as it arrives. I have tried it with and without flow control. With flow control the data just stops and the Particle behaves accordingly. Without flow control it goes into a dumb state for a while and the function is not completed. The log data is not sent via the webhook.

It is as if the buffer is overflowing but the function should be removing each byte from the buffer as soon as it arrives. This is the code:

bCount = 1;    //byte count
cCount = 0;    //cycle count
pFound = false;
while (bCount < 1031 && cCount < 6000 && !pFound)     //6000 x 50 us = 300 ms
{
     while (Serial1.available() && bCount < 1031 && !pFound)
     {
          rByte = Serial1.read();
          dldBytes[rPages][bCount] = rByte;
          bCount++;
          if(bCount > 1028 && rByte == B_END_PAGE)         //end of packet found  ¶ - 0xB6
          {
               pFound = true;
          }
    }
    cCount++;
    delayMicroseconds(50);
}

Is there a limit to how quickly this code would remove bytes from the buffer? It should be fine at 115,200 baud. I have similar code running on a much slower Arduino and it works fine.
Not having a second serial port for debugging is a massive pain.

Any help would be appreciated.

Thanks in advance,

Rob

2

Hi Robert, the first thing I would add to my firmware would be this, so I can minimize the cloud connection impacting your serial activities:

SYSTEM_THREAD(ENABLED);

Have you?

3

Thank you very much. I think I am on the right track.

Each page can take between 100 and 300 ms to arrive. Once it is on its way a delay of 1 ms will cause corruption or the buffer to overload. The handshake is not as time critical so it might be possible to put the critical data transfer into a SINGLE_THREADED_BLOCK.

The reference says I can't use a SINGLE_THREADED_BLOCK for a lengthy operations. Is 300 ms too lengthy?

If that works I can put the speed back up to 250k baud, which is the default for the device.

Rob

4

You could be overflowing the serial buffer. The default buffer is 64 bytes. You may want to make the buffer at least large enough for a full page using acquireserial1buffer.

The blockOnOverrun affects sending, not receiving. In the absence of hardware flow control, any bytes that don't fit in the serial buffer are discarded.

Is there are reason you can't use the USB serial port for debugging? You can use it and the hardware UART at the same time.

I'm also suspicious of the delayMicroseconds. The MCU is fast enough to empty the serial buffer significantly faster than it is being filled at 115200 baud, so the inner while will often exit before the buffer is full. But then you will delay before checking again. I'm not positive this is a problem, but it's potentially an issue.

Also remember that Particle devices are running FreeRTOS with a thread scheduler. You do not get all of the CPU all the time, so keep in mind your thread will be swapped out at least every millisecond. It will be fast enough to read at 115200, but it will affect the serial buffer depth, which may overflow at 64 bytes.

I would make the serial buffer bigger, not attempt to prevent a thread swap.

5

@rlkeith I learned a lot from how serial comms are described in this Arduino post:

It may help you see things differently, or spot an issue in your code. Or maybe change it so you do not experience the issue anymore.
Apologies if serial comms is a skill you have already for years.

1 Like
6

I thought the USB serial port was not available on the B5 SOM. If it is I'll give it a go.

7

Thanks. I'll read that too, but what I really want to know if 300 ms is too lengthy to use in a SINGLE_THREADED_BLOCK.

I'll just give it a try, but might it cause other problems.

I have worked with serial devices on numerous platforms since the 90s but they were mostly much slower than this.

Rob

8

Thanks. I looked at acquireSerial1Buffer() but the explanation was opaque. Do you know of any examples that might help me understand it better?

9

Copy and paste this as a global function (not in a function):

hal_usart_buffer_config_t acquireSerial1Buffer()
{
    const size_t bufferSize = 1030;
    hal_usart_buffer_config_t config = {
        .size = sizeof(hal_usart_buffer_config_t),
        .rx_buffer = new (std::nothrow) uint8_t[bufferSize],
        .rx_buffer_size = bufferSize,
        .tx_buffer = new (std::nothrow) uint8_t[bufferSize],
        .tx_buffer_size = bufferSize
    };

    return config;
}

The way it works is Device OS implements acquireSerial1Buffer with weak linking. If you implement the function in your code it defaults to strong linking and overrides the default implementation.

10

Thank you. I added that code. It compiled and ran OK but the problem is still happening at precisely the same place. It must be that the Particle is wandering off at that point.

When I had flow control enabled, the Particle would de-assert CTS at the same point, keep it there for for 4 or 5 seconds, and only assert it again after the operation had timed out. The data would come flooding out of the 512 byte buffer of the device.

I am wondering whether to use a Teensy with two UARTs an an intermediate device. Collect the data from the device and drip-feed it to the particle.

Rob

11

Hi there, have you tried completely turning off all the wireless device's (cellular&bluetooth)? Ive had some funny lockup situations on Photon1's and patchy wifi.
Also i see you have a delay after your bCount while, if for some reason there is no serial1.available then it'll exit the bCount loop, wait a bit (50us), then go again. Is this causing sometime wierd in your datastream? Like missing the end of the page?

12

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.