Parsing serial data from photon

Firmware
1

Hello,

I connected my sensor to the Photon via RX in order to see incoming stream of bytes and parse the array.
I started to write my code but I am having dfficult with two aspects:

  1. I can’t see all the incoming data stream. It appears that in each reading only some of the bytes are receiving (the data is coming from the sensor in HEX).
    By the way, When I connect my sensor to Realterm I can see all the data packet flow without any breaks.

  2. I am trying to parse all the data output by this logic:
    Reading the first 52 bytes (Header).
    Each header always start with 8 bytes of the next value: ‘02 01 04 03 06 05 08 07’ but I didn’t know how to mark it as a start point for my stream.
    Then checking the total packet length by sum the bytes 19-22 in the array. This indicate the total length of the stream that I am expecting for (including the main header).

And this is the place that I got lost…
After reading the main header I need to read the next 8 bytes (TLV header).
The first 4 bytes in this array indicate the Type of the TLV.
There can be three kind of types - 06, 07, 08
If I am getting type 07 I need to read the first 4 bytes in the TLV array (this is after the 8 bytes TLV header), store it in a variable and publish to the cloud.
The last 4 bytes are the total length of the TLV so I could calculate the total bytes I need to receive (included the 8 bytes TLV header) untill the next TLV header.

This is my code so far:

int TotalPacketLength = 0;
int TLVLength = 0;
int TLVtype = 0;

void setup() {
  Serial.begin(115200);
  Serial1.begin(921600);
}

void loop() 
{
  
    delay(2500);

    byte FrameheaderData[52];
    Serial.println("\n\nFrame Header:");
    //read the first 52 bytes
    for(int i=0; i<52; i++)
    {
        if (Serial1.available())
        {
            Serial.print("[" + String(i) + "]-");
            FrameheaderData[i] = Serial1.read();
            Serial.print(String(FrameheaderData[i]) + "\t");
            delay(20);
        }
    }
               
    Serial.println("\n\nPacket Length:");
    TotalPacketLength = FrameheaderData[19] + FrameheaderData[20] + FrameheaderData[21] + FrameheaderData[22];
    Serial.println(String(TotalPacketLength));
  

    Serial.println("\n\nTLV Header:");
    //read the tlv header (after the first 52 bytes)
    byte TlvHeader[8];
    for(int i=0; i<8; i++)
    {
        if(Serial1.available())
        {
            TlvHeader[i] = Serial1.read();
            Serial.print("[" + String(i) + "]-");
            Serial.print(String(TlvHeader[i]) + "\t");
            delay(10);
        }
    }  
    Serial.println("\n\nTLV Type:");
    //read the tlv object type in the header (first 4 bytes)
    TLVtype = (TlvHeader[0] + TlvHeader[1] + TlvHeader[2]) + TlvHeader[3];
    Serial.println(String(TLVtype));

    //read the tlv length and in the header (last 4 bytes)
    Serial.println("\n\nTLV Length:");
    TLVLength = TlvHeader[4] + TlvHeader[5] + TlvHeader[6] + TlvHeader[7]-8;
    Serial.println(String(TLVLength));
    
    byte TlvBody[TLVLength];
    for(int i=0; i<TLVLength; i++)
    {
        if(Serial1.available())
        {
            TlvBody[i] = Serial1.read();
            Serial.print("[" + String(i) + "]-");
            Serial.print(String(TlvBody[i]) + "\t");
            delay(10);
        }
    }

     
  delay(8000);
  Serial.println("\n\n================================================================================");
  
}

Any help will be appreciated
Thanks,
Avi

2

A few things to first:

  • especially when communicating with an asynchronous data source you want to avoid delay() wherever possible.
  • when you are expecting a structured input as you describe it's usually cleaner to feed your data into a struct than just dumping it into an unstructured byte array
  • "the data is coming from the sensor in HEX" this is typically a misconception. The data always arrives as binary data, you may be looking at that in hexadecimal representation, but you could also look at it in any other way
  • don't use String wherever you can avoid it
  • definetly don't use delay(20) within the read loop, you will most likely run into conditions where the RX buffer will be overflowed (especially with your baudrate)
  • make sure to start (and optionally finish) with a clean RX buffer

You can have a look at this
https://docs.particle.io/reference/device-os/firmware/photon/#find-

once you found the header you can use this to fill your payload struct
https://docs.particle.io/reference/device-os/firmware/photon/#readbytes-

3

Thank you for your quick reply @ScruffR If you can kindly give me an example of how to dump the data into the unstructured byte array that would bee usfull.
I know delay was not a good idea but because I didn't get all the data stream as I expected I thought to make some break point with the delay().

But the first main issue that I am struggling with is that I am not getting all the bytes in data output with each transmission.
I was searching in this forum and found some code that you wrote for seeing serial1 ouput:

#include "Particle.h"
#include "SerialBufferRK.h"

SerialBuffer<4096> serBuf(Serial1);

char data[64];
char recv[128];

void setup() {
  Serial.begin(115200);
  Serial1.begin(921600);
  Particle.function("SerTest", test);
  Particle.variable("data", data);
  Particle.variable("received", recv);
}

void loop() {
  int i = 0;  
  
  while(Serial1.available())
   {
     Serial.printf("%02x ", recv[i++] = Serial1.read());
   }
  if (i)
    {
     recv[i] = '\0';
     Serial.printf("\n<%s>\n", recv);
    }
  }

 int test(String cmd)
 {
     cmd.toCharArray(data, sizeof(data));

     Serial.printf("\n<%s>\n", data); // leading and trailing newline for loggin 
     Serial1.printf("%s\n", data);    // trailing newline as end marker
       
     return cmd.length();
 }

I added the SerialBuffer becuse the photon was getting into blinking red and stoped but It works like charm for seeing the data output. But from this I realized that I don't receive all of the data with each UART transmission.
Any suggestion what is causing that?

Thanks,
Avi

4

That's probably due to this potential issue I pointed out before

With these delays you will end up with an RX buffer being filled with fragments of not fully read packets being partly overwritten with new data.
That's why I suggested

Also you are expecting to read data at 921600 baud, inflate the data by reformatting each byte into three characters and then send that across USB serial.
You may misinterpret not seeing the data at the end of the chain as not actually getting it while you actually may only be losing it on the way.

Have you given the two links I provided above a read?

I'd not use SerialBufferRK as a "blind" bandaid before having established the fundamental transmission to work without error.

5

Yes I was reading the links. So actually you say it’s better go with the readBytes() and the readBytesUntil() implementation instead of the serial1.read() Because of the high baud rate? And probably this is the reason for the missing bytes at the end of the stream?

6

Not because of that, but since you already know how many bytes you need after you found the header I'd not reinvent the wheel and rather rely on an optimised, more feature rich, tested function that exactly does what you need.

If this doesn't work you'll probably be looking at a HW limitation rather than a software problem.

7

I see, So the 921600 baud not going to be an issue for the photon using the RX connectivity?
I just want to make sure that he can handle with all the data burst.

8

I’m not saying that either.
I said when this (meaning Serial1.find() and Seria1.readBytes()) doesn’t work then you may have to consider this possibility.

9

Ok, I’ll give the find() and the readBytes() a try. Hopefully, it will work.
Thanks @ScruffR

10

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.