Trying to see the incoming data through the serial

Ahmedsa1983 · July 6, 2015, 7:17pm

Hey Guys
I have a an issue with reading the data from the serial port. I have incoming data from sensors through the serial1 (RX and TX pins) ,and I’d like to read these data before the cloud because there are either some missing characters in the data or a data which is longer than normal when I’m reading the data through the Dashboard. So, I want to make sure that I’m receiving a correct data from the serial before reading these data on the cloud. Is anyone had this issue before? Also I want to know how can I read these data on the serial?
Any suggestion please? @ScruffR @tjp
Thanks in advance.

Here is my code:

int RXPIN = RX;
int LED = D7;
String value;
void setup()
{
  pinMode(RXPIN,INPUT);
  pinMode(LED,OUTPUT);
  Serial1.begin(19200);
}
void loop()
{
  if (Serial1.available())
  {
  //This is to read the data as a string and go to the next line instead of reading the data as char (My data start with S)
    value = "S" +  Serial1.readStringUntil('S');
    delay(1000);
    Serial1.println(value);
    Spark.publish("Data1", value,PRIVATE);
    Serial1.flush();
    digitalWrite(LED,HIGH);
    delay(1000);
    digitalWrite(LED,LOW);
    delay(1000);
 }
}

bko · July 6, 2015, 7:59pm

You don't need pinMode for Serial1 RX input. Can you try it without this line?

Also the first step should probably be to just dump the data via publish without any fancy logic to make sure it is coming through OK. Something like calling Spark.publish() when you have 1-10 characters in a string depending on your data source.

Ahmedsa1983 · July 6, 2015, 8:14pm

@bko I deleted this line:

but nothing changed. Please I couldn't understand what you mean by:

Also I'd like to make sure that I understand what you mean by:

Do you mean that I may have an issue with the sensor's data which is causing this weird behavior in reading from the dashboard?
Thanks a lot for your response.

peekay123 · July 6, 2015, 8:17pm

@Ahmedsa1983, you could also dump the Serial1 data to Serial and monitor on you PC via a terminal program (eg. Putty). The biggest concern I would have with using Spark.publish() is a data string that is larger than 63 bytes. Another issue is that the data from the sensor is not ASCII. What sensor are your using?

bko · July 6, 2015, 8:22pm

I meant to change this line to something simple like:

 if (Serial1.available())
  {
    int inByte = Serial1.read();
    value = value + (char)inByte;
  }
if (value.length() > 10) {  //some number here based on your data stream
   Spark.publish("Data1",value,PRIVATE);
   value = "";  //clear old values
}

Ahmedsa1983 · July 6, 2015, 8:26pm

Dear @peekay123
Please could you tell me how can I dump the serial1 data to serial data? Do mean that I can say:

instead of

I'm using a pressure sensor that gives me Ascii characters through another board.
Thanks in advance.

Ahmedsa1983 · July 6, 2015, 8:32pm

@bko
Are these lines perform the same job instead of the single line above them?

bko · July 6, 2015, 8:36pm

I would not start off with readStringUntil() since it assumes a lot about the stream you are getting. I would get rid of this and do something more like what I wrote, but it is up to you to decide how you want to debug it.

Ahmedsa1983 · July 6, 2015, 8:44pm

@bko
I tried the code that you posted, but I received a stream of data without the right order. I should see something like this at the output:

SA00000000B00000000C00000000D00000000E

instead of that I received:

00SA00000BF
1FC00000C0D
A0000B00B00
0000000D0ES

Any suggestion please? Or can I make a condition with this line:

So I can start publishing the data if it is starting with S?
Thanks.

bko · July 6, 2015, 8:58pm

Hi @Ahmedsa1983

I see your data there starting with “SA00000BF1F…D0E”. What was your previous plan for parsing this stream of data? To wait for the next burst starting with “S” to trigger the previous one? That has timeout and storage problems you would need to fix.

If you want more help, you need to tell us more about this stream of serial data:

All the “packets” of data seem to start with “S”, right?
Are all the “packets” of data the same length?\

If so, your parser should wait for an “S” char and then read until the count of chars is the right length (possibly with a timeout if you want it to be more robust), and then do something with that data.

Ahmedsa1983 · July 6, 2015, 10:49pm

Dear @bko
Yes, my previous plan was to wait until I see S, and start triggering the new line of data. Please what do you mean by the timeout and storage problems? For more details about the data that I’m receiving from the sensor, all of the data started with S char. Also, the length of these data should be equal, which is 38 bytes. Now, how can I achieve the parser method that you told me about at the end of your post (your parser should wait…etc)?
Thanks.

ScruffR · July 7, 2015, 6:56am

If you are reading the serial input too slowly, you might be loosing data due to limited size of the receive buffer (64 byte).
Once it's full it won't take any more in, even if the sender carries on sending.
You need to code for this, since there is no hardware flow control.

To avoid running into above problem you should read as quickly as possible
e.g.

  char c;
  
  // read all available
  while(Serial1.available())
  {
    c = Serial1.read();
    // do something with c  
  }

But since this might keep your code occupied in the loop for too long, you should leave the loop from time to time (let it timeout)
e.g. like this

  char c;
  uint32_t ms = millis();
 
  // read all available, but max for 3sec
  while(Serial1.available() && millis() - ms < 3000)
   {
    c = Serial1.read();
    // do something with c  
  }

peekay123 · July 7, 2015, 12:23pm

@ScruffR, you are correct about using a timeout. However, going back to the original post, I believe the OP was not far from a working solution.

First, readStringUntil() does have a 1 sec timeout (fixed). However, implementing the code you suggest where the String is built one char at a time with a user-defined delay is more flexible. Each received char would need to be tested against ‘S’ to look for a start condition. The OP has indicated that each frame is 38 bytes long so when the start condition is met, a counter to read the 38 chars could be used so the code does not wait on the next frame start condition.

The addition of all the delay() code in the subsequent code could also cause the receive buffer to overflow. I would suggest creating a “frameReceived” flag that is set once a complete frame is received. Then, in loop(), test the flag, do the publish and use non-blocking (millis() based vs delay() based) timers for the LED flashing.

Since Spark.publish() is limited to firing once per second, the received frames may need to be queued if they are expected to be received at a rate exceeding one per second.

ScruffR · July 7, 2015, 2:53pm

@peekay123, you have summed up what I was thinking too.

I just wanted to answer the OPs questions first and wait for his response, before I would have come out with this

We do know about the inbuilt timeout of readStringUntil() but not erverybody does, but not knowing about it might contribute to not being able to see the reason for unexpected read results.
This is another reason why I’d rather suggest the char-by-char aproach (“write it yourself then you know what happens” ;-))

Ahmedsa1983 · July 7, 2015, 7:19pm

Dear @ScruffR
Thanks for your reply. I tried at the beginning to read char by char, but I changed my code to use readStringUntil() since it will display the data as a stream and that is what I need to see at the output. However, if reading the data char by char will explain to me what is happening, I'll try to use it again with the millis() and check if there is a weird behavior in the length of data or not. My concern right now is with the time delay, because I tried the following code yesterday, and I could get the correct data in a delay of 2-3 seconds between each frame of data, and this is too slow for me. Here is the modification that I did on the code:

//int RXPIN = RX;
int LED = D7;
String value;
void setup()
{
//pinMode(RXPIN,INPUT);
pinMode(LED,OUTPUT);
Serial1.begin(19200);
}

void loop()
{
if (Serial1.available())
{
value = "S" + Serial1.readStringUntil('S');
if (value.length() > 36 && value.length()<39) {
Spark.publish("Carpet1", value,PRIVATE);
value = ""; //clear old values
digitalWrite(LED,HIGH);
delay(1000);
digitalWrite(LED,LOW);
delay(1000);
}
}
}

So, what is your suggestion for that please?

Here is the output that I received:

SA00000000B1FC0007FC00000000D00000000E, Published at Mon Jul 06 2015 19:09:48 GMT-0500 (Central Daylight Time)
SA00000000BC003F000C00000080D00000000E, Published at Mon Jul 06 2015 19:09:50 GMT-0500 (Central Daylight Time)
SA00000000B3FC000FFC00000000D00000000E, Published at Mon Jul 06 2015 19:09:52 GMT-0500 (Central Daylight Time)
SA00000000B800FF000C00000080D00000000E, Published at Mon Jul 06 2015 19:09:54 GMT-0500 (Central Daylight Time)
SA00000000B3FC001FFC00000000D00000000E, Published at Mon Jul 06 2015 19:09:56 GMT-0500 (Central Daylight Time)
SA00000000B000FE000C00000000D00000000E, Published at Mon Jul 06 2015 19:09:58 GMT-0500 (Central Daylight Time)
SA00000000BFE8007F2C00000080D00000000E, Published at Mon Jul 06 2015 19:10:01 GMT-0500 (Central Daylight Time)
SA00000000B003FE000C00000400D00000000E, Published at Mon Jul 06 2015 19:10:03 GMT-0500 (Central Daylight Time)
SA00000000BFA001FE0C00400080D00000000E, Published at Mon Jul 06 2015 19:10:05 GMT-0500 (Central Daylight Time)
SA00000000B007FC001C00000000D00000000E, Published at Mon Jul 06 2015 19:10:07 GMT-0500 (Central Daylight Time)
SA00000000BF8001FE0C00400080D00000000E, Published at Mon Jul 06 2015 19:10:09 GMT-0500 (Central Daylight Time)

Ahmedsa1983 · July 7, 2015, 7:49pm

Dear @peekay123
I'll start review your reply to see if I'm understanding what you mean by each suggestion. Starting with the first suggestion:

Do you mean that I should put a counter after c = Serial1.read(); such as count++ that increases whenever there is a char reading?

For the second suggestion, do you mean that I may remove the time delay for the LED to make sure that it'll not affect on the buffer, and cause it to be overflowed?

For the next one,

I think that I tried something similar yesterday, but it gave me a null at the output. However, I tried to use more that one publish to make the code read the data faster than before but I'm not sure it is correct to do that or not. That's what I've tried:

int LED = D7;
String value;
boolean flag=true;
void setup()
{
pinMode(LED,OUTPUT);
Serial1.begin(19200);
}

void loop()
{
if (Serial1.available())
{
while(flag)
{
value = "S" + Serial1.readStringUntil('S');
Spark.publish("Carpet1", value,PRIVATE);
value = ""; //clear old values
flag=false;
}
value = "S" + Serial1.readStringUntil('S');
Spark.publish("Carpet1", value,PRIVATE);
value = ""; //clear old values
flag=true;
digitalWrite(LED,HIGH);
delay(1000);
digitalWrite(LED,LOW);
delay(1000);
}
}

For the last suggestion, how can I make a queue for the received frames?

Thanks in advance.

ScruffR · July 7, 2015, 9:10pm

You can do the same with char-by-char this way

char szReceive[64] = { '\0' };
int idx = 0;
uint32_t ms;
uint32_t msPublish;
bool frameStart = false;
...
void loop()
{
  ms = millis();

  while (Serial1.available() && millis() - ms < 1000 && idx < 38)
  {
    char c = Serial1.read();
    if (c == 'S')
    { 
      frameStart = TRUE;
      idx = 0;
    }
    if (frameStart)
    {
      szReceive[idx++] = c;
      szReceive[idx] = '\0';
    }
  } 
  
  if (idx >= 38 && millis() - msPublish > 1000)
  {
    Spark.publish("Carpet1", szReceive, PRIVATE);
    msPublish = millis();  // here was an error, causing following discussion
    //ms = millis();       // that was wrong
    idx = 0;
    frameStart = FALSE;
  }
}

Ahmedsa1983 · July 8, 2015, 2:30am

Thanks for your code @ScruffR
I tried the code, and it started to give me correct data for just three frames, and it stopped after that.
Here is the Dashboard output:

ScruffR · July 8, 2015, 6:10am

How does your code look now?

The code I gave you should work without problems forever unless something else interferes - e.g. other code, sensor stalls, WiFi interrupts, …

Just add a Serial.println(szReceive); before the Spark.publish() line, to see whether it’s only the cloud part that stops or if the code does not work propperly.
You may also add Serial.write(c); inside the while() to see what “raw” data you receive.
(Don’t forget Serial.begin(115200); inside your setup()).

Ahmedsa1983 · July 8, 2015, 6:15am

Here is the code that I’m using right now:

char szReceive[64] = { '\0' };
int idx = 0;
uint32_t ms;
uint32_t msPublish;
bool frameStart = false;

void setup()
{
   Serial1.begin(19200);
}
void loop()
{
   ms = millis();

   while (Serial1.available() && millis() - ms < 1000 && idx < 38)
   {
     char c = Serial1.read();
     if (c == 'S')
     { 
       frameStart = TRUE;
       idx = 0;
     }
     if (frameStart)
     {
       szReceive[idx++] = c;
       szReceive[idx] = '\0';
     }
   } 

   if (idx >= 38 && millis() - msPublish > 1000)
   {
     Spark.publish("Carpet1", szReceive, PRIVATE);
     msPublish = millis();  // here was an error, causing following discussion
     //ms = millis();       // that was wrong
     idx = 0;
     frameStart = FALSE;
   }
}

I just added the Serial1.begin(19200);, and changed zsReceive[idx++] = c; to szReceive[idx++] = c;

(ScruffR: I have reformatted your code - please use this format for code blocks – sorry for the typo)

code block

Topic		Replies	Views
Reading external serial and publishing [Solved] Troubleshooting	9	3426	October 12, 2016
Trying to send data from spark to the serial port(by using TX pin) Hardware	11	3854	July 15, 2015
Reading data from a serial sensor	2	2216	January 14, 2014
Trying to serial read from the Core Troubleshooting	0	897	July 6, 2015
Serial console full of binary data Troubleshooting	6	2042	March 15, 2014

Trying to see the incoming data through the serial

Related Topics