Reading serial sensor data (COZIR) into string , intermittently the order of the data is jumbled

First - thank you in advance for any input here - has me scratching my head!
I am reading data in via Serial from a COZIR temp/humidity/co2 sensor.
The serial Format of the data is “H ##### T ##### Z ##### z ##### \r\n”
where H is the humidity, T is temperature, Z is filtered CO2, and z is instantaneous CO2 value.
as you can see in the serial output at the end of my post, sometimes that data gets read into the string in the incorrect order(I think?)
Most of the time its correct , but intermittently … its in the wrong order.
and sometimes there is extra data as well(second output has an extra zero after the ‘z’ value - z 007820 which causes CO2 to read 7,820 ppm instead of the correct 782 ppm)
Does anyone see anything in my code that could cause this?

// - Receive with start- and end-markers combined with parsing

const byte numChars = 32;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

      // variables to hold the parsed data
//char messageFromPC[numChars] = {0};
float humidity = 0;
float temp1 = 0;
float co2avg = 0;
float co2 = 0;


boolean newData = false;

//============

void setup() {
    Serial.begin(9600);
    Serial1.begin(9600);
    Serial.println("Data String Expected from sensor is H ##### T ##### Z ##### z ##### \r\n");
//    Particle.variable("humidity", humidity);
//    Particle.variable("temperature", temp1);
 //   Particle.variable("co2avg", co2avg);
 //   Particle.variable("co2", co2);
    
}

//============

void loop() {
    recvWithStartEndMarkers();
    if (newData == true) {
        strcpy(tempChars, receivedChars);
            // this temporary copy is necessary to protect the original data
            //   because strtok() used in parseData() replaces the commas with \0
        parseData();
        calculations();
        showParsedData();
        sendToCloud();
        newData = false;
        
    }
    
}

//============

void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = 'H';
    char endMarker = '\r';
    char rc;

    while (Serial1.available() > 0 && newData == false) {
        rc = Serial1.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

//============

void parseData() {      // split the data into its parts
   
    char * strtokIndx; // this is used by strtok() as an index

    Serial.println("String ");
    Serial.println(tempChars);
    Serial.println("-----------------");
    strtokIndx = strtok(tempChars,"TZz");      // get the first part - the string
    humidity = atof(strtokIndx);  // convert to int
 
    strtokIndx = strtok(NULL, "TZz"); // this continues where the previous call left off
    temp1 = atof(strtokIndx);     // convert this part to an integer

    strtokIndx = strtok(NULL, "TZz");
    co2avg = atof(strtokIndx);     // convert this part to an int

    strtokIndx = strtok(NULL, "TZz");
    co2 = atof(strtokIndx);     // convert this part to an int

}

//============



void calculations() {
    
    temp1 = (((temp1 - 1000) / 10) * 1.8) + 32;
    humidity = humidity / 10;
}


void showParsedData() {
    
    
    Serial.print("humidity: ");
    Serial.println(humidity);
    Serial.print("temperature: ");
    Serial.println(temp1);
    Serial.print("CO2 Avg ");
    Serial.println(co2avg);
    Serial.print("CO2 ");
    Serial.println(co2);
 
}

void sendToCloud(){
 
 
    Particle.publish("UbidotsHumidity", String(humidity), PRIVATE);
    delay(2000);
    Particle.publish("UbidotsTemp", String(temp1), PRIVATE);
    delay(2000);
    Particle.publish("UbidotsCo2", String(co2avg), PRIVATE);
    delay(2000);
delay(90000);  //wait 15 minutes
}

Here are the Serial Results:

Data String Expected from sensor is H ##### T ##### Z ##### z ##### \r\n

String  00628 T 01212 Z 00758 z 00726
-----------------
humidity: 62.80
temperature: 70.16
CO2 Avg 758.00
CO2 726.00

String  00628 T 01212 Z 00761 z 007820
-----------------
humidity: 62.80
temperature: 70.16
CO2 Avg 761.00
CO2 7820.00

String  00614 T 012 H 00589 T 01220 Z
-----------------
humidity: 61.40
temperature: -145.84
CO2 Avg 1220.00
CO2 0.00

String  00571 T 01207 Z 00688 z 00659
-----------------
humidity: 57.10
temperature: 69.26
CO2 Avg 688.00
CO2 659.00

Without (mentally) stepping through your code your symptom description may suggest some RX buffer (63byte) roll-over which may be due to some timing issue.

Especially all these delay() calls won’t help.

Thank you for your response. That is helpful to know at least what is probably causing the problem!
Any advice on how to

  1. avoid buffer rollover ? I see a Serial.flush() command, should i use that after every loop to clear the buffer out?
  2. only publish every x minutes without using delay ? Or is it just a matter of where i am calling delays?

Serial1.flush() won’t help as it doesn’t actually flush the RX buffer. It just makes sure that the TX buffer gets pushed out completely before moving on.
To flush the RX buffer I usually do this

  while(Serial1.read() >= 0);

This would at least ensure, that you won’t have any “old” data piling up in the buffer till you find time to receive new data.
It’s hard to advise without knowing how your transmitter sends data (continous, in bursts, on demand, back-to-back, …).
But a safe bet - although potentially slower - would be to

  • flush the RX buffer
  • wait for your begin marker to be sent (without removing it from the buffer)
  • Serial1.readBytesUntil() to “await” the entire packet
  • optionally add a terminating '\0' to the buffer to make it a proper string
  • rinse, repeat

About delays: They are generally a bad idea as they just waste precious controller time that can be put to better use.
Adopting a non-blocking coding scheme (e.g. FSM) would be a better way.

One simple way to make your sendToCloud() non-blocking would look like this

void sendToCloud() {
  static uint32_t msLastTime = 0;
  if (millis() - msLastTime < 90000) return; // don't do anything until it's time to
  msLastTime = millis();

  char data[64];
  snprintf(data, sizeof(data)
          , "{ \"H\" : %.1f, \"T\" : %.1f, \"C\" : %.1f }"
          , humidity
          , temp1
          , co2avg
          );
  Particle.publish("Ubidots", data, PRIVATE);
}

You can also send bursts up to four events without any delay as long you don’t to do that again for at least four seconds (avg. 1 evente per second).
And instead of sending three events I’d incorporate all three data points into one single event (as shown above).

BTW, for your parsing you could also use this (unfortunately doesn’t work for float)

  int i, H,T,Z,z;
  i = sscanf(receivedChars, "H %d T %d Z %f z %d \r\n", &H, &T, &Z, &z);
  if (4 == i) {
    // all four items received so deal with it
  }
  else {
    Serial.printlnf("Only %d of 4 items received in\r\n%s", i, receivedChars); 
  }

Hi
The transmitter sends data in streaming Mode: Measurements are reported twice per second.

Can you point me to a project or example that uses this serial receipt method you mention, i am having a hard time visualizing how that would work.

Also I get an error here, but it looks correct to me.

snprintf(data, sizeof[data]
, “{ “H” : %.1f, “T” : %.1f, “C” : %.1f }”

lambda-expression in unevaluated context
expected ‘{’ before ‘,’ token

Sorry, that should be parenthesis not square brackets in sizeof(data) (corrected above).

Ah man i looked at that 20 times against another snippet example and didn’t catch that.

Removing the Delays seems to have fixed the issue so far. Leaving the original serial read routine intact.