SD Data Logger - Problems At Higher Speed

Hi All,
I am working on a project that receives data over serial to a particle and writes it to an sd card. I found several libraries, mostly by @rickkas7, that were hugely helpful in getting started and things seem to be working pretty well until the data rate increases and then I start to see problems with the data being received/recorded correctly.

Based on a couple quick calcs, I believe my max data rate coming in to the Argon over serial is about 15,000 bytes/s which I believe is doable from the reading I’ve done but please correct me if I am incorrect. I assume it is a problem with the buffer but am not sure where to go from there.

My hardware includes the adafruit RTC sd card logger, adafruit display board with a couple buttons and an Argon.

See code below:

#include "oled-wing-adafruit.h"
#include "SdFat.h"
#include "Particle.h"
#include "SerialBufferRK.h"
#include "Adafruit_RTClib_RK.h"
#include "AdafruitDataLoggerRK.h"
#include <string>

SYSTEM_THREAD(ENABLED);

const int SD_CHIP_SELECT = D5;



SdFat sd;
File myFile;
SerialLogHandler serialLogHandler;
OledWingAdafruit display;
RTC_PCF8523 rtc;
RTCSynchronizer rtcSync;
DateTime now;




// Allocate a serial buffer of 256 bytes attached to Serial1
// call serBuf.read() instead of Serial1.read()
SerialBuffer<4096> serBuf(Serial1);




int timestamp = millis();

bool cardReady = false;
bool startupLoop = true;
bool logging = false;

char fileName[] = "LOGGER00.txt";




void setup()
{
	rtcSync.setup();

// setup the oled display and clear
	display.setup();
	display.clearDisplay();
	display.display();


//turn on serial connection
	Serial.begin();
	Serial1.begin(230400);

	// You must call serBuf.setup() from setup!
	serBuf.setup();

// get dateTime from real time clock
	now = rtc.now();
	timestamp = millis();

	// Initialize the library
	if (!sd.begin(SD_CHIP_SELECT, SPI_FULL_SPEED))

	{
		updateDisplayString("ERROR: SD Init", true);
		return;
	}
	else
	{
		updateDisplayString("SD init succesful", true);
		delay(500);
	}

	getFilename();

	// Open with fileName
	if (!myFile.open(fileName, O_CREAT | O_WRITE | O_AT_END))
	{
		updateDisplayString("ERROR: Open SD", true);
		return;
	}
	else
	{
		updateDisplayString("SD open succesful", true);
		cardReady = true;
		delay(500);
	}
}

void loop()
{
	if (cardReady == true) //check that there isn't a problem with the card
	{
		if (logging == false)
		{
			updateDisplayString("Press button A\nto start logging", false);
			if (display.pressedA())
			{
				logging = true;
				updateDisplayString("Logging In Progress,\nHold Button B to Stop", false);
			}
		}
		else
		{
			if (display.pressedB())
			{
				updateDisplayString("Logging Stopped", true);
				int starttime = millis();
				while (millis() - starttime <= (3000))
				{
					String dataRow = serBuf.readStringUntil('^');
					if (dataRow.length() != 0)
					{
						String writeToSDCard = (String)millis() + "," + dataRow; //ToDo switch away from micros(), we expect 800hz data so micros is probably not enough resolution for timestamp
						writeToSD(writeToSDCard);
						Log.info(dataRow);
					}
				}
				writeToSD("END OF FILE"); 
				myFile.close(); //close the file now that nothing else is coming.  //ToDo change the setup of the loop so you can start new files without restering device

				//ToDo add in the file upload to web
				
				return;
			}

			// ToDo - it seems like this is taking too loing, data overflows expected format in high data rate events (impact)
			String dataRow = serBuf.readStringUntil('^');
			if (dataRow.length() != 0)
			{
				String writeToSDCard = (String)millis() + "," + dataRow; //ToDo switch away from micros(), we expect 800hz data so micros is probably not enough resolution for timestamp
				writeToSD(writeToSDCard);
				Log.info(dataRow);
			}
		}
	}
	else
	{
		updateDisplayString("ERROR: No SD Card", true);
		return;
	}

	display.loop();
	rtcSync.loop();
}

void updateDisplayString(String inputString, bool serialLog)
{
	display.clearDisplay();
	display.setTextSize(1);
	display.setTextColor(WHITE);
	display.setCursor(0, 0);
	display.println(inputString);
	if (serialLog == true)
	{
		Log.info(inputString); //also log to serial monitor
	}
	display.display();
}

void writeToSD(String dataRow)
{
	myFile.println(dataRow);
	myFile.flush();
}

void getFilename()
{
	// ToDo better way to make filenames. This can only do 100.
	for (byte i = 1; i <= 99; i++)
	{
		Log.info("loop num: %i", i);
		// check before modifying target filename.
		if (sd.exists(fileName))
		{
			// the filename exists so increment the 2 digit filename index.
			fileName[6] = i / 10 + '0';
			fileName[7] = i % 10 + '0';
			Log.info("filename already in use, adding to name: %s", fileName);
		}
		else
		{
			updateDisplayString((String)fileName, true);
			delay(2000);
			myFile.open(fileName, O_CREAT | O_WRITE | O_AT_END);
			printDate(now);
			myFile.write("Time,aX(mG),aY(mG),aZ(mG),|Vector|,vR(r/s),aR(r/s/s)");
			myFile.close();
			delay(2000);
			return;
		}
	}
}

void printDate(const DateTime &now)
{
	myFile.print(' ');
	myFile.print(now.year(), DEC);
	myFile.print('/');
	myFile.print(now.month(), DEC);
	myFile.print('/');
	myFile.print(now.day(), DEC);
	myFile.print(' ');
	myFile.print(now.hour(), DEC);
	myFile.print(':');
	myFile.print(now.minute(), DEC);
	myFile.print(':');
	myFile.print(now.second(), DEC);
	myFile.println();
}

I realized I didn’t explain what I am seeing when I say good data vs bad data.

The expected data format is csv data for timestamp, accelerometer and gyro. There is sometimes other text that comes across that doesn’t follow the same csv format but it is handled ok using just the end-line character. When the data goes “bad” it seems like several lines get merged together and format is no longer received as expected. Results are similar looking at the data on SDCard or in serial monitor.


“Good” Data

Time,aX(mG),aY(mG),aZ(mG),|Vector|,vR(r/s),aR(r/s/s)
1280679,Battery: 100%, avg ADC: 860
1286336,1,-1,8
1287693,-3,-3,10
1287967,Accel event. Phase: ACTVATE_DET, Thres:, 8, 175
1288973,0,-7,8
1290500,1,-9,-1
1292172,4,22,1
1292180,Accel event. Phase: MIN_ACT_DET, Thres:, 9, 252
1292187,Set Gyro state:  On
1292194,,,,16405,26441,23905
1292201,,,,308,300,236
1292208,,,,436,197,257
1292214,,,,466,120,241
1292221,,,,365,55,337
1292227,,,,218,45,412
1292234,,,,136,114,421
1292240,,,,181,271,272
1292247,,,,390,499,78
1292254,,,,744,710,439
1292261,,,,1509,743,552
1292267,,,,2269,293,368
1292274,,,,2084,93,552
1292281,,,,1677,660,930
1292288,,,,982,1163,809
1292295,,,,175,1452,2
1292303,,,,518,1330,799

“Bad” Data

1299562,,,,156,263,216,3781
1299569,,,,106,8386
1299577,,,,229,1349
1299584,,,,286,1534,3
1299591,,,,
1299600,,,,364,2079,32,,,436,2241,3067,,,3020,1,72,-11,749,272,7961
1299608,,,,732,30,3063
1299616,,,,1027,2
1299623,,,,15,1348
1299630,,,,45
1299638,,,,947,225,4974
1299647,,,,1344,650,720K_DET, Thres:, 1490,2282,5481
1299655,,,,1,243,,,474,2388,71480,219
1299665,,,,1054,1868,4789
1299672,,
1299680,,,,779,1034,3306
1299687,16,925,22,3058
1299695,,,,681,604,3102

Hey guys, any tips on this?

This could be part of the problem.
Can you be sure that you have a '^' at least once every 127 bytes? The default buffer length for Serial1 on Gen3 is 128 byte (I haven’t looked into the implementation of SerialBufferRK how @rickkas7 handles this).

You can also tweak the timeout for this command to fit your needs.

BTW, we ususally recommend to avoid String (mainly for heap fragmentation reason).
But if you need to use String you should preset it’s expected length to avoid the need for inflation and consequently relocation as the String outgrows its default starting size of 16 byte (then 32, 64, …)

Another tip would be to start with a stripped down test code that focuses solely on the issue at hand.

Great, thank you @ScruffR, these are helpful. Good advice about starting with the stripped down code, I’ll do that before pounding my head against the wall anymore :slight_smile: .

A couple more notes/questions:
The '^' character is my line break and I am fairly certain that for each data collection event it is included in the data sent from my sensor. What I am not as certain about is if there may be cases where the data line is longer than 127 bytes. Depending on the data event type, the data being sent by my sensor over to the particle is of varying lengths.

The varying length of the incoming data is what lead me to using String so that it could handle the length variation for me. I see from your comment that my approach is probably not a good way to handle this but I don’t know how to do it differently. Could you point me in the right direction for a better approach?

I guess you do have an estimate of the longest expected line you want to capture.
If so, you can just use a character array of that length (plus some margin) or use a pre-reserved (global) String like this

String sBuff;
...
void setup() {
  sBuff.reserve(128);
}

But I’d go with the char sBuff[128]; tho’ - this can also be a local variable without further issues.