SdFat Library: reading data from file

boron
Tags: #<Tag:0x00007fe21c23d060>

#1

I’m using a Boron and the Adalogger FeatherWing with this library to publish some sensor data to the cloud or record the data to the SD card if the Boron cannot connect to the cell network. That part is working just fine.

The tricky part comes when I want to read the data from the file after the Boron has reconnected and publish that data to the cloud, marking it in the file as confirmed so that lines from the file will not be published more than once.

Here is my basic setup:

File with example sensor data (relevant character for identifying lines is marked with ‘^’)

p08/21/2019 19:44:51,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:52,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:53,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:54,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:45:55,03.1,01.2,105.4,41.5,0983.0
                   ^

When data is written to the card, it is marked ‘p’ (for pending). My program works up to this point just fine

I then use this method, readLine, that I have added to the SdCardLogHandlerRK.cpp file to scan the file for the character ‘p’. If a ‘p’ is found, it will scan the next 48 characters (There are 47 characters in a line excluding the first ‘p’ label and I left one character for null termination). The 48th character encountered by the fgets method would be a line break anyway, which the method is set to stop at by default as a delimiter. After the data is read into the char* textLine, the program backtracks the cursor back to the ‘p’ label and writes over it with a ‘c’ (for confirmed).

String SdCardPrintHandler::readLine (char textLine[]) {

	FatFile tempFile;
	char currentChar; //will track what character the cursor is viewing

    //initialized array with known values so that I would know if it was modified
	for (int i = 0; i < 48; i++) {
		textLine[i] = char('o');
	}

    // return string for the same data being written to the array
	String ab = "";

    // opens the working directory
	if (logsDir.open(sd.vwd(), logsDirName, O_READ)) {

		FatFile tempFile;
		tempFile.openNext(&logsDir, O_RDWR | O_SYNC); //opens the most recently made file in the directory for reading and writing
		
		tempFile.rewind(); // sets cursor to beginning of file if not already
		int currentByte = tempFile.read(&currentChar, 1); // reads the first char in the file
		
		while (currentChar != 'p' && currentByte > 0) { // continues reading the next character until 'p' or EOF is found

			currentByte = tempFile.read(&currentChar, 1);
			
		}
		
        // if 'p' was found before EOF, read 48 bytes out of the file after 'p'
		if (currentChar == 'p') {

			tempFile.fgets(textLine, 48);
			tempFile.seekCur(-48);
			tempFile.write('c');  // marks data as confirmed published

		}

	}

	tempFile.close(); // close the file as I am done with it

	ab = ab + textLine; // append the empty string with the information in the char array
	
    return ab; // return string should have the same data as the char array

}

The method is called from the main program file in loop()

RTCLogger rtclogger;
int readTime;

void setup () {
    rtclogger.initialize();
    readTime = millis();

}

void loop() {
    if (millis() - readTime >= 10000) { // code runs every 10 seconds(ish)
        char arr[48]; // array to write data from file to

        // String to write array to, also calls method in RTCLogger class
        String stri = rtclogger.readLineFromPending(arr); 

        // publish the array and string up to the cloud (I had both in case one was working and the other wasn't)
        Particle.publish("line(chars): ", arr, PRIVATE);
        Particle.publish("line(string): ", stri, PRIVATE);

        // resets timer to the current time the device has been on
        readTime = millis();
    }
}

This is the method from RTCLogger that will call my readLine() method

SdCardPrintHandler printToPending(sd, SD_CHIP_SELECT, SPI_FULL_SPEED);

void RTCLogger::initialize () {

    pinMode(SD_CHIP_SELECT, OUTPUT); // Sets cs pin as an output

    printToPending.withLogsDirName("pending"); // changes working directory name to "pending"

}

String RTCLogger::readLineFromPending (char buf[]) {

    // s will contain data from file, also calls to the readLine method
    // passed buf array will also be populated with data from the file
    String s = printToPending.readLine(buf);

    return s; // return s after checking file for unpublished data

}

My program when given the example file I included earlier will successfully see the first character as ‘p’, replacing the ‘p’ with ‘c’, and publishing the line to the cloud.

The problem lies in that any further attempts to run this method in the same power cycle will return an array and String filled with ‘o’, indicating that it did not find ‘p’ in the file.

Here are the results:

  • File
c08/21/2019 19:44:51,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:52,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:53,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:54,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:45:55,03.1,01.2,105.4,41.5,0983.0
  • Publishes to console
line(string): oooooooooooooooooooooooooooooooooooooooooooooooo
line(chars): oooooooooooooooooooooooooooooooooooooooooooooooo
line(string): 08/21/2019 19:44:51,03.1,01.2,105.4,41.5,0983.0
line(chars): 08/21/2019 19:44:51,03.1,01.2,105.4,41.5,0983.0

The program will continue to do this, until I restart the boron. If I don’t change the file before the program runs again, the program will then skip the first line, as it is supposed to, now that the line starts with ‘c’ and not ‘p’. It will then continue on down to the second line, where it will see ‘p’, and successfully handle the second line, but will fail on reading any further lines.

  • File after restarting the boron
c08/21/2019 19:44:51,03.1,01.2,105.4,41.5,0983.0
c08/21/2019 19:44:52,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:53,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:44:54,03.1,01.2,105.4,41.5,0983.0
p08/21/2019 19:45:55,03.1,01.2,105.4,41.5,0983.0

If I restart the boron 3 more times, every line from the file would publish and have its ‘p’ replaced with a ‘c’. So the problem seems to lie in the 2nd call to this method and any subsequent calls, but the first call in the program works as intended.

If anyone can see where exactly I am making a mistake, any help would be greatly appreciated.

If anyone needs more information, please let me know and I will bring it forward. I’ve left out some code that I didn’t see as relevant, mostly network handling.

Thanks for reading


#2

Since you are initialising your string with o in here

you shouldn’t be surprised to see these in your output :wink:
And that gives you some place to start debugging since obviously some of the following conditions won’t be met and the initial values stay unaltered.
Just add some debugging Serial.print() statements that give you a snapshot of the used variables and conditions to see where the misbehaviour starts.

BTW, try to avoid String objects wherever you can to avoid heap fragmentation and potential hangups or crashes with long running code.


#3

I am surprised that my code does this only after the first run though. Why does it work on the first run, even if it has to scan through entire lines before getting to a valid line on the first try? To me, this means that the method is capable of parsing through several character and successfully stopping at the first ‘p’.

I have been having issues with the Serial monitor not working and have had little luck finding solutions, so I’ve been working around it. I’m not as interested in getting the Serial monitoring tool to work as I am getting that SD card to work, especially since I have also been having strange issues using the Particle Debugger. Since you believe that is the best approach, I will spend some time troubleshooting. Perhaps a fresh look will reveal the problem.

If that doesn’t work, I will probably try adding an hmi I have laying around and print the values to the screen and report back.

Thanks for the reply, I will try removing the String objects from my code as well


#4

If you have a Particle Debugger hardware you can use Serial1 and the FTDI feature of the debugger instead of USB Serial.


#5

I’ve figured out that there is something in the SdFat library that is not allowing any serial debugging. The serial port is never initializing for some reason.

This is the modified code with a ‘*’ for lines that involve the Serial debugging

RTCLogger rtclogger;
int readTime;

void setup () {
    rtclogger.initialize();
    readTime = millis();
*   Serial.begin();

}

void loop() {
    if (millis() - readTime >= 10000) { // code runs every 10 seconds(ish)
        char arr[48]; // array to write data from file to

        // calls method in RTCLogger class and passes array for storing data
        rtclogger.readLineFromPending(arr); 

        // publish the array and string up to the cloud (I had both in case one was working and the other wasn't)
        Particle.publish("line(chars): ", arr, PRIVATE);

        // resets timer to the current time the device has been on
        readTime = millis();
    }
}
void SdCardPrintHandler::readLine (char textLine[]) {

	FatFile tempFile;
	char currentChar; //will track what character the cursor is viewing

    //initialized array with known values so that I would know if it was modified
	for (int i = 0; i < 48; i++) {
		textLine[i] = char('o');
	}

    // opens the working directory
	if (logsDir.open(sd.vwd(), logsDirName, O_READ)) {

		FatFile tempFile;
		tempFile.openNext(&logsDir, O_RDWR | O_SYNC); //opens the most recently made file in the directory for reading and writing
		
		tempFile.rewind(); // sets cursor to beginning of file if not already
		int currentByte = tempFile.read(&currentChar, 1); // reads the first char in the file
		
		while (currentChar != 'p' && currentByte > 0) { // continues reading the next character until 'p' or EOF is found
*                       Serial.println("Char at Cursor: " + String(currentChar));
			currentByte = tempFile.read(&currentChar, 1);
			
		}
		
        // if 'p' was found before EOF, read 48 bytes out of the file after 'p'
		if (currentChar == 'p') {
*                       Serial.println("Found this: " + String(currentChar));
			tempFile.fgets(textLine, 48);
			tempFile.seekCur(-48);
			tempFile.write('c');  // marks data as confirmed published

		}

	}

	tempFile.close(); // close the file as I am done with it



}

For some reason, the serial port never initializes and I believe that it is the SdFat library interfering.

I’ve compiled and flashed a very simple program that initializes the Serial port and sends the millis() runtime over USB, which worked.

int x;

void setup() {
  Serial.begin();
  delay(500);
  Serial.println("Serial worked");
  x = millis();
}


void loop() {
  if (millis() - x >= 2500) {
    Serial.println(millis());
  }
}

I’ve been reading through the library and haven’t found anything particularly suspect. The library definitely uses the Serial object mostly for error handling in the SdFat.h class. Though I’m not sure how I should modify the class to see if this is the conflict I’m looking for.

Anybody know of common incompatibilities with certain libraries and Serial debugging, or what would prevent the Boron from using Serial over USB?


#6

If you have a Particle Debugger hardware you can use Serial1 and the FTDI feature of the debugger instead of USB Serial .

I had not considered this, thanks for pointing me to it.

I will look into it and try something, although it would be much more convenient to use Serial over USB if anyone knows of anything I can try there.


#7

I don’t know of a way a library would interfere with Serial in the way you describe.
Sure, a library could call Serial.end() (which is unlikely the case) or when running asynchronously spam the interface resulting in garbled output.

You could try putting Serial.begin() before you initialise rtclogger.