Library for logging to SD card: SdCardLogHandlerRK

Firmware Libraries
42

What did you connect the second USB port to on the E Series module? The u-blox USB port pins? That definitely will not work. That’s directly connected to the modem and is not accessible to user firmware.

The SDFAT library is meant to use an SPI-connected SD card adapter. It will definitely not work with a USB connected one, on any USB port. That would require a USB host mode and a device driver for the SD card. Neither exist and would be exceptionally difficult to implement. For all practical purposes, impossible.

43

@rickkas7 Thank you for the response! As far as I am aware, yes, the second USB port is connected to the u-blox USB port pins. I will double check with the person who designed the board, but I’m about 90% positive that’s what happened.

44

@rickkas7 Okay, I checked with the designer and he said that the second USB port is NOT connected to the u-blox USB port pins. However, the two USB ports are exactly the same. He was aware that while using one of the ports the other could not be used because they are the same. We are going to see if there are any other free pins we can use to make this work.

45

@rickkas7 hopefully not bugging you (or the community) but just wanted to follow up about the idea of multiple log handlers including SD log handlers on top of reading and reading and writing to regular files per earlier post. Would there be any theoretical conflicts or edge cases to worry about?

Thank you

46

It’s an unusual case to use two different log handlers like that, but as long as they are writing to two different directories as you have done it should work properly, I think. I never tested it, but it probably will work.

47

@rickkas7 Thanks for making this great library! it’s been super helpful.

I am trying to make a data logging system that can write data to an sd card at upwards of 100Hz, and I have all of the code working but it doesn’t seem to want to log faster than 30Hz or so. Looking at the data written to the sd card it appears that everytime I call the “logHandler.loop();” line in my loop it takes around 30ms, which bottlenecks the sampling speed. Do you know of any way I can shorten this time?

I am completely new to all of this so sorry if this has already been mentioned or I bothered you.

Thanks

48

It’s probably going to be hard to get a constant 100 Hz sampling rate out of the loop thread. What I would do is:

  1. Use a worker thread to get the samples at a constant rate. Save these to a FIFO queue in RAM. The queue does not have to be that long, just long enough to buffer during long SD card operations.

  2. Make sure you use the logHandler.withSyncEveryEntry(false); option. The default is true, which flushes the log after every write which assures that data is not lost, but it’s a super expensive operation that takes a long time to execute, so you definitely don’t want to do it on every write. Permanent writes to flash also occur every 512 bytes, so you’ll hit that frequently enough to keep your data safe.

  3. Grab the samples out of the queue from loop() and write the log entries there, instead of from the sampling thread.

2 Likes
49

@rickkas7 Have you seen this yet? Figured it may be something that you may find useful.

50

@RWB, MRAM is very nice but like FRAM, it is expensive! However, for commercial/industrial applications where reliability is key, the cost may not be an issue.

1 Like
51

I’m using SdCardLogHandlerRK to log sensor data. It works great except I need to write a header line every time a new file is created. Any suggestions?

52

The easiest solution will be to copy the 3 files in the library directly into your project, remove the entry from project.properties, and edit the .cpp file here:

bool SdCardPrintHandler::openLogFile() {
	const char *name = getName(lastFileNum);
	if (curLogFile.open(&logsDir, name, O_RDWR | O_APPEND | O_CREAT)) {
		fileNums.insert(lastFileNum);
		DEBUG_HIGH(("using log file %s", name));
		return true;
	}
	else {
		DEBUG_HIGH(("failed to open log file %s", name));
		return false;
	}
}

After the first DEBUG_HIGH add something like:

if (curLogFile.fileSize() == 0) {
  const char *hdr = "Put file header here\n";
  curLogFile.write(hdr, strlen(hdr));
}
53

Perfect -that works issue solved. Thanks for your help.