Try SdFat, a Library for SD Cards

whg · December 31, 2015, 6:58pm

I have published a first version of SdFat. I would appreciate feedback.

This is the first library I have published for Particle. I assume it will be at the bottom of the list of libraries. It appears as my library so I can’t be sure but looks to be public:

Published Library
SdFat 0.0.2
An SD card library for Particle.

SdFat evolved from a library I wrote in 2008 for Arduinos with 1KB of RAM. In 2009 the Arduino company used SdFat as the core of the “Standard SD Library”. The 2009 version of SdFat has been ported to many systems, including Particle devices.

This version of SdFat has many new features such as long file names. I support DMA on all SPI interfaces.

I have tested SdFat on Photon and P1. SdFat compiles for Core and Electron but I can’t test since I have no hardware. Please let me know if SdFat has problems on Core or Electron.

There are a number of example. I suggest you start with TryMeFirst.cpp. This is a simple read/write example and the functions in this example are sufficient for many applications.

To try this example in the Web IDE go to libraries, find SdFat, select the TryMeFirst.cpp file and hit the “use this example” button.

SdFat has many features so you should download html documentation from GitHub. Doxygen html documentation is in the doc folder.

Here is a list of example file I have tested on P1/photon.

bench.cpp - Benchmark SD read/write speed.

DirectoryFunctions.cpp - Demonstrate chdir(), ls(), mkdir(), and rmdir().

LowLatencyLogger.cpp - High speed binary data logger.

OpenNext.cpp - Open all files in a directory.

ReadCsvArray.cpp - Function to read an array from a csv file.

ReadCsvFields.cpp - Function to read csv fields.

SdFormatter.cpp - Format an SD card according to the SD standard.

SdInfo.cpp - Display information about an SD card.

Timestamp.cpp - Shows how to time-stamp files.

TryMeFirst.cpp - A simple read/write example.

VolumeFreeSpace.cpp - Determine free space in a volume.

ScruffR · December 31, 2015, 7:26pm

I can confirm that your lib is public
I’ll definetly give it a test on Core and Electron once I get home to my Particles

Anything you want tested in particular?

Always nice to have programmers of your caliber on board!

tjp · December 31, 2015, 7:29pm

I will give it a try right now, I’m struggling with truncating a file using the other library! lol

Thanks, @whg!

whg · December 31, 2015, 7:57pm

On core and electron I need to know if the low level SPI works. I like to run the bench.cpp example. On core you will need to reduce the transfer size to less than 32KB.

Most of the SdFat library was not modified. In fact I use the basic FAT library for USB flash keys and USB hard drives. I also use it on ChibiOS/RT.

It’s amazing what kills programs. I found that Particle kept the ugly Arduino preprocessor in the Web IDE. At least you can disable it with this.

#pragma SPARK_NO_PREPROCESSOR

SdFat has almost all of the Arduino SD.h features but I don’t use some of them in examples. Like this style open

file = SD.open("name.ext", FILE_WRITE);

There are lots of Arduino based SdFat example on GitHub. Very little needs to be done to port these but I already probably have enough examples.

tjp · December 31, 2015, 9:31pm

Well, I’m having great success on my Photon. Unfortunately, I don’t have a Core or Electron either. I do appreciate all the work you’ve put into the library over the years! Great stuff! Thanks!

peekay123 · January 1, 2016, 3:39am

@whg, on (beta) Electron, running bench.cpp, I get:

Use a freshly formatted SD for best performance.

Type any character to start
FreeMemory: 27908

Type is FAT32
Card size: 8.04 GB (GB = 1E9 bytes)

Manufacturer ID: 0X73
OEM ID: BG
Product: NCard
              Version: 1.0
Serial number: 0X25176021
Manufacturing date: 5/2012

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2084.86,476465,10913,15660

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3111.02,11402,10406,10530

Well done!

peekay123 · January 1, 2016, 3:40am

@whg, I’ll try running the code on a Bluz tomorrow as well.

bpr · January 1, 2016, 6:31am

@whg, Excellent! (using photon only)

whg · January 1, 2016, 4:40pm

Thanks for testing with electron. The fact it works is speaks well for the Particle firmware HAL.

peekay123 · January 6, 2016, 5:58pm

@whg, on latest DEVELOP branch and running on Photon, the numbers have dropped substantially:


FreeMemory: 32148
Type is FAT32
Card size: 8.04 GB (GB = 1E9 bytes)

Manufacturer ID: 0X73
OEM ID: BG
Product: NCard

Version: 1.0
Serial number: 0X25176021
Manufacturing date: 5/2012

File size 5 MB

Buffer size 32768 bytes

Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
657.61,70193,48375,49767

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
683.51,49523,47794,47936

Done

Not sure what’s happened!?

whg · January 6, 2016, 8:17pm

I downloaded the latest DEVELOP branch and ran bench with the version of SdFat from GitHub

Here are the results with a 16GB SanDisk Extreme MicroSD.

FreeMemory: 24728
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE16G
Version: 8.0
Serial number: 0XD4D2262A
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2889.06,19891,10623,11326

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3270.35,10400,10002,10021

One puzzle is why you have:

FreeMemory: 32148

But I have:

FreeMemory: 24728

I downloaded the DEVELOP firmware at 11:50 PST 1/6/2016.

whg · January 6, 2016, 8:39pm

I ran bench with:

#define SPI_CONFIGURATION 2

Instead of:

#define SPI_CONFIGURATION 0

This selects byte at a time SPI instead of DMA for SD blocks.

I get results very close to yours.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
677.47,60289,47590,48342

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
682.85,50727,47488,47990

peekay123 · January 6, 2016, 9:05pm

@whg, crap! I now realize I was running config 2 instead of 0!!! Thanks for checking up and setting me straight

Jerware · January 6, 2016, 11:01pm

@whg I requested an SdFat port on this forum in Jun '14, and have been patiently waiting ever since! I’m overjoyed. Thank you.

joe4465 · January 7, 2016, 1:28pm

Thanks for this, it works really well for me.

Could you post an example of the correct way to completely delete everything on the SD card?

Cheers

whg · January 7, 2016, 1:52pm

This example is for Arduino. Just look at the wipe() call. wipe() does a quick format of a previously formatted card. It just clears the root directory and the File Allocation Tables, FATs.

You should format SD cards using the SD Association formatter or the example included with SdFat. Correct format is important for micro-controllers since little cache is used in the implementation of SdFat.

// Example to wipe all data from an already formatted SD.
#include "SdFat/SdFat.h"
const int chipSelect = SS;

SdFat sd;

void setup() {
  int c;
  Serial.begin(9600);
  while (!Serial) {}  // wait for Leonardo
  Serial.println("Type 'Y' to wipe all data.");
  while ((c = Serial.read()) <= 0) {}
  if (c != 'Y') {
    sd.errorHalt("Quitting, you did not type 'Y'.");
  }
  if (!sd.begin(chipSelect)) {
    sd.initErrorHalt();
  }
  // Use wipe() for no dot progress indicator.
  if (!sd.wipe(&Serial)) {
    sd.errorHalt("Wipe failed.");
  }
  // Must reinitialize after wipe.
  if (!sd.begin(chipSelect)) {
    sd.errorHalt("Second init failed.");
  }
  Serial.println("Done");
}

void loop() {
}

Jerware · January 7, 2016, 7:43pm

I’ve successfully brought most of my project from Teensy to Photon, thanks to your SdFat port, with the exception of errors surrounding this function ripped from an old Adafruit Arduino sketch:

uint16_t read16(SdFile& f) {
  uint16_t result;
  ((uint8_t *)&result)[0] = f.read(); // LSB
  ((uint8_t *)&result)[1] = f.read(); // MSB
  return result;
}

When it comes to references and pointers, I’ll admit I’m out of my league. The compile errors indicate that “SdFile was not declared in this scope,” and also oddly that “read16 cannot be used as a function.” Perhaps the latter error is just a factor of it not compiling to begin with? Any suggestions?

FYI SdFile initializes fine at the top of my sketch as

SdFile myFile; // set filesystem

and the function in question is called using that “myFile” object.

ScruffR · January 7, 2016, 8:19pm

Could this have to do with a known preprocessor issue?

Try #pragma SPARK_NO_PREPROCESSOR (which in turn requires you to #include "application.h" and add function prototypes for your local functions where needed)

Jerware · January 7, 2016, 8:59pm

49 function prototypes later.... and it's working! (Man, what a pain.) Thank you for the help!

whg · January 8, 2016, 4:00pm

Jerware,

You can speed up functions like read16() by reading multiple bytes instead of byte at a time.

This function is about 50% faster. 620 KB/sec vs 403 KB/sec.

uint16_t read16(SdFile& f) {
  uint16_t result;
  f.read(&result, 2);
  return result;
}

Somtimes you need to swap bytes due to big/little endian problems but it is still faster to read multiple bytes and use shift/or.

Here is the time in microseconds for reading 20,000 uint16_t values. I ran the byte test twice to make sure there were no cache or SD busy problems.

Writing test.bin
byte: 99533
word: 64339
byte: 99193
done.

Here is the test program.

#include "SdFat/SdFat.h"
SdFat sd;
SdFile myFile;
const uint8_t chipSelect = SS;

uint16_t read16byte(SdFile& f) {
  uint16_t result;
  ((uint8_t *)&result)[0] = f.read(); // LSB
  ((uint8_t *)&result)[1] = f.read(); // MSB
  return result;
}

uint16_t read16word(SdFile& f) {
  uint16_t result;
  f.read(&result, 2);
  return result;
}

void setup() {
  Serial.begin(9600);
  while (!Serial) {}  // wait for Leonardo
  Serial.println("Type any character to start");
  while (Serial.read() <= 0) {}

  if (!sd.begin(chipSelect, SPI_FULL_SPEED)) {
    sd.initErrorHalt();
  }

  if (!myFile.open("test.bin", O_RDWR | O_CREAT | O_TRUNC)) {
    sd.errorHalt("opening test.bin for write failed");
  }
  // if the file opened okay, write to it:
  Serial.println("Writing test.bin");
  for (uint16_t i = 0; i < 20000; i++) {
    myFile.write(&i, sizeof(i));
  }
  myFile.flush();
  delay(1000);
  myFile.rewind();
  uint32_t m = micros();
  for (uint16_t i = 0; i < 20000; i++) {
   if (read16byte(myFile) != i){
     Serial.println("bad read16byte");
     return;
   }
  }
  m = micros() - m;
  Serial.print("byte: ");
  Serial.println(m);

  myFile.rewind();
  m = micros();
  for (uint16_t i = 0; i < 20000; i++) {
   if (read16word(myFile) != i){
     Serial.println("bad read16word");
     return;
   }
  }
  m = micros() - m;
  Serial.print("word: ");
  Serial.println(m);

  myFile.rewind();
  m = micros();
  for (uint16_t i = 0; i < 20000; i++) {
   if (read16byte(myFile) != i){
     Serial.println("bad read16byte");
     return;
   }
  }
  m = micros() - m;
  Serial.print("byte: ");
  Serial.println(m);
  
  // close the file:
  myFile.close();
  Serial.println("done.");
}

void loop() {
}

Topic		Replies	Views
Test of the SdFat library on Photon Libraries	11	4404	December 6, 2016
SDFat Library and SPI1 Libraries	6	3055	September 24, 2016
Micro SD card support Libraries	5	3630	August 16, 2016
[SDFat] Logging data from sensor to sd card Libraries	10	1883	July 15, 2019
SDFat usage on Particle Photon Troubleshooting	6	884	May 12, 2019

Try SdFat, a Library for SD Cards

Related topics