Try SdFat, a Library for SD Cards


#142

bench is one of several examples included with the SdFat library. I can see them when I use the Particle IDE and use the select libraries tab.

It is also on GitHub bench.


#143

Got it!


#144

I compiled and loaded bench into a photon with Ver 0.7.0 Firmware and verified that you can remove the SD between tests.

see this line

CARD REMOVED AND and INSERTED HERE <<=============

Here is the output:

FreeMemory: 19264
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE16G
Version: 8.0
Serial number: 0X91D1262A
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2817.16,19457,10927,11616

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3176.49,10784,10297,10318

Done
Type any character to start

CARD REMOVED AND and INSERTED HERE <<=============

FreeMemory: 19264
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE16G
Version: 8.0
Serial number: 0X91D1262A
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2802.89,23331,10925,11666

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3174.47,10667,10295,10320

Done
Type any character to start


#145

This library is <expletive_list/> amazing. Wired up an Adafruit micro sd card holder, loaded up the lib, pasted the benchmark code and it all worked first try. Thank you so much for sharing, @whg!
I’m using SPI1 and have a need to simply transfer bytes to SPI as fast as possible.

  • Using standard Photon.
  • Speed test shows weak speed. Any tips to attain that attractive 2-3 megabytes per second I see in this thread?
  • Is there any way to slave SPI to SPI1 via DMA or something? Otherwise I’ll need to write a double-buffer to ensure the transfer is limited only by the slower of SPI and SPI1. While that’s fun code to write, I’d rather not if I don’t have to.
Benchmark Output
Type is FAT32
Card size: 3.98 GB (GB = 1E9 bytes)

Manufacturer ID: 0X27
OEM ID: PH
Product: SD04G
              Version: 3.0
Serial number: 0X1F59046D
Manufacturing date: 7/2010

File size 5 MB
              Buffer size 32768 bytes
                                     Starting write test, please wait.
[I don't care about write, it's ~700 kb/second]

Starting read test, please wait.
read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
923.56,58532,21662,35482

EDIT: Speed issue is resolved as per your suggestion @whg. I’ll plan to do DMA double-buffer for high-throughput transfer. Quick benchmark (minimally refactored) below:

SanDisk Extreme Plus 32gb benchmark on SPI1
0000035245 [app.command.router] INFO: Active command set to SdBenchmark
0000035246 [app.command.sdbenchmark] INFO: Use a freshly formatted SD for best performance.
0000035246 [app.command.sdbenchmark] INFO: FreeMemory: 9896b
0000035261 [app.command.sdbenchmark] INFO: Type is FAT32
0000035261 [app.command.sdbenchmark] INFO: Card size: 31.91gb (gb = 1E9 bytes)
0000035261 [app.command.sdbenchmark] INFO: Manufacturer ID: 0x3
0000035262 [app.command.sdbenchmark] INFO: OEM ID: SD
0000035262 [app.command.sdbenchmark] INFO: Product: SP32G
0000035262 [app.command.sdbenchmark] INFO: Version: 8.0
0000035263 [app.command.sdbenchmark] INFO: Serial number: 0xb1bf8abb
0000035263 [app.command.sdbenchmark] INFO: Manufacturing date: 03/2012
0000035268 [app.command.sdbenchmark] INFO: File size 5mb
0000035269 [app.command.sdbenchmark] INFO: Buffer size 32768b
0000035269 [app.command.sdbenchmark] INFO: Starting write test.
0000035269 [app.command.sdbenchmark] INFO: write speed and latency
0000035270 [app.command.sdbenchmark] INFO:    speed      max      min      avg
0000035270 [app.command.sdbenchmark] INFO:   KB/Sec     usec     usec     usec
0000038397 [app.command.sdbenchmark] INFO: 1592.82    26880    19865    20554
0000038398 [app.command.sdbenchmark] INFO: Starting read test.
0000038398 [app.command.sdbenchmark] INFO: read speed and latency
0000038399 [app.command.sdbenchmark] INFO:    speed      max      min      avg
0000038399 [app.command.sdbenchmark] INFO:   KB/Sec     usec     usec     usec
0000041325 [app.command.sdbenchmark] INFO: 1702.23    20170    19192    19248
0000041326 [app.command.sdbenchmark] INFO: Benchmark complete.
0000041326 [app.command.router] INFO: Active command complete.

#146

The problem is not SPI. Your card has huge read latency, an average of 35482 usec.

Looks like a fairly old Patriot card.

I think SPI has a max clock of 30 MHz so the max rate with overhead will be about 3 MB/sec. SPI1 is limited to 15 MHz so it is limited to much less.

In the above test I used a SanDisk Extreme card that has consistent latency.

Here is the result with a Samsung Pro+ 32GB card.

FreeMemory: 19264
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Version: 1.0
Serial number: 0X1550FE3
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2847.76,19497,10885,11486

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3009.51,11266,10728,10887


#147

@whg Sorry, back on this thread because I have been seeing a file read corruption issue and also stack overflow and potentially this is pointing to SDFat library.

I am using SD card to hold .bmp image files that are read and displayed on a colour TFT. There is also a event message FIFO buffer file held on SD which can be written to at any time and read from once per second both from the application thread.

I have recently implemented a new motherboard design with support for full speed SPI (100K pull-ups on the data lines) - to date (previous board revision) I have used half speed SPI and had no problems. With the new board I have up’d the SPI to full speed but have been experiencing 2 issues. First, corruption of the image files displayed on screen and second, stack overflow at certain points in the application. If I switch to half speed the problems disappear.

I have run the ‘bench’ application on the latest board and with a class 4 Sandisk SD card get the following results:

Type any character to start
FreeMemory: 18216
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SS16G
Version: 8.0
Serial number: 0XC60E436C
Manufacturing date: 4/2012

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2900.84,16640,10996,11282

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3166.39,10726,10308,10348

Done

Would you have any ideas what might be causing the issues with SPI bus speed and SD card reads? And possible solutions. I have some Sandisk Ultra SD cards on order to try but the ordinary SD card does not appear to be that slow!


#148

Update for anyone using SD card to store data and experiencing corruption of the files on the SD card.

I have narrowed down the issue to being caused by an OTA flash occurring/finishing whilst the application had a file open for read. This then appeared to not be closed properly or the file index corrupted.

SD card cannot be opened at all on a Mac and with Windows 10 it shows some strange file meta data and is unable to open certain files on the SD card.

I have implemented the following generally around file read and write functions:

if (System.updatesPending())
{
     return error_code;    //cannot read or open file
}
else
{
    System.disableUpdates();
    Do SD file open and read/write here...
    System.enableUpdates();
}

Is there anything better I could implement?

More generally, in the loop() should I be detecting System.updatesPending() and then go straight to just looping in a safe state without changing any eeprom/writing to SD card, etc.?


#149

The better option than checking System.updatesPending() would be to subscribe to the respective event (firmware_update_pending) in a System.on() handler.


#150

@Scuffr, is this because the System.updatesPending() function is not reliable? The documentation contains a To Do: note and a bit of a talk down on what this function does.

Generally would you recommend having firmware updates disabled until one is pending and then handling stopping or finishing current activity with a global flag and finally once all activity stopped enabling the update process?


#151

In parts, but mainly because an event can inform you of any change in state asynchronously. It also informs you of the fact even when System.disableUpdates() was set but System.updatesPending() will only ever go true once the OTA update has already been started, which won’t happen when disabled (at least that was the behaviour when the note was added).

In the event handler you can also encapsulate any actions that have to be done to ensure a safe state, and won’t need to wait for your normal flow to reach the place where that’s done.


#152

Absolutely brilliant. It just works, and I was sure I was going to have to do gymnastics.


#153

A bit of advice - leaving SD files open when a system update occurs (reset) can leave the file index corrupted and then unreadable. I have implemented checks for update pending before opening files and then I have System.disableReset() before opening and System.enableReset() after closing.