Try SdFat, a Library for SD Cards

whg · May 23, 2018, 2:30pm

bench is one of several examples included with the SdFat library. I can see them when I use the Particle IDE and use the select libraries tab.

It is also on GitHub bench.

armor · May 23, 2018, 2:38pm

Got it!

whg · May 23, 2018, 4:14pm

I compiled and loaded bench into a photon with Ver 0.7.0 Firmware and verified that you can remove the SD between tests.

see this line

CARD REMOVED AND and INSERTED HERE <<=============

Here is the output:

FreeMemory: 19264
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE16G
Version: 8.0
Serial number: 0X91D1262A
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2817.16,19457,10927,11616

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3176.49,10784,10297,10318

Done
Type any character to start

CARD REMOVED AND and INSERTED HERE <<=============

FreeMemory: 19264
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE16G
Version: 8.0
Serial number: 0X91D1262A
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2802.89,23331,10925,11666

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3174.47,10667,10295,10320

Done
Type any character to start

warriorofwire · May 25, 2018, 4:48am

This library is <expletive_list/> amazing. Wired up an Adafruit micro sd card holder, loaded up the lib, pasted the benchmark code and it all worked first try. Thank you so much for sharing, @whg!
I’m using SPI1 and have a need to simply transfer bytes to SPI as fast as possible.

Using standard Photon.
Speed test shows weak speed. Any tips to attain that attractive 2-3 megabytes per second I see in this thread?
Is there any way to slave SPI to SPI1 via DMA or something? Otherwise I’ll need to write a double-buffer to ensure the transfer is limited only by the slower of SPI and SPI1. While that’s fun code to write, I’d rather not if I don’t have to.

Benchmark Output

Type is FAT32
Card size: 3.98 GB (GB = 1E9 bytes)

Manufacturer ID: 0X27
OEM ID: PH
Product: SD04G
              Version: 3.0
Serial number: 0X1F59046D
Manufacturing date: 7/2010

File size 5 MB
              Buffer size 32768 bytes
                                     Starting write test, please wait.
[I don't care about write, it's ~700 kb/second]

Starting read test, please wait.
read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
923.56,58532,21662,35482

EDIT: Speed issue is resolved as per your suggestion @whg. I’ll plan to do DMA double-buffer for high-throughput transfer. Quick benchmark (minimally refactored) below:

SanDisk Extreme Plus 32gb benchmark on SPI1

0000035245 [app.command.router] INFO: Active command set to SdBenchmark
0000035246 [app.command.sdbenchmark] INFO: Use a freshly formatted SD for best performance.
0000035246 [app.command.sdbenchmark] INFO: FreeMemory: 9896b
0000035261 [app.command.sdbenchmark] INFO: Type is FAT32
0000035261 [app.command.sdbenchmark] INFO: Card size: 31.91gb (gb = 1E9 bytes)
0000035261 [app.command.sdbenchmark] INFO: Manufacturer ID: 0x3
0000035262 [app.command.sdbenchmark] INFO: OEM ID: SD
0000035262 [app.command.sdbenchmark] INFO: Product: SP32G
0000035262 [app.command.sdbenchmark] INFO: Version: 8.0
0000035263 [app.command.sdbenchmark] INFO: Serial number: 0xb1bf8abb
0000035263 [app.command.sdbenchmark] INFO: Manufacturing date: 03/2012
0000035268 [app.command.sdbenchmark] INFO: File size 5mb
0000035269 [app.command.sdbenchmark] INFO: Buffer size 32768b
0000035269 [app.command.sdbenchmark] INFO: Starting write test.
0000035269 [app.command.sdbenchmark] INFO: write speed and latency
0000035270 [app.command.sdbenchmark] INFO:    speed      max      min      avg
0000035270 [app.command.sdbenchmark] INFO:   KB/Sec     usec     usec     usec
0000038397 [app.command.sdbenchmark] INFO: 1592.82    26880    19865    20554
0000038398 [app.command.sdbenchmark] INFO: Starting read test.
0000038398 [app.command.sdbenchmark] INFO: read speed and latency
0000038399 [app.command.sdbenchmark] INFO:    speed      max      min      avg
0000038399 [app.command.sdbenchmark] INFO:   KB/Sec     usec     usec     usec
0000041325 [app.command.sdbenchmark] INFO: 1702.23    20170    19192    19248
0000041326 [app.command.sdbenchmark] INFO: Benchmark complete.
0000041326 [app.command.router] INFO: Active command complete.

whg · May 25, 2018, 12:15pm

The problem is not SPI. Your card has huge read latency, an average of 35482 usec.

Looks like a fairly old Patriot card.

I think SPI has a max clock of 30 MHz so the max rate with overhead will be about 3 MB/sec. SPI1 is limited to 15 MHz so it is limited to much less.

In the above test I used a SanDisk Extreme card that has consistent latency.

Here is the result with a Samsung Pro+ 32GB card.

FreeMemory: 19264
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Version: 1.0
Serial number: 0X1550FE3
Manufacturing date: 10/2015

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2847.76,19497,10885,11486

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3009.51,11266,10728,10887

armor · September 11, 2018, 9:30am

@whg Sorry, back on this thread because I have been seeing a file read corruption issue and also stack overflow and potentially this is pointing to SDFat library.

I am using SD card to hold .bmp image files that are read and displayed on a colour TFT. There is also a event message FIFO buffer file held on SD which can be written to at any time and read from once per second both from the application thread.

I have recently implemented a new motherboard design with support for full speed SPI (100K pull-ups on the data lines) - to date (previous board revision) I have used half speed SPI and had no problems. With the new board I have up’d the SPI to full speed but have been experiencing 2 issues. First, corruption of the image files displayed on screen and second, stack overflow at certain points in the application. If I switch to half speed the problems disappear.

I have run the ‘bench’ application on the latest board and with a class 4 Sandisk SD card get the following results:

Type any character to start
FreeMemory: 18216
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SS16G
Version: 8.0
Serial number: 0XC60E436C
Manufacturing date: 4/2012

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2900.84,16640,10996,11282

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3166.39,10726,10308,10348

Done

Would you have any ideas what might be causing the issues with SPI bus speed and SD card reads? And possible solutions. I have some Sandisk Ultra SD cards on order to try but the ordinary SD card does not appear to be that slow!

armor · September 19, 2018, 6:14pm

Update for anyone using SD card to store data and experiencing corruption of the files on the SD card.

I have narrowed down the issue to being caused by an OTA flash occurring/finishing whilst the application had a file open for read. This then appeared to not be closed properly or the file index corrupted.

SD card cannot be opened at all on a Mac and with Windows 10 it shows some strange file meta data and is unable to open certain files on the SD card.

I have implemented the following generally around file read and write functions:

if (System.updatesPending())
{
     return error_code;    //cannot read or open file
}
else
{
    System.disableUpdates();
    Do SD file open and read/write here...
    System.enableUpdates();
}

Is there anything better I could implement?

More generally, in the loop() should I be detecting System.updatesPending() and then go straight to just looping in a safe state without changing any eeprom/writing to SD card, etc.?

ScruffR · September 19, 2018, 6:21pm

The better option than checking System.updatesPending() would be to subscribe to the respective event (firmware_update_pending) in a System.on() handler.

armor · September 19, 2018, 6:36pm

@Scuffr, is this because the System.updatesPending() function is not reliable? The documentation contains a To Do: note and a bit of a talk down on what this function does.

Generally would you recommend having firmware updates disabled until one is pending and then handling stopping or finishing current activity with a global flag and finally once all activity stopped enabling the update process?

ScruffR · September 19, 2018, 7:14pm

In parts, but mainly because an event can inform you of any change in state asynchronously. It also informs you of the fact even when System.disableUpdates() was set but System.updatesPending() will only ever go true once the OTA update has already been started, which won't happen when disabled (at least that was the behaviour when the note was added).

In the event handler you can also encapsulate any actions that have to be done to ensure a safe state, and won't need to wait for your normal flow to reach the place where that's done.

warriorofwire · November 30, 2018, 2:35am

Absolutely brilliant. It just works, and I was sure I was going to have to do gymnastics.

armor · December 10, 2018, 6:55pm

A bit of advice - leaving SD files open when a system update occurs (reset) can leave the file index corrupted and then unreadable. I have implemented checks for update pending before opening files and then I have System.disableReset() before opening and System.enableReset() after closing.

bare.foot · May 29, 2019, 4:50am

Hi peekay, I didn’t find with wich solution zou ended up.
Have here the same problem. Everything works well till it comes to sleep.
Any ideas?

peekay123 · May 29, 2019, 12:12pm

@bare.foot, I didn’t continue work on that issue (too much to do, too little time). Perhaps @rickkas7 or @ScruffR have some insights.

ScruffR · May 29, 2019, 12:26pm

Have you followed the entire thread?
As it seems raised issues were addressed in here, which one are you still struggling with after you applied all strategies suggested here?

bare.foot · May 29, 2019, 11:42pm

Hey there,

I am sorry for botherig you. I just should have make the basic checks.
I thought about this yesterday and it look’s like i have just to much consumer on the 3V3 trace.
So I hooked up the SD adapter to a breadboard and everything works.
Thanks anyway.

EDIT: Hey again.
I guess I find out now what was causing those problems.
I use the electron with the asset tracker kit v2 as an automated datalogger for field application. I use the AssetTracker (0.0.10) library to manage the GPS handling. Each cycle I will write every second to the SD card which is attached to A3 - A5 + D5 as CS. As A3 - A5 is also connected to the gyro I didn’t realize that I have to set the A2 pin (CS gyro) always high. Thought as long as I don’t ask for data from the gyro it will be quiet. So i decided to make this pin always high to keep the gyro quiet. Thought the library should manage this.
Does this makes sense to you guys?
I will probably just cut the traces to the gyro to free up some more pins as I really could need a bit more. This should give me tow more pins. A2 and WKP or A7.
Didn’t have a deep look into the librarys for this tracker but maybe I have also to use another library to make things work better or so…

Topic		Replies	Views
Micro SD card support Libraries	5	3575	August 16, 2016
SDFat File Opening Error, 0x4,0xFF Libraries	4	1254	November 5, 2018
SDfat library not working on particle electron Firmware	21	4701	January 6, 2018
SDFAT 1.0.16, ParticleSoftSerial 0.0.8 Firmware	11	1915	January 17, 2020
How to get a local library work? Firmware	8	930	July 27, 2021

Try SdFat, a Library for SD Cards

Related Topics