Test of the SdFat library on Photon

Firmware Libraries
1

I just did a performance test of the latest version of my SdFat library on Photon.

I am really pleased with Particle’s support for SPI and porting Arduino libraries. I was able to get a version of SdFat running with DMA transfers in about two hours.

Here are performance results for the bench example.

Type is FAT32
Card size: 31.91 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SE32G
Version: 8.0
Serial number: 0X838DE929
Manufacturing date: 9/2015

File size 10 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2655.22,20395,11506,12333
2658.04,21573,11514,12322

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3030.39,11146,10788,10814
3031.31,11148,10742,10809

Done

Write at 2,655 KB/sec and read at 3,030 KB/sec is better than I had hoped for. This is with large 32 KB transfers.

The port is a quick hack but I plan to clean it up and put the photon version on GitHub.

3 Likes
2

@whg, can you share the library with the community? :grinning:

3

+1, can’t wait to try this port, I need the high speed writing for my project

4

It’s going to take a bit longer.

I have found problems with the Photon SPI library and need to find workarounds. I am close but need to do lots of testing.

The main problem is the block transfer function callback for transfer complete happens before the last byte has been shifted out/in of the SPI data register. This is mainly a problem at low SPI clock rate so I don’t want to put in delays at high speed.

This is a common problem with full duplex implementations of DMA SPI. I hope particle fixes this.

There are also strange glitches on SCK when changing clock rate. SD cards are initialized at slow clock so I need to make sure I set SPI speed when the SD is not selected. This is more my problem than Particle.

1 Like
5

I have made lots of progress. I have workarounds for the SPI problems.

I hope to start experimenting with the Web IDE and GitHub soon.

Here are more results.

SdFat is designed to allow many SD cards. You can use both Photon hardware SPI interfaces with multiple cards on each interface in one program.

Software SPI allows any digital pins to be used. Probably not very needed.

All results are for 5MB files with 32768 byte transfers.

Results for first SPI interface with DMA .

SCK => A3
MISO => A4
MOSI => A5
SS => A2 (default)

// Declare as:
SdFat SD;
// or
SdFat SD(0);

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2799.74,18634,10880,11690

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3272.49,10366,10000,10016

Results for second SPI interface with DMA.

SCK => D4
MISO => D3
MOSI => D2
SS => D1

// Declare as:
SdFat SD(1);

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1618.70,29633,19579,20222

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1735.45,19377,18855,18877

Results for the first hardware SPI with no DMA. Arduino style byte transfers.

// Declare as: 
SdFatLibSpi SD;

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
672.89,55569,47887,48672

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
689.09,48509,47505,47555

Results for Software SPI. You can use any digital pins.

SdFatSoftSpi<MISO_PIN, MOSI_PIN, SCK_PIN> SD;

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
210.56,176224,154405,155568


read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
264.05,126261,124046,124099
3 Likes
6

Eager to use this code! The current SD Card library leaves a lot to be desired. lol Although, I’m grateful that it was ported at all :smile:

7

I wrote the core of the current SD library in 2009. It is a mod of an earlier FAT16 library for Arduinos with 1KB of RAM.

Arduino was upgraded to the 2KB RAM 328 chip in 2009 so I could support FAT32.

Arduino didn’t install bug fixes or new features. I thought of the 2009 library as a prototype so it is amazing that it is still being ported to new systems.

2 Likes
8

It’s mostly the wrapper that is hindering. I can’t seem to get it to do what I want lol

9

@whg I have installed the sdfat library that is now on the web IDE and covered over my project to use this. It does appear to perform much better than before. Thanks.

I have a couple of questions about using the sdfat and iostream?

Essentially, I am reading image files from SD card to put on a TFT display. This works well but I would like to log events to the SD card and then read them back to send them when cloud connected. Potentially this means having more than one file open at a time. This was not possible with the old sdfat library, I tried a collision avoidance flag but this wasn’t atomic enough and occasionally ended up with a block/failed read. Can this new library handle multiple files open and how would be the best way to do this. I have seen some use of fstream/iostream on C++ forums but I am a C++ beginner/novice.

10

Hello armor, did you make progress with storing events on the SD card and read them back to send them when cloud connected? I’m trying to make a demo prototype with this functionality

11

@nuevo

Yes, the basic outline of the design is as follows. I have 4 publish modes; none, send only, store and send and store only.
initBacklog() needs to be called at startup to either create a new events backlog file or truncate the file down if no events are in backlog. The log just has 2 pointers; one to the head and one to the tail, each write of the stored event (and local publish time) is stored on the SD card log with a simple ‘\n’ termination and read back as a string.

// call this to publish in place of Particle.publish()
void securePublish(const char *eventName, String eventData)

// handler for timer to regulate send events every PUBTIME mess
// sets isPublishNextBacklogEvent = true;
void publishBacklogEvent()

// function to publish next event from backlog if none then timer stopped - timer interrupts every PUBTIME
// called from loop() if isPublishNextBAcklogEvent is true
void publishBacklog()

// remove event string data from head of backlog
// actually just move the head pointer to the next in the log
// Return true = OK, false = error could not remove
bool deleteEventFromBacklog()

// function to add event (name and data ) string to tail of backlog
// Return true = OK, false = error could not add
bool addEventToBacklog(const char *_eName, String eData)

// function to initialise backlog file to manage growth or setup new file
// if no eventlog or has no events then open (new) file and truncate to just head and tail pointers
// Return true = OK, false = error init failed
bool initBacklog()
1 Like
12

Excellent!!! I’ll study this, baby steps…