Try SdFat, a Library for SD Cards

Firmware Libraries
42

Try another brand 32GB card. The fact the card is “UHS-I Class 10” is meaningless.

The card controller is immediately switched to SPI mode which is limited to 50 MHz and has nothing in common with the SDIO protocol. I have seen a few cards that just don’t work well in SPI mode.

The SD standard requires cards to ignore CRC in SPI mode but I have seen cards that default to CRC mode.

You could try enabling software CRC by editing SdFatConfig.h at about line 152 and changing USE_SD_CRC to two like this.

/**
 * To enable SD card CRC checking set USE_SD_CRC nonzero.
 *
 * Set USE_SD_CRC to 1 to use a smaller slower CRC-CCITT function.
 *
 * Set USE_SD_CRC to 2 to used a larger faster table driven CRC-CCITT function.
 */
#define USE_SD_CRC 2

One more sad fact, there are lots of counterfeit Samsung and SanDisk cards floating around. I bought a high end 8GB SanDisk card from Amazon sold by Amazon that was counterfeit.

Few cards are used in SPI mode so counterfeit cards are likely to have SPI problems.

One more Edit: I tested the Particle version of SdFat with a Samsung 32GB Pro microSD during development. It was excellent with great performance.

43

Sorry to sound the false alarm here but the code seems to work fine on the stock Photon board so clearly there’s something going on either with our chip select (the potential DAC pin issue discussed) or with how our traces run, etc. The signals appeared clean but we’ll now check for higher frequency noise.

I still can’t explain why the smaller cards work fine (even at high baud-rates) while the larger cards don’t–or why the card didn’t work in the past on the stock Photon, but at least I’ve got a starting point now for debugging.

As a side note for anyone curious, the 32GB card does now identify as a “Version 1.0” card whereas the 8GB card I use as a reference identifies as a “Version 3.0” card. Probably not significant but just thought I’d mention that.

Thanks for all the input!

44

I searched my boxes of SD cards I have been sent and found a 32GB Samsung EVO microSD.

It works like a charm.

Bench example. Using DAC1 as SD chip select with 4.9.

Type any character to start
FreeMemory: 30052
Type is FAT32
Card size: 31.44 GB (GB = 1E9 bytes)

Manufacturer ID: 0X1B
OEM ID: SM
Product: 00000
Version: 1.0
Serial number: 0X780282A4
Manufacturing date: 5/2014

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2991.43,25838,10555,10935

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3205.11,10563,10216,10227

Done

Don't be surprised at variation in SPI performance of cards. It's not size it's the controller. Manufacturers change SD controllers and flash chips frequently. Cards look the same but don't perform the same.

The competition is intense in the SD market so consumer cards are a real price/performance/life trade-off.

45

@whg
Thanks for sharing this new library - it does appear to be much faster and the long file names is a useful feature. I am using an SD card reader on the back of a TFT module to read images (.bmp) to then draw onto the screen. This works fine with either library. However, I would like to implement a store and forward buffer for Particle event messages (to cope more reliably with loss of Cloud connection) and therefore need to (potentially) have more than one file open at a time - something the old SD library cannot handle.

Can your version manage more than one file open at a time?
Is there a better way for me to write and read data records for the event logging - iostream? Sorry, I am not familiar with these C++ features.

Thanks

46

@Jerware

Did you get your reading of HTML file(s) from SD working for the Webduino? Would you be able to share how you did this - simple open and read and close or other C++ function iostream or stdio? Thanks

47

The only limit on the number of open files is memory. Each file object takes about 40 bytes except for the StdioStream class which has an additional 64 byte buffer.

Older versions ported from SD.h can have multiple files. The standard Arduino SD.h is a wrapper for a version of SdFat that I wrote over six years ago. The first version of the wrapper was limited to one open file. Comments in examples still mention the old limit

2 Likes
48

@armor Sure, here is a basic implementation that serves most file types. I found the code in the comments on this page.

// utilities for mime types, in order to provide correct file type info to web browser
char* mime_types=
  "HTM*text/html|"
  "TXT*text/plain|"
  "CSS*text/css|"
  "XML*text/xml|"
  "GIF*image/gif|"
  "JPG*image/jpeg|"
  "PNG*image/png|"
  "BMP*image/bmp|"
  "ICO*image/vnd.microsoft.icon|"
  "MP3*audio/mpeg|";

  static const uint16_t text_html_content_type = 4;  // index of standard HTM type.

  uint16_t get_mime_type_from_filename(const char* filename) {
    uint16_t r = text_html_content_type;
    if (!filename) {
      return r;
    }
    char* ext = strrchr(filename, '.');
    if (ext) {     // We found an extension. Skip past the '.'
      ext++;
      char ch;
      int i = 0;
      while (i < strlen(mime_types)) {
        // Compare the extension
        char* p = ext;
        ch = mime_types[i];
        while (*p && ch != '*' && toupper(*p) == ch) {
          p++; i++;
          ch = mime_types[i];
        }
        if (!*p && ch == '*') { 	// We reached the end of the extension while checking
          r = ++i;   // equality with a MIME type: we have a match. Increment i
          break;    	// to reach past the '*' char, and assign it to `mime_type'.
        } else { 	// Skip past the the '|' character indicating the end of a MIME type.
          while (mime_types[i++] != '|');
        }
      }
    }
    //  free(ext);
    return r;
  }

void failureCmd(WebServer &server, WebServer::ConnectionType type, char *url_tail, bool tail_complete)
{
  /* if we're handling a GET or POST, we can output our data here.
     For a HEAD request, we just stop after outputting headers. */
  if (type == WebServer::HEAD)
    return;

  // this checks if the requested url is perhaps a file that needs to be served
  SdFile filetosend;
  char* filename=url_tail;  //
  Serial.print("requesting file: ");
  Serial.println(filename);
  if (!filetosend.open(filename, O_READ))
  {
    server.httpFail();
    server.println("<html><body><h1>Requested Page not found.</h1></body></html>");
    return;
  }
  else if (filetosend.isFile()) {  // if not, file is not found, corrupt or else.
    Serial.print("Serving file: ");
    Serial.println(filename);
    uint16_t mimetype=get_mime_type_from_filename(filename);
    //send_error_code(server,mimetype, 200);
    server.println("HTTP/1.0 200 OK");
    server.print("Content-Type: ");
    // next part writes the content type (mime types)
    char ch;
    int i = mimetype;
    while ((ch = mime_types[i++]) != '|') {
     server.print(ch);
    }
    server.println("\n");
    // finished writing mimetypes,
    int16_t n;
    uint8_t buf[1000];//
    while ((n = filetosend.read(buf, sizeof(buf))) > 0) {
         server.write(buf,n);
    }
    Serial.println("File sent.");
    filetosend.close();
  }
  else Serial.println("file is corrupt or not found.");
}
49

@whg Thanks for confirming that - good news.

On the reading and writing of text (CSV) with CR at the end - which of the C++ functions is better iostream, fstream, stdio? Any advice? Thanks

50

I am not sure what you mean by “CR at the end”. Ending lines with only a CR is unusual.

Windows uses CR followed by a LF.

Linux and Unix use a LF.

The old classic Mac OS was one of the few systems to use a single CR for end of line.

Could you please clarify with more details about the CSV files you want to write/read.

Perhaps you could provide some example lines.

1 Like
51

@whg I’ve been using your library for a bit now and had it working great for a while on 0.4.8 for the electron, but when I upgraded to 0.5.1, I get a hard fault (SOS then one blink) when my code gets to the SdFAT portion. I get this fault even when using the TryMeFirst sketch (CS = D5, SPI_CONFIGURATION == 1). It doesn’t seem to have the issue with SPI_CONFIGURATION ==0 or soft spi though. Any ideas?

52

I can't reproduce the problem with CS = D5, SPI_CONFIGURATION == 1

I ran TryMeFirst and bench.

I downloaded the 0.5.1 firmware from gitHub and updated a photon:

put the Photon in DFU mode
cd modules
make PLATFORM=photon clean all program-dfu

I edited the configuration for TryMeFirst like this:

// Pick an SPI configuration.
// See SPI configuration section below (comments are for photon).
#define SPI_CONFIGURATION 1   <------------------------------Change from zero to one
//------------------------------------------------------------------------------
// Setup SPI configuration.
#if SPI_CONFIGURATION == 0
// Primary SPI with DMA
// SCK => A3, MISO => A4, MOSI => A5, SS => A2 (default)
SdFat sd;
const uint8_t chipSelect = SS;
#elif SPI_CONFIGURATION == 1
// Secondary SPI with DMA
// SCK => D4, MISO => D3, MOSI => D2, SS => D1
SdFat sd(1);
//const uint8_t chipSelect = D1;
const uint8_t chipSelect = D5;   <-------------------- Use D5
#elif SPI_CONFIGURATION == 2

I did the same with bench. Here is the result:

FreeMemory: 29700
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SL16G
Version: 8.0
Serial number: 0X91203A25
Manufacturing date: 4/2014

File size 5 MB
Buffer size 32768 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1571.21,105882,20108,20822

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1699.91,20389,19242,19276

Done
Type any character to start

There have been major changes in the SPI driver. I have a bug fix/workaround in SdFat for a bug in 0.4.x.

You could remove it and see if that helps.

Edit SdSpiParticle.cpp and use the following code for the two functions with workarounds.

uint8_t SdSpi::receive(uint8_t* buf, size_t n) {
  spiPtr[m_spiIf]->transfer(0, buf, n, 0);
  return 0;
}

void SdSpi::send(const uint8_t* buf , size_t n) {
  spiPtr[m_spiIf]->transfer(const_cast<uint8_t*>(buf), 0, n, 0);
}
53

I can confirm that this problem is not solved. I tried firmware V0.5.2 and V0.5.1, both with and without the code you suggested above. But the Electron keeps generating Hard faults. Is there something I can do to help you find the issue? (Except for sending you an Electron) ;). Can it have something to do with pin D5 also being the Slave select pin for the hardware SPI peripheral?

Edit: Nope, Using pin C5 instead of D5 does not solve the problem…

54

There is no way I can guess what is happening. The SdFat code does not have any code specifically for Electron.

The only places where the code might be different is where the SPI interface count is defined in SdFatConfig.h

#if defined(PLATFORM_ID)
#if Wiring_SPI1 && Wiring_SPI2
#define SPI_INTERFACE_COUNT 3
#elif Wiring_SPI1
#define SPI_INTERFACE_COUNT 2
#endif  // Wiring_SPI1 && Wiring_SPI2
#endif  // defined(PLATFORM_ID)

And where pointers to the SPI classes are defined in SdSpiParticle.cpp

static SPIClass* const spiPtr[] = {
  &SPI
#if Wiring_SPI1
  , &SPI1
#if  Wiring_SPI2
  , &SPI2
#endif  // Wiring_SPI2
#endif  // Wiring_SPI1
};

The Electron has Wiring_SPI2 defined as one. Photon does not.

I ordered an Electron 3G. I always wanted to check out Electron but I couldn’t get a 3G Electron when I was porting SdFat to Particle. Should be here soon.

1 Like
55

@whg I am trying to run the TryMeFirst example on an Electron with 0.5.2 and keep getting the following error:

Can’t access SD card. Do not reformat.
No card, wrong chip select pin, or SPI problem?
SD errorCode: 0X1,0XFF

I am using a 8GB Kingston micrSD card adapter wired to SCK => D4, MISO => D3, MOSI => D2, SS => D5 (SPI_CONFIGURATON 1)

#define SPI_CONFIGURATION 1

//------------------------------------------------------------------------------
// Setup SPI configuration.
#if SPI_CONFIGURATION == 0
// Primary SPI with DMA
// SCK => A3, MISO => A4, MOSI => A5, SS => A2 (default)
SdFat sd;
const uint8_t chipSelect = SS;
#elif SPI_CONFIGURATION == 1
// Secondary SPI with DMA
//SCK => D4, MISO => D3, MOSI => D2, SS => D1
SdFat sd;
const uint8_t chipSelect = D5;
#elif SPI_CONFIGURATION == 2

Notice that if have SdFat sd; and not sd(1); (in the rest of the code you use sd and not sd(1).

Have tried connecting to A1-A5 (SPI=0) and same error. I have tried with HALF and FULL SPEED and nothing changes. I have tried with CS on D1 and still same problem.

Any ideas?

Thanks much!

56

See the above. There is some problem with Electron using SPI_CONFIGURATION 1. I will able to investigate after I get access to an Electron.

57

On SPI0 with SCK => A3, MISO => A4, MOSI => A5, SS => A2 I get the following error

No card, wrong chip select pin, or SPI problem?
SD errorCode: 0X1,0X0 (instead of 0XFF)

I am going to try it with de SDcard on a breakout board.

58

Means the card does not reply to the first initialization command and MISO is high (the 0XFF).

The case 0X1,0X0 means no reply to the first command and MISO is low.

59

@whg Thanks for your support!

I finally got it to work using an Adafruit microSD Breakout board with SPI0 but with pin A1 and not A2 for SS.

The problem now is that the Accelerometer (I have an Asset Tracker) uses some of the same pins:

Adafruit_LIS3DH accel = Adafruit_LIS3DH(A2, A5, A4, A3)

and will probably be in conflict with the microSD card.
I tried with SPI1 (on Electron) and sure enough the SDcard does not work. Are you actually working on this fix?

Thanks much!

60

It appears that DMA transfers don't work for SPI1 on electron.

SPI seems to work but I need to do more tests.

You can try this fix, it disables DMA SPI transfers on Electron. It will be slow.

Replace exiting functions in SdSpiParticle.cpp whith these:

uint8_t SdSpi::receive(uint8_t* buf, size_t n) {
#if PLATFORM_ID == 10 // Electron
  for (size_t i = 0; i < n; i++) {
    buf[i] = spiPtr[m_spiIf]->transfer(0xFF);
  }
  return 0;
#else  // PLATFORM_ID == 10
  SPI_DMA_TransferCompleted = false;
  spiPtr[m_spiIf]->transfer(0, buf, n, SD_SPI_DMA_TransferComplete_Callback);

  while (!SPI_DMA_TransferCompleted) {SysCall::yield();}
  if (bugDelay) {
    delayMicroseconds(bugDelay);
  }
  return 0;
#endif  //  PLATFORM_ID == 10  
}

void SdSpi::send(const uint8_t* buf , size_t n) {
#if PLATFORM_ID == 10 // Electron
  for (size_t i = 0; i < n; i++) {
      spiPtr[m_spiIf]->transfer(buf[i]);
  }
  return;
#else  //  PLATFORM_ID == 10
  SPI_DMA_TransferCompleted = false;
  spiPtr[m_spiIf]->transfer(const_cast<uint8_t*>(buf), 0, n,
                            SD_SPI_DMA_TransferComplete_Callback);

  while (!SPI_DMA_TransferCompleted) {SysCall::yield();}
  if (bugDelay) {
    delayMicroseconds(bugDelay);
  }
#endif  //  PLATFORM_ID == 10
}
61

That should not be a problem since SPI is meant to share the datalines - only the Slave Select needs to be exclusive.
BTW, does the LIS3DH library use hardware SPI? Then why do they need the pins stated explicitly?