Try SdFat, a Library for SD Cards


#22

I assume it might be advisable to disable updates while writing to the SD card?


#23

I didn’t know a firmware update could occur while a user program is running. Maybe someone can answer this question.

The only danger of SD corruption or data loss is if a file that is open for write is not properly closed.

You can reduce the chance of data loss by calling sync() or flush() like this:

file.sync(); 

flush() is just another name for sync().

This is equivalent to closing and immediately reopening the file. No data will be lost if your program ends after this call but before you call write() or print().


#24

Yes, OTA flashing can occure at any time while the device is connected to the cloud.
Which is usually no problem if the user and the one flashing new code are the same person (mental diseases negelcted here ;-)).

But since this might become a problem with products sold to customers, Particle is working on solutions like protection against OTA flashing or an event to subscribe to which signals a pending flash/update.

Have a look at this
https://docs.particle.io/reference/firmware/photon/#ota-updates

So @tjp, yes it might be advisable, but this should be the final developers responsibility and not necessary up to the library contributor.
The final dev knows best which way of dealing with the situation is the best for his product.


#25

Are you telling me that a photon I own and have setup can do an OTA firmware update spontaneously without my action while a program is executing?

Would this program suddenly stop printing a series of numbers for a firmware update?

void setup() {
 Serial.begin(9600);
}
int n = 0;
void loop() {
  Serial.println(n++);
  delay(500);
}

#26

Yes, absolutely! I did not mean to imply that I was expecting SdFat to do that. I was simply asking if that is something I should do :smile:

Thanks, @ScruffR!


#27

If you’re the end developer, no. You’d have to trigger the OTA update yourself. But, you wouldn’t want to do that in the middle of a write and risk rebooting before close is called. That’s why I asking about using the APIs that @ScruffR linked to disable OTA updates during writes.


#28

Disabling updates only during writes won’t help. SD cards are written in 512 byte blocks and there are cache blocks in any SD library.

In addition to the cache blocks, directory info is also cached and not updated on the SD until a file is closed or sync/flush is called. I like the name sync better than flush since these calls do more than flush buffers. They read directory info, update it, and write it to the SD. You must read an entire directory block, update it, and write the block back to the SD.

SdFat on Particle uses two cache blocks, one for file data and one for system structures. These are not synced after every write since the overhead would be huge. Updating file directories after each write would add even more overhead.

You can call sync/flush for every open file after any write and enable updates until the next write. This is OK if it does not cause too large of a performance hit.


#29

Understood. My SD card shares the SPI bus with my OLED so I keep my writes atomic (I open, write then close every time). My writes only happen in response to a cloud function call, so the file does not need to stay open.


#30

You can leave files open when sharing the SPI bus. There can be a very large overhead in opening files and seeking to EOF. The FAT filesystem requires a sequential search of directories so an open with 1,000 file in a directory can take a long time. Seeking to the end of a very large file also takes time since the FAT must be read to get the cluster chain.

I have had users that create many small files in a directory so the open close sequence finally took a long time.


#31

I’m glad we’re having this conversation then. Perhaps, I will leave the file open then. However, the way I’m currently using it is with O_TRUNC so EOF is pretty easy to find. :stuck_out_tongue: If I leave the file open, I could just call truncate() manually, correct? I’m writing a list of alarms (described by cron expressions), so I just write the whole list every time since the list could change drastically (especially in development). Is this what you’d recommend, @whg?


#32

The fastest way would be to rewind the file, write the new version, truncate to the current position, and finally call sync().

A good and simpler way is to truncate the file to zero length, write the new version, and call sync().

The first way saves freeing and allocating a cluster, usually 32 KiB on a properly formatted SD.


#33

Like this:

// ...
file.rewind();
file.print(alarms); // alarms is Printable
file.truncate(file.curPosition());
file.sync();
// ...

Well, after writing it out next to what you wrote, I’m no longer questioning it. lol

Thanks, @whg!


#34

@whg I’m toiling away trying to dual wield SdFat and the Particle Webduino library:

I have a web server listing files, and I’m able to click them to make BMP graphics appear on an LED display controlled by the Photon. It’s great, but I also want to:

  • Read HTML file from SD
  • Write files to SD

All examples of this I’ve found on the web use very old versions of SdFat, and more often than not appear to be less than optimized (e.g. reading/writing one byte at a time). I realize an SdFat-powered web server is outside the scope of your example files, due to dependency on another library, but if you find this interesting I would be grateful for a basic sketch that shows an example of each of these techniques using your latest code.


#35

The multiple byte API for read and write is used in the bench example. See lines 152 and 187.

You should look at the Doxygen html documentation for the complete SdFat API.

SdFat has several file classes. The most basic class is FatFile. This class has no Arduino style features. The “File” class tries to be compatible with the Arduino SD.h library. The Arduino SD.h library is a wrapper for a version of SdFat that I wrote in 2009. Sadly the result of history is more file classes than I would like.

I would be happy to help if you have specific questions about your needs in Webduino.


#36

Has anyone had any luck using this library with a 32GB uSD card? I’ve been able to get things working very well with up to 16GB but 32GB cards just throw an error 0x2,0xDF (presumably card not supported) fault.

The cards I’m trying this with are Samsung 32GB class 10, FAT32 formatted.


#37

SdFat supports any size card.

You card fails in initialization before the size is determined. The response code 0xDF to the CMD8 initialization request is invalid.

What is the hardware setup? Some cards are more inclined to SPI failure with marginal SPI signals. Try a slower SPI speed to check for an SPI problem.

It is also possible to enable software CRC to check for SPI errors.

32GB cards are commonly used with SdFat. I have tested with 128GB SDXC cards formatted FAT32.

I have used the core library for 500GB USB hard drives in the UsbFat version of the library.


#38

Interesting–glad to hear others have been able to use 32GB cards at least though I’m still confused as to why I’m unable to see similar results.

I’m using a custom P1 module but the SD is on the primary SPI lines (SPI_CONFIGURATION 0). The only thing that might be slightly different is we use the “DAC” pin for chip select instead of the standard chip select pin, but I haven’t come across any problems at all with lower capacity cards so I’m doubtful it’s related to a chip select problem. In the past I’ve seen the same behavior with a standard Photon board and an off-the-shelf SD card shield (which is why I suspected this may have been more systemic) but I haven’t tried the normal Photon with this newer library.

Currently I’m running SPI_HALF_SPEED, I can try 1/4 speed though to see if there are any improvements or changes.

Any other suggestions on what to try?

Is there anything special about how the card has to be formatted perhaps?


#39

The failure is simple, it has nothing to do with the amount of flash in the card or the card format.

It almost always is an SPI noise problem or flaky chip select.

Early in initialization I use SD CMD8 to make sure the card is a version 2 card and the SPI bus is clean. CMD8 has a field for a one byte check pattern that is traditionally filled with 0XAA. If that is not returned, you know there is a problem.

This loop never returns the check pattern. The last value in your case was 0XDF.

while (1) {
  if (cardCommand(CMD8, 0x1AA) == (R1_ILLEGAL_COMMAND | R1_IDLE_STATE)) {
    type(SD_CARD_TYPE_SD1);
    break;
  }
  for (uint8_t i = 0; i < 4; i++) {
    m_status = spiReceive();
  }
  if (m_status == 0XAA) {
    type(SD_CARD_TYPE_SD2);
    break;
  }
  if (((unsigned)millis() - t0) > SD_INIT_TIMEOUT) {
    error(SD_CARD_ERROR_CMD8);
    goto fail;
  }
}

I have been using simple microSD sockets from ebay that only have 10k pull-ups and caps on 3v3. I have had no failures.


#40

Give another pin a try for this. I’ve filed an issue in connection with the DAC pins (introduced with 0.4.9), where some actions on a different pin have cross effects on the digitalWrite() behaviour of the DAC pins.
This might not be the cause for your problem, but worth considering.


#41

I agree with your assessments though I’m still not quite sure what’s going on here.

I tried running the SPI line at SPI_SIXTEENTH_SPEED (32 divider) which is the slowest I could configure it to operate at and I’m getting the same results (SD errorCode: 0X2,0XDF). Again this only happens with 32GB cards (I haven’t tried larger, though I have pretty extensively tried cards smaller than 32GB, at least 4 different types and they all worked flawlessly). The signals show clean on the oscilliscope though I will check this again to verify.

We do have pull-ups and caps on the signal lines.

The P1 module is running V0.4.9 (though I did try it also with V0.4.7), system threads are enabled (I did try it with them disabled as well though).

To recap the exact card I’m using is a Samsung 32GB EVO UHS-I Class 10 micro-SD card–my understanding is it should be fully compatible with other micro-SD cards. I’m able to successfully access (read and write reliably) the SD card using a different micro-SD board (non-particle though, so not this library and not this architecture) so the card itself doesn’t seem to be the problem.

@ScruffR Unfortunately I can’t easily try another chip select as the board has already been fabricated–this could be switched when the board goes to production if it really is the chip select pin though.

I guess my next step will be to try this code again on a standard Photon board with the off-the-shelf SD shield at a lower SPI bit-rate (have tried this in the past but am willing to give it another go) unless anyone has any other ideas?