SD Library - Mesh Devices


#1

I wanted to check with the community about any SD library that works with Mesh devices and the SD/clock feather.

The popular SDFat library does not compile with Mesh devices but works on Photon/Electron.

Thank you.


#2

@Jimmie, my understanding is that the SDFat author is working on Mesh compatibility for his next release which will also support ExFAT.


#3

I have used SDFat with mesh devices by downloading the sources (in Particle Workbench) and just alter all occurences of O_RDONLY, O_WRONLY & O_RDWR to not collide with the stock firmware.


#4

Just to add what ScruffR said, I’m using SdFat on an Argon by adding SDFAT_ in front of all the occurrences of those three terms. If you’re building online from local files through the CLI, you’ll need to have the SdFat library in your local lib folder, and you’ll need to NOT have it as a dependency in your project.properties file. Otherwise the online compiler will still use the online version of SdFat.


#5

Thank you @peekay123, @ScruffR and @psherk for helping me.

I will download Particle Workbench and read up on the CLI as I have only used it to flash firmware locally.


#6

I have posted a version of SdFat as a beta.

This version of SdFat has been modified to use the standard POSIX/Linux file fcntl.h to define the open flags that caused problems.

Please try it and post problems as issues on github.


#7

Thanks again @whg for your efforts.

How can one use this library online? Do I have to import each file? In Arduino, the library was placed under the libraries folder but I am not sure how to do it in Particle development.


#8

I’d guess some testing is needed before publishing for the Particle Library Manager.
If you can use CLI or Particle Workbench you can simply copy the library repo into the lib folder.


#9

OK, thank you. I just received the link for downloading Particle Workbench.


#10

I have been doing testing with an Argon and am having SPI timeouts when I do large writes.

Typically this happens after 50kB to 100kB.

When I run the same program on a Photon, I get very good results, write 2,550 kB/sec, read 3,050 kB/sec.

Here is a benchmark on a Photon writing/reading a 50 MB file. This test transfers 200 MB over the SPI bus. I have run it a number of times with no errors.

File size 50 MB
Buffer size 16384 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2556.91,23653,5435,6405
2569.79,14272,5544,6373

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
3052.30,6514,5247,5366
3052.49,6474,5246,5366

I have tried various SPI speeds and buffer sizes but always get errors with Argon.

Is the source for the Argon SPI driver available?


#11

I increased the write timeout from 500 ms to 3 seconds and had success with Argon.

I found that there can be a very long latency for some SPI transfers on Argon. Note the first write test has a max write latency of 926391 μs. This is way longer than normal.

Here are results.

File size 5 MB
Buffer size 16384 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1321.99,926391,8697,12371
1751.53,16236,8667,9330

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
2214.05,8057,7324,7393
2213.07,8057,7324,7397

I am beginning to suspect the delay has something to do with the Argon connecting to the net after flashing the program.


#12

Not sure if this applies to the tests you are doing, but IIRC on Gen3 devices SPI doesn’t support DMA and the slower clock may cost some performance too.

Are you running your tests in SYSTEM_MODE(MANUAL)?


#13

I am not running the tests in SYSTEM_MODE(MANUAL). I wanted to check for any problems with a simple app.

The problem was that soon after flashing a new version of the app there can be an SPI transfer that takes about a second. I had a timeout of 600 ms for a loop that checked the status of the SD. I changed the timeout to 2000 ms. Now there is one slow transfer but no error return.

Gen3 devices must have DMA SPI, I get a read rate of 2214 KB/sec on an Argon with a custom Particle driver that I will post on github soon. That’s about 0.5 μs/byte.

In fact, single byte transfers are extremely slow on Argon. Here is a test with a loop of single byte transfers.

  void send(const uint8_t* buf, size_t n) {
    for (size_t i = 0; i < n; i++) {
      m_spi->transfer(buf[i]);
    }
  }

File size 5 MB
Buffer size 16384 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
80.43,1026916,199860,203632
81.51,243714,199890,200925

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
81.50,210389,200592,201023
81.48,207824,200592,201071

An Arduino Uno gets a read speed of 520 KB/sec with the same loop, more than six times faster.

Also there is the long 1026916 μs transfer in the first write test.

Edit: I tried the test with SYSTEM_MODE(MANUAL) and the one second transfer no longer happens.


#14

I see, I revisited the respective statement by avtolstoy and must admit I have misinterpreted it. He said “no DMA on GPIOs” but, yes SPI does have DMA.

BTW, that is an interesting finding, would you be willing to file a GitHub issue about that?


#15

I found the real problem.

I have this function for devices that have a net connection like Particle and ESP8266.

It is used to check for timeout and avoid watchdog timer errors.

bool isTimedOut(uint32_t startMS, uint32_t timeoutMS) {
#if WDT_YIELD_TIME_MICROS
  static uint32_t last = 0;
  if ((micros() - last) > WDT_YIELD_TIME_MICROS) {
    // This function for Particle. 
    Particle.process();
    last = micros();
  }
#endif  // WDT_YIELD_TIME_MICROS
  return (millis() - startMS) > timeoutMS;
}

One of the first calls to Particle.process() on Argon takes about 800 ms.

Here is a test program:

#include "SPI.h"

#define WDT_YIELD_TIME_MICROS 100000
#define TIMEOUT_MS 600
//------------------------------------------------------------------------------
bool isTimedOut(uint32_t startMS, uint32_t timeoutMS) {
#if WDT_YIELD_TIME_MICROS
  static uint32_t last = 0;
  if ((micros() - last) > WDT_YIELD_TIME_MICROS) {
    // this function for Particle.
    Particle.process();
    last = micros();
  }
#endif  // WDT_YIELD_TIME_MICROS
  return (millis() - startMS) > timeoutMS;
}
//------------------------------------------------------------------------------
void setup() {
  Serial.begin(9600);
  while (!Serial) {
    delay(1);
  }
  Serial.println("Type any character to start");
  while (!Serial.available()) {
    delay(1);
  }
  SPI.begin();
  SPI.beginTransaction(SPISettings(50000000, MSBFIRST, SPI_MODE0));   
  uint32_t maxmicros = 0;
  uint32_t minmicros = 9999999;
  uint32_t startmicros = micros();
  for (int n = 0; n < 100000; n++) {
    bool timeout = false;    
    uint32_t m = micros();
    uint32_t ms = millis();
    if (isTimedOut(ms, TIMEOUT_MS)) {
      timeout = true;
    }
    SPI.transfer(0XFF);
    // Normally test the return from SPI.transfer().
    m = micros() - m;
    if (m < minmicros) minmicros = m;
    if (m > maxmicros) maxmicros = m;
    if (timeout) {
       Serial.println("timeout");
    }       
  }
  uint32_t totalmicros = micros() - startmicros;
  Serial.print("totalmicros: ");
  Serial.println(totalmicros);
  Serial.print("minmicros: ");
  Serial.println(minmicros);
  Serial.print("maxmicros: ");
  Serial.println(maxmicros);
  SPI.endTransaction();
}

void loop() {
}

Typical output:

Type any character to start
timeout
totalmicros: 5387054
minmicros: 30
maxmicros: 827026

“timeout” is only printed once.


#16

Can I assume you are testing this with WDT_YIELD_TIME_MICROS defined and with 0.8.0-rc.27?
I just want to be sure when I file an issue about that, or do you rather want to file it?


Update:
I filed the issue.
Could you please fill in the gaps there?
Thanks


#17

I am using 0.8.0-rc.27 and in SdFat:

// If Particle device or ESP8266 call yield.
#define WDT_YIELD_TIME_MICROS 100000

Particle.process() only gets called if I am in a loop waiting for some response from the SD and it has been more than 100 ms since the last call to Particle.process().