SpiffsParticle Library

The excellent SPIFFS library provides a simple file system on a NOR flash chip. This is a port of the library for the Particle platform, with a few convenience helpers to make using it easier from C++ and using standard Particle/Arduino/Wiring APIs.

Both the original SPIFFS library and this port are MIT licensed, so you can use them in open-source or in closed-source commercial products without a fee or license.

The full browsable API documentation can be found here.

The library source and documentation can be found here.

Flash Support

This library supports SPI-connected NOR flash chips. These are typically tiny 8-SOIC surface mount chips, intended to be included on your own circuit board. There are also breadboard adapters that are available, shown in the examples below.

The chips are really inexpensive, less than US$0.50 in single quantities for a 1 Mbyte flash. They’re available in sizes up to 16 Mbyte.

SPI flash is less expensive than SD cards, and do not need an adapter or card slot. Of course they’re not removable.

The underlying SpiFlashRK library library supports SPI NOR flash from

It is sometimes possible to find the 8-PDIP (0.3") versions suitable for plugging directly into a breadboard. Both Macronix and Winbond make them, but they’re infrequently used and often not available.

It does not support I2C flash, SD cards, or non-flash chips like FRAM.

Connecting the hardware

The real intention is to reflow the 8-SOIC module directly to your custom circuit board. However, for prototyping, here are some examples:

For the primary SPI (SPI):

Name Flash Alt Name Particle Pin Example Color
SS CS A2 White
SCK CLK A3 Orange
MISO DO A4 Blue
MOSI DI A5 Green

For the secondary SPI (SPI1):

Name Flash Alt Name Particle Pin Example Color
SS CS D5 White
SCK CLK D4 Orange
MISO DO D3 Blue
MOSI DI D2 Green

Note that the SS/CS line can be any available GPIO pin, not just the one specified in the table above.

  • Electron using Primary SPI

  • Photon using Secondary SPI (SPI1)

  • Photon using Primary SPI and a poorly hand-soldered 8-SOIC adapter

  • P1 module (extra flash is under the can)

Using the SPIFFS API

Background

A few things about SPIFFS:

  • It’s relatively small, way smaller than SDFAT at least.
  • It has the bare minimum of things you need to store files.
  • You can allocate all or part of the flash chip to SPIFFS.

In particular, there are two important limitations:

  • It does not support subdirectories; all files are in a single directory.
  • Filenames are limited to 31 characters.
  • It does not have file timestamps (modification or creation times).

Within the scope of how you use SPIFFS these shouldn’t be unreasonable limitations, though you should be aware.

Like most file systems, there are a few things you must do:

  • You must mount the file system, typically at startup.
  • If your flash is blank, you’ll need to format the file system.
  • You can iterate the top level directory to find the file names in it.
  • In order to use data in a file, you must open it and get a file handle, read or write, then close it.
  • There are a finite number of files that can be open, but you can set the maximum at mount time. The default is 4.
  • If you think the file system is corrupted, you can check and repair it.

Once you get it working, there is some fine-tuning you can do:

  • The cache size is programmable, which speeds up operations at the cost of additional RAM. The default is 2 logical blocks.
  • The logical block size is programmable. Using larger blocks can make using large files more efficient in flash storage, at the cost of more RAM and making small files less efficient in flash storage. The default is 4K, and can’t be made smaller but can be made larger.
  • Even with the default settings, it’s surprisingly fast (see the benchmarking section, below).

Example 1 - Simple

This example should be pretty straightforward:

#include "Particle.h"

#include "SpiffsParticleRK.h"

// Pick a debug level from one of these two:
// SerialLogHandler logHandler;
SerialLogHandler logHandler(LOG_LEVEL_TRACE);

// Chose a flash configuration:
SpiFlashISSI spiFlash(SPI, A2); 		// ISSI flash on SPI (A pins)
// SpiFlashISSI spiFlash(SPI1, D5);		// ISSI flash on SPI1 (D pins)
// SpiFlashMacronix spiFlash(SPI1, D5);	// Macronix flash on SPI1 (D pins), typical config for E series
// SpiFlashWinbond spiFlash(SPI, A2);	// Winbond flash on SPI (A pins)
// SpiFlashP1 spiFlash;					// P1 external flash inside the P1 module

// Create an object for the SPIFFS file system
SpiffsParticle fs(spiFlash);

void setup() {
	Serial.begin();

	spiFlash.begin();

	fs.withPhysicalSize(1024 * 1024);

	s32_t res = fs.mountAndFormatIfNecessary();
	Log.info("mount res=%d", res);

	if (res == SPIFFS_OK) {
		SpiffsParticleFile f = fs.openFile("test", SPIFFS_O_RDWR|SPIFFS_O_CREAT);
		if (f.isValid()) {
			f.println("hello world");

			f.seekStart();

			String s = f.readStringUntil('\n');
			Log.info("got: %s", s.c_str());

			f.close();
		}
	}
}

void loop() {

}

In more detail:


// Pick a debug level from one of these two:
// SerialLogHandler logHandler;
SerialLogHandler logHandler(LOG_LEVEL_TRACE);

This determines the log level. If you want fewer logs, uncomment the first SerialLogHandler definition and comment out the second.


// Chose a flash configuration:
SpiFlashISSI spiFlash(SPI, A2); 		// ISSI flash on SPI (A pins)
// SpiFlashISSI spiFlash(SPI1, D5);		// ISSI flash on SPI1 (D pins)
// SpiFlashMacronix spiFlash(SPI1, D5);	// Macronix flash on SPI1 (D pins), typical config for E series
// SpiFlashWinbond spiFlash(SPI, A2);	// Winbond flash on SPI (A pins)
// SpiFlashP1 spiFlash;					// P1 external flash inside the P1 module

This sets up an ISSI flash chip on primary SPI (SPI), with A2 as the CS (chip select or SS line). You can comment this out and uncomment one of the other lines for other configurations.


SpiffsParticle fs(spiFlash);

This sets up the SpiffsParticle object using that flash chip. You will typically create this object as a global variable.


	spiFlash.begin();

	fs.withPhysicalSize(1024 * 1024);
	
	s32_t res = fs.mountAndFormatIfNecessary();
	Log.info("mount res=%d", res);

You must call begin() on the flash object. Then you must set the size of the file system. This is 1 Mbyte. The SPIFFS file system can use only a part of the flash, if you want. It can’t be resized without reformatting, however.

Finally, you must mount the file system. This call mounts it, if not formatted, will format it and try to mount it again.


		SpiffsParticleFile f = fs.openFile("test", SPIFFS_O_RDWR|SPIFFS_O_CREAT);
		if (f.isValid()) {
			f.println("hello world");

			f.seekStart();

			String s = f.readStringUntil('\n');
			Log.info("got: %s", s.c_str());

			f.close();
		}

This block of code opens the file, creating it if necessary.

It writes the line hello world to the file.

Then it reads the line back and prints it to the debug serial.

Once you get your filesystem working, you may want to eliminate the info messages from Spiffs. This can be done by using the categories feature of the log handler:

SerialLogHandler logHandler(LOG_LEVEL_WARN, { // Logging level for non-application messages
    { "app", LOG_LEVEL_INFO }, // Default logging level for all application messages
    { "app.spiffs", LOG_LEVEL_WARN } // Disable spiffs info and trace messages
});

Benchmarking

The 5-benchmark example runs some tests to evaluate the speed of the file system. This log is testing a 1 Mbyte ISSI flash chip on a Photon.

The format operation is slow, about 14 seconds for a 1 Mbyte flash. Fortunately you shouldn’t have to format often (probably only once).

0000010000 [app] INFO: starting chipErase
0000011853 [app] INFO: finished chipErase: 1853 ms
0000011853 [app] INFO: starting format
0000026245 [app] INFO: format res=0
0000026245 [app] INFO: finished format: 14392 ms
0000026245 [app] INFO: starting mount
0000026291 [app] INFO: mount res=0
0000026292 [app] INFO: finished mount: 47 ms

The bulk write and read test does 512 byte operations sequentially. It’s able to write 256 Kbytes in 1.5 seconds. It can read it in 264 milliseconds.

0000026292 [app] INFO: testing 262144 bytes in 512 byte blocks 
0000026360 [app] INFO: starting write
0000027924 [app] INFO: finished write: 1564 ms
0000027924 [app] INFO: starting read
0000028188 [app] INFO: finished read: 264 ms

The append and flush test writes 100 bytes and flushes the contents to flash so they’ll be preserved if there is a power outage. It’s able to do 5000 append and flush operations in 831 milliseconds.

Reading the blocks back only takes 104 milliseconds.

0000028708 [app] INFO: testing append and flush 100 bytes 5000 times 
0000028776 [app] INFO: starting append
0000029607 [app] INFO: finished append: 831 ms
0000029607 [app] INFO: starting read
0000029711 [app] INFO: finished read: 104 ms

Macronix 1 Mbyte on SPI1 (D pins)

0000010000 [app] INFO: starting chipErase
0000012944 [app] INFO: finished chipErase: 2944 ms
0000012944 [app] INFO: starting format
0000021291 [app] INFO: format res=0
0000021291 [app] INFO: finished format: 8347 ms
0000021291 [app] INFO: starting mount
0000021356 [app] INFO: mount res=0
0000021356 [app] INFO: finished mount: 65 ms
0000021356 [app] INFO: testing 262144 bytes in 512 byte blocks 
0000021461 [app] INFO: starting write
0000023808 [app] INFO: finished write: 2347 ms
0000023808 [app] INFO: starting read
0000024164 [app] INFO: finished read: 356 ms
0000024989 [app] INFO: testing append and flush 100 bytes 5000 times 
0000025093 [app] INFO: starting append
0000026331 [app] INFO: finished append: 1238 ms
0000026331 [app] INFO: starting read
0000026466 [app] INFO: finished read: 135 ms

Winbond 16 Mbyte on SPI (A pins), only 1 Mbyte file system (format would take longer with a full 16 Mbyte file system).

0000010000 [app] INFO: starting chipErase
0000038317 [app] INFO: finished chipErase: 28317 ms
0000038318 [app] INFO: starting format
0000044716 [app] INFO: format res=0
0000044716 [app] INFO: finished format: 6398 ms
0000044716 [app] INFO: starting mount
0000044763 [app] INFO: mount res=0
0000044763 [app] INFO: finished mount: 47 ms
0000044763 [app] INFO: testing 262144 bytes in 512 byte blocks 
0000044831 [app] INFO: starting write
0000046392 [app] INFO: finished write: 1561 ms
0000046392 [app] INFO: starting read
0000046656 [app] INFO: finished read: 264 ms
0000047201 [app] INFO: testing append and flush 100 bytes 5000 times 
0000047269 [app] INFO: starting append
0000048118 [app] INFO: finished append: 849 ms
0000048119 [app] INFO: starting read
0000048223 [app] INFO: finished read: 104 ms

Resource usage

The code space used by the library is slightly less than 30 Kbytes. This includes both the SpiFlashRK and SpiffsParticleRK libraries and the Wiring/Arduino Stream compatibility.

The RAM usage is dependent on various settings.

  • Work buffers (2 * logical page size), default is 2 * 256 = 512
  • File descriptor buffers (32 * max open files), default is 32 * 4 = 128
  • Cache (logical page size + 32) * cachePages + 40 byte, default is (256 + 32) * 2 + 40 = 616
  • Thus the total RAM allocated during mount is by default is 1256 bytes

This memory is freed if you unmount, though in most cases you’ll keep the volume mounted all of the time.

9 Likes

This is amazing thank you Rick!

Looking very good Rick, congrats.

I have been following your progress and I’m considering switching to a simple file system instead of managing raw EEPROM for very dynamic persistence of objects.

The P1 has 1 MB of flash, which would work great with this library, but a lot of my users are still on the Photon, without external flash.

Your library does not seem to target the Photon’s internal flash memory, only external memory.
Particle has reserved some space in internal NOR flash to be used for emulated EEPROM. Do you know if I could use SPIFFS to manage that same space?

The Particle EEPROM Flash emulation is defined as:

#include "eeprom_emulation.h"
#include "flash_storage_impl.h"

constexpr uintptr_t EEPROM_SectorBase1 = 0x0800C000;
constexpr uintptr_t EEPROM_SectorBase2 = 0x08010000;

constexpr size_t EEPROM_SectorSize1 = 16*1024;
constexpr size_t EEPROM_SectorSize2 = 64*1024;

using FlashEEPROM = EEPROMEmulation<InternalFlashStore, EEPROM_SectorBase1, EEPROM_SectorSize1, EEPROM_SectorBase2, EEPROM_SectorSize2>;

And in eeprom_hal.cpp a global FlashEEPROM object is instantiated, but I believe only when init is called, flash memory is accessed. That memory range seems free to use for SPIFFS if EEPROM is not used.

Do you also have any idea of the overhead for a small file?
I’d like to store objects ranging from 10 bytes to 150 bytes.

I’ll definitely look into your library for storing data while offline (on the P1), but to also use it for configuration storage would be great.

With flash memory you can only erase to sector boundaries. For SPI flash devices, this is almost always 4K bytes. SPIFFS assumes the smallest unit of storage is a sector, so storing a 1 byte file will use 4096 bytes (plus some more overhead for the file entry). It also assumes a large number of available sectors, all the same size. It’s not at all suitable for use in the emulated EEPROM space (2 sectors). The STM32 flash sectors are few in number and in some cases quite large, up to 256K bytes and thus poorly matched to how SPIFFS works, unfortunately.

2 Likes

Thanks for the answer. I was afraid that would be the case. Now I know for sure. I’ll stick to EEPROM for storing settings then.

I tried using the library with an MX25L51245GMI-08G flash chip with 64MBytes (Boron, 1.5.2). Mounting after formatting fails with [app.spiffs] INFO: mount after format res=-10025 when using a size >16MBytes.
I described some details in https://github.com/rickkas7/SpiffsParticleRK/issues/5

@rickkas7 I found a bigger chip, but is there a limit in the library with 16MByte?

Has anyone used this library with a flash >16MBytes?

The 16 MB limit is a hard limit in the version of SPIFFS that I forked. The reason is the logical page size is 256 bytes and the size of the logical page index is a uint16_t so 256 * 65536 = 16777216 = 16 MB.

In theory, the logical page size could be changed, however it would reduce efficiency on smaller devices, and since it’s a #define I can’t easily change it at runtime based on the device size.

You could try making a fork with it changed and see if it works.

1 Like

Thank you, after changing the logical page size and the index data type did not help, I continued searching and identified another issue.

I made a short summary for everyone coming here with a flash larger than 16MByte:

In the underlying SpiFlashRK library the flash addressing happens in 3-byte mode (seems to be the standard mode), which can address a maximum of 2^24 bytes, which equals the 16*1024*1024 size limit I hit. I forked your SpiFlashRK library and added an option for the 4-byte addressing mode of my MX25L51245GMI-08G flash chip, according to its datasheet.

Of course the page size or the index variable datatype has to be changed too. I have not fully tested this yet, but both seem to work. I prefer changing the data types in spiffs_config.h, as there are 4 variables which need to be large enough for the filesystem size. I think at least three of them need to be uint32_t.

Isn’t the singleton mode, where it would be a define, disabled here?

This method seems to be the way to change the logical page size:

I am looking to replace an SD card and SDFat use with NOR flash but I really need at least 32MB - looking at what @nils is doing I was wondering what happened to LittleFS and the POSIX file system that was talked about. Concern with SPIFFS is the resilience to unplanned power down or resets. Any advice you could offer?

I asked the same question about the Gen3 LittleFS access to the 2MB of the 4MB on the Gen3 devices and when it will be available.

@rickkas7 Said it was enabled in firmware 2.0rc1 and you can use the access info in the docs for the asset tracker until it’s switched on for the Gen3 devices also in the docs.

@RWB thanks for the update. I am looking for something that could be implemented with the Photon and not Gen3. I want to do a spin of a mother board with the Photon replacing the SD connector and SD card with some NOR Flash - I really need 32MB, ideally 64MB and a drop-in replacement for SDFat (accept that the simpler file system commands are sufficient for my needs). The SD cards or rather the connectors have failed when operated in environments subject to high humidity - corrosion I guess.

1 Like

Anyone had any experience with Adafruit_nRF52_Arduino/libraries/ Adafruit_LittleFS / ?

Hi @nils, does your fork work for 32MB as well?

Hi @Gorgo,
with the changes I mentioned it worked for a 64MB flash, so 32MB will work too.

But make sure to test this extensive before using in production. I had a simple test program writing test data to about 90% of the flash (leaving some empty space for the filesystem internal work) and reading this out again, but there have been some issues. Depending on the test data structure, my program got stuck at different points, but I did not have time for further testing and my application did not require more than 16MB yet.