I have read most of the threads regarding the publishing libraries PublishQueueAsyncRK or PublishQueuePosixRK but still could use some advice.
In our application, the host hardware generates telemetry messages that are received serially (SerialBufferRK) by a Boron device and published to the Particle cloud and other services. This works well, though we wish to add the capability to prevent telemetry loss for up to 7 days for reasons such as lost connectivity, power failure etc.
To this end I have been investigating the option of using non-volatile flash memory with community libraries PublishQueueAsyncRK or PublishQueuePosixRK as a means of implementing this. The documentation recommends using PublishQueuePosixRK for the Boron as the implementation is simpler. I have a couple of questions relating to the FLASH memory, library limits and other issues.
Can we use the 2MB user segment of the internal 4MB QSPI Flash with these libraries?
Can we have maximum file queue = 7000 events?
RAM queue = 0 for maximum data retention
Issues related to wear leveling, is this handled by the library?
We will eventually use an external W2Q32JV 32MBIT SPI/QUAD SOIC, but for development purposes is it possible to use the the internal FLASH? The very large file queue is necessitated by the 7 day offline operation with the understanding of a possible long resynchronization time in the worst case (approx. 2 hours).
Can we use the 2MB user segment of the internal 4MB QSPI Flash with these libraries?\
No. That's reserved by the system. Among other things, it holds the images that are being flashed OTA.
Can we have maximum file queue = 7000 events?
Definitely not. You'll exceed the number of available blocks.
RAM queue = 0 for maximum data retention
That is the safest.
Issues related to wear leveling, is this handled by the library?
Handled by LittleFS.
The problem is that PublishQueuePosix only works with the Device OS Little FS POSIX file system, which only works with the built-in file system. It can't be configured to use other parts of the built-in flash or an external flash chip. PublishQueuePosix stores the events one-per-file because it makes removing the old events efficient. Since each file takes two sectors (of 4096) bytes this will limit the number of events that can be stored.
However, you also don't want to use PublishQueueAsync with long queues. PublishQueueAsync stores the events in a single file, appending new events to the end. The file is truncated only after all events have been sent, which means it has the potential to grow forever and fill the flash even though there aren't that many events outstanding.
Also keep in mind you may not want to send 7000 events as 7000 separate publishes. Since the maximum rate is 1 per second, it will take nearly 2 hours to upload the events separately.
Instead you'll probably want to store your data using SdFAT (SD card) or Little FS (as a library, with external SPI flash) in a more efficient method using your own code, then figure out how to generate a more efficient upload mechanism.
I’m trying to use examples in library LittleFS-RK (0.0.1) - unittest-SpiFlashRK and keep getting the following compile errors. This happens even with a new project with stubs for setup and loop. Using latest versions of VS Code, Particle, and toolchains. I am not having problems with any other libraries. Using firmware 4.01. Cloud compile produces same results. Any suggestions?
:::: COMPILING APPLICATION
Creating c:/VSC_Projects/Test_NV_File/target/4.0.1/boron/platform_user_ram.ld ...
In file included from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/syscalls_posix.cpp:35:
c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/scope_guard.h:142:7: error: redefinition of 'class particle::NamedScopeGuard<FuncT>'
142 | class NamedScopeGuard {
| ^~~~~~~~~~~~~~~
In file included from ../wiring/inc/spark_wiring_ethernet.h:28,
from ../wiring/inc/spark_wiring.h:46,
from ./inc/application.h:42,
from ./inc/Particle.h:5,
from c:/VSC_Projects/Test_NV_File/lib/SpiFlashRK/src/SpiFlashRK.h:12,
from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/LittleFS-RK.h:4,
from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/filesystem.h:20,
from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/syscalls_posix.cpp:32:
../services/inc/scope_guard.h:142:7: note: previous definition of 'class particle::NamedScopeGuard<FuncT>'
142 | class NamedScopeGuard {
| ^~~~~~~~~~~~~~~
In file included from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/syscalls_posix.cpp:35:
c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/scope_guard.h:174:31: error: redefinition of 'template<class FuncT> particle::NamedScopeGuard<FuncT> particle::makeNamedScopeGuard(FuncT&&)'
174 | inline NamedScopeGuard<FuncT> makeNamedScopeGuard(FuncT&& func) {
| ^~~~~~~~~~~~~~~~~~~
In file included from ../wiring/inc/spark_wiring_ethernet.h:28,
from ../wiring/inc/spark_wiring.h:46,
from ./inc/application.h:42,
from ./inc/Particle.h:5,
from c:/VSC_Projects/Test_NV_File/lib/SpiFlashRK/src/SpiFlashRK.h:12,
from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/LittleFS-RK.h:4,
from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/filesystem.h:20,
from c:/VSC_Projects/Test_NV_File/lib/LittleFS-RK/src/syscalls_posix.cpp:32:
../services/inc/scope_guard.h:174:31: note: 'template<class FuncT> particle::NamedScopeGuard<FuncT> particle::makeNamedScopeGuard(FuncT&&)' previously declared here
174 | inline NamedScopeGuard<FuncT> makeNamedScopeGuard(FuncT&& func) {
| ^~~~~~~~~~~~~~~~~~~
make[3]: *** [../build/module.mk:274: ../build/target/user/platform-13-m/Test_NV_File/LittleFS-RK/src/syscalls_posix.o] Error 1
make[2]: *** [../../../build/recurse.mk:12: user] Error 2
make[1]: *** [../build/recurse.mk:12: modules/boron/user-part] Error 2
make: *** [C:\Users\RobertVinci\.particle\toolchains\buildscripts\1.11.0\Makefile:68: compile-user] Error 2
LittleFS-RK only works on Gen 2 devices (E Series, Electron, P1, Photon). It does not work on any other devices, because it conflicts with the LittleFS implementation in Device OS.
In theory it could be made to work on Gen 3 so it could be used with external SPI flash, but it doesn’t currently work. The POSIX shim would need to be removed from the library and the library version called directly so it doesn’t conflict with the built-in implementation.