Adapting PublishQueueAsyncRK for Flash Memory

General
1

I’ve been using the fantastic PublishQueueAsyncRK library for dealing with downtime in the cellular connection on the Electron in order to ensure (mostly) that all events eventually are sent. However, 2048 bytes doesn’t hold nearly enough events for several hour ‘out’ periods. I’ve gone ahead and added a 32M Winbond Flash IC to my board to use temporary storage.

My plan was to adapt @rickkas7 library to use the Flash memory, rather than retained memory - are there any reasons this won’t work or is ill-advised before I begin on this journey? Any tips appreciated. Thanks!

2 Likes
2

Hi @hagandh

That is a fantastic idea! But instead of using an additional Flash IC, we’ve decided to use FRAM. Here is why:

In fact we’ve been having the same idea and plan to use this for our commercial celluar PLC controller which is based on the E Series:

We have Edge Counters on the PLC controller and being interrupt controlled they can counting impulses that go up to 4kHz.

We are using these to count things like consumption in Milliliters (Beer) or for Energy, Gas and Water.

Users can define a Publish Interval which usually is set between 10 secs (for realtime) or 1 minute for just logging.

So we had the idea of using @rickkas7 library as well but when publishing events every minute or each ten seconds the Flash IC will be written and erased very very often.

And since we have units in the field that are running for more than a year now we would have more than half a million write/erase cycles.

So we thought of going for FRAM since wear is not a problem in this technology. Here’s how easy it is attached to the E Series:

39
391046×508 42.1 KB

However,

we have not yet implemented the library into our device operating system in such a way that I can share the code now. I will have a look and add this later.

But back to your idea. Using PublishQueueAsyncRK is a very good idea!

It should be straightforward to adapt this using the FRAM (just like using external FRAM). There is also a Library available:

So Pro:

  • No wear / unlimited write/erase cycles

Contra:

  • less space compared to FLASH
  • a bit more expensive

Since our product is being used in production and industrial machines and we needed a way to keep lost publishes (due to cellular connection drops) the FRAM is the best way here.

We are using protobuf and highly minimize the publish payload but however using the FRAM and Async Publish library you will not be able to preserve publishes for weeks or month but it holds the most important ones and we’ve seen that outtakes in the field are mostly no longer than one or two days. And for us this works!

Best,

Simon

EDIT

We’ve been using FRAM also on the so called “Masterbrick” - a Dev-board using E Series and P1 (P1 is currently at the PCB manufacturer).

Read more here:

If you are interested in working with one of these (it has both FRAM and FLASH attached) - contact me!

4 Likes
3

Hey! Did you ever get this working? We are planning on doing the same thing using PublishQueueAsyncRK with FRAM. I appreciate any guidance!

Thank you!

4

Hi,

I’m also interested in this FRAM publishQueueAsyncRK.
i’m not a particle (or C) expert so i’m not able to modify this library myself.

in my project i would need to use this lib in a FRAM and also store other thing in this memory, i don’t really know how to make sure i don’t overwrite the Lib data.

Thanks in advance.

Nicolas

5

There is a new version of the PublishQueueAsyncRK library (0.1.0). This version supports pluggable storage modules, including:

  • Retained memory
  • Regular RAM
  • FRAM (MB85RC256V ferro-electric non-volatile RAM) connected by I2C
  • SPI NOR Flash (ISSI, Winbond, or Macronix) using SpiffsParticleRK
  • SPI Flash memory on P1 module
  • SPI Flash soldered to E Series module (Macronix MX25L8006EM1I-12G)
  • SD cards using the SdFat library

This is a pretty major refactoring of the library. Please use with caution as there could be bugs still.

5 Likes
6

Awesome! Thank you!

7

Wow, amazing update rickkas7.
Many thanks for that.

8

@rickkas7 I’ve been using this library since v0.0.5 and have relied on this variable haveEvent to control my connection behavior in battery mode. If there was data in the queue, I would have my device try to connect to the cellular network to dump the data, but if there is no data the device doesn’t need to connect thus saving battery power. In v1.0.0 the haveEvent variable is not longer used and I am wondering if I add the below will be ok?

This was the check you used before in v0.0.5, but it seems the code has many updates.

Or is there a better way of determining if there is data in the queue?

FYI this is just the beginning of your checkQueueState function.

void PublishQueueAsyncBase::checkQueueState() {
	haveEvent = false;
	SINGLE_THREADED_BLOCK() {
        PublishQueueHeader *hdr = reinterpret_cast<PublishQueueHeader *>(retainedBuffer);
		haveEvent = (hdr->numEvents > 0);
	}
	
	if (!pausePublishing &&
9

0.1.3 (2019-11-21)

  • Added getNumEvents() method so you can know if the queue is empty or not
6 Likes
10

@rickkas7 Thanks for the update. I have one more question is the below bug present in v0.0.5?

0.1.1 (2019-11-15)

  • Fixed a bug that causes thread deadlock when the publish queue is full.
11

The thread deadlock bugs fixed in 0.1.1 (retained memory) and 0.1.2 (FRAM) were introduced in 0.1.0, and were not present in 0.0.5.

5 Likes