Volatile Structures and EEPROM

@All,

I am currently using a data structure to keep count of cars entering a park:

struct currentCounts_structure {                    // currently 10 bytes long
  int hourlyCount;                                  // In period hourly count
  int hourlyCountInFlight;                          // In flight and waiting for Ubidots to confirm
  int dailyCount;                                   // In period daily count
  unsigned long lastCountTime;                      // When did we record our last count
  int temperature;                                  // Current Temperature
  int alertCount;                                   // What is the current alert count
  int maxMinValue;                                  // Highest count in one minute in the current period
} current;

I can then save the structure occasionally to FRAM using @rickkas7’s MB85RC256V-FRAM-RK library as an object such as:

fram.put(FRAM::currentCountsAddr,current);

Thing is that I can miss counts if the Particle device is busy sending data or waiting for a response from my back-end web service. As the cars trigger a hardware interrupt, I would like to move from simply raising a flag in the ISR to updating the counts in the ISR so I will miss less counts. Problem is that when I tried to make either the individual variables in the structure volatile or the whole structure volatile, I get this compile error complaining about the fram.put() and fram.get() commands.

warning: implicit dereference will not access object of type 'volatile currentCounts_structure' in statement
   fram.get(FRAM::currentCountsAddr, current);

I have read about the various ways this can be done but none seem to work when I want to store or retrieve from FRAM.

https://embeddedgurus.com/barr-code/2012/11/how-to-combine-volatile-with-struct/

Any suggestions?

How about having an independent volatile variable and transfer it to the struct just before writing to FRAM?

@ScruffR,

Yes, this is the path I originally started going down. It certainly could work but I’m trying to reduce the number of global variables in my programs. I’m hoping that I could use the structure which already exists and is global for this purpose.

There are a number of articles on the web which just there’s no issue with making a structure volatile. I am curious as to why you can’t store a structure just because it’s volatile.

Thanks, Chip

If I remember correctly, volatile is no longer guaranteed to work the way we used to get by with in multi-threaded applications.

What I would do is as soon as you enter a cloud exchange, make a copy of your structure and use that as your “frozen” struct for reporting. You can even save it to FRAM in a separate location in case something goes wrong with the reporting.*

Then you can still use your original struct to update your counts while waiting on cloud operations to complete.

*This copy operation should still be guarded by a mutex but it reduces the amount of code that must be guarded.

The limitation with volatile is how the FRAM library works. You can’t pass a volatile variable to it, because the FRAM library needs to obtain a pointer to it, which cause the variable to lose its volatile attribute.

However, there’s a more fundamental reason why you can’t do that.

Even though the variable is declared volatile, that only prevents the compiler from making assumptions about the variable, mostly with regards to optimization. It always assumes a side-effect has changed the value and always re-reads it from memory instead of caching it in a register.

What volatile does not do is guarantee atomicity. Say the ISR is incrementing a variable. It’s going from 0x0000ff to 0x000100. Because there is no guarantee of atomicity, the FRAM library could read this as 0x00000000 or some similar invalid value because bytes changed in the middle of a read and you got a half-updated value.

The normal techniques like critical sections and mutexes can’t generally be used from an ISR, but this is one technique that you can use: atomic variables.

This example shows how to atomically increment a variable in the ISR. In loop, it reads and clears the ISR variable, and adds it to another variable. This is where you’d add to your structure you store in FRAM.

#include "Particle.h"

#include <atomic>

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

const pin_t INT_PIN = D2;

void testISR();

// interruptCounter is an atomic variable, so sets and gets occur atomically
std::atomic<uint32_t> interruptCounter;

// This is the counter we manage from loop
uint32_t counter = 0;

unsigned long lastReport = 0;

void setup() {
    pinMode(D2, INPUT);

    // 
    interruptCounter.store(0, std::memory_order_relaxed);

    attachInterrupt(D2, testISR, FALLING);
}

void loop() {
    // Take the value from and clear it to 0 atomically. Even if
    // the ISR triggers at this exact moment, either the interruptCounter
    // will be updated before reading, or after setting to 0, but never
    // in the middle.
    counter += interruptCounter.fetch_and(0, std::memory_order_relaxed);

    if (millis() - lastReport >= 1000) {
        lastReport = millis();
        Log.info("counter=%lu", counter);
    }

}

void testISR() {
    // This increments the value atomically. Even if the ISR triggers
    // while we're resetting the value from loop, the count will
    // not be lost.
    interruptCounter.fetch_add(1, std::memory_order_relaxed);
}


4 Likes

@rickkas7,

Wow, thank you. A lot to think about here. I am going to take a stab at this after I do some reading on atomic variables. I will start here but there are some arguments you used like memory_order_relaxed that will require more searching.

https://en.cppreference.com/w/cpp/atomic/atomic

Chip

There is also a good article on using std:atomic in Interrupt Service Routines here:

1 Like

@rickkas7, @ScruffR, @peekay123 and @HEng ,

Thank you for your help on this. I think I understand this new concept (or am starting to) and have implemented it in my test environment. It seems to be working and I am no longer missing counts when the device is reporting or waiting for a web hook response. This combined with @rickkas7’s PublishQueueAsyncRK library means that I can catch every event while reporting and being responsive to Particle.functions. This is a big step forward.

Here is what I have done so far:

In the header, I identified the variables that needed to be Atomic

#include <atomic>

void sensorISR();

// Atomic variables - values are set and get atomically for use with ISR
std::atomic<uint32_t> hourlyAtomic;
std::atomic<uint32_t> dailyAtomic;

In Setup:

hourlyAtomic.store(0,std::memory_order_relaxed);
dailyAtomic.store(0,std::memory_order_relaxed);
attachInterrupt(intPin, sensorISR, RISING);                       // Pressure Sensor interrupt from low to high

The ISR looks like this:

// Here are the various hardware and timer interrupt service routines
void sensorISR()
{
  static bool frontTireFlag = false;
  if (frontTireFlag) {
    sensorDetect = true;                              // sets the sensor flag for the main loop
    hourlyAtomic.fetch_add(1, std::memory_order_relaxed);
    dailyAtomic.fetch_add(1, std::memory_order_relaxed);
    frontTireFlag = false;
  }
  else frontTireFlag = true;
}

From the main loop, I call this function if the sensorDetect flag is raised:

void recordCount() // This is where we check to see if an interrupt is set when not asleep or act on a tap that woke the Arduino
{
  static byte currentMinutePeriod;                                    // Current minute

  pinSetFast(blueLED);                                                // Turn on the blue LED

  if (currentMinutePeriod != Time.minute()) {                       // Done counting for the last minute
    currentMinutePeriod = Time.minute();                            // Reset period
    current.maxMinValue = 1;                                        // Reset for the new minute
  }
  current.maxMinValue++;

  current.lastCountTime = Time.now();
  current.hourlyCount += hourlyAtomic.fetch_and(0,std::memory_order_relaxed);   // Increment the hourlyCount from the atomic variable
  current.dailyCount += dailyAtomic.fetch_and(0,std::memory_order_relaxed);    // Increment the dailyCount from the atomic vairable
  if (sysStatus.verboseMode && Particle.connected()) {
    char data[256];                                                    // Store the date in this character array - not global
    snprintf(data, sizeof(data), "Count, hourly: %i, daily: %i",current.hourlyCount,current.dailyCount);
    publishQueue.publish("Count",data, PRIVATE);                           // Helpful for monitoring and calibration
  }

  currentCountsWriteNeeded = true;                                    // Write updated values to FRAM
  pinResetFast(blueLED);                                              // Turn off the blue LED
  sensorDetect = false;                                               // Reset the flag
}

Everything else functions as normal.

One question: why memory_order_relaxed?

Thanks,

Chip

2 Likes

This particular option is the least restrictive and lightest weight. I didn’t test the others, but I’m not positive they work from an ISR, but they’re not needed in this case anyway.

In this mode, atomicity is still guaranteed, but the order of execution is not guaranteed, especially with compiler optimization, and also between threads and the ISR. This is not an issue, because both the increment and the get and set to zero operations are atomic. It doesn’t really matter if the ISR increments first (returned counter get incremented sooner) or right after clearing (counter will be 1). No counts can be lost.

2 Likes

Hi Chip, thanks for raising this, great explanations and input from all here. Trying to understand why you are checking Particle.connected() before calling publishQueue.publish() which will queue the event and wait until Particle connected before sending - thus if your device weren’t Cloud connected you would lose the data?

@armor,

Right, the cool thing about publishQueue is that it can cache the messages until the device connected. Still, since my devices sleep most of the time, I did not want a big logjam of messages to be sent when it does connect. My intent is to put the “if connected” condition before messages that are not relevant unless they are read in real time.

This message is a good example as it is meant to provide near real time feedback when a car is counted.

Make sense?

Chip

1 Like

@chipmc Your explanation makes sense. I have a Jira ticket waiting for my attention entitled “device event backlog management”. I need to send all events to the cloud but similar to your sleeping beauties the product this ticket applies to is battery powered and thus it sleeps when not being charged. Due to the uncertainties of WiFi I have had a backlog manager working off of SD card since the beginning (and before publishQueue). I’m toying with the idea of creating a dual queue - the first is held in retained memory and is only small (4-5 events) and is only there to de-couple the application thread from the system thread when the device is cloud connected, the second queue is the same SD file based and stacks events when offline. This might sound silly but we have products that have been left off WiFi for months and the event backlog grows - whilst these events may no longer be relevant the web app really needs to decide whether to throw away the event data not the device.

Back to your near real time event, if the device can’t connect (and surely the cellular comms takes a while) then you are OK to lose the data, your is it that the next event will pick up the latest totals?

@armor,

Wow, months of backlogged message? The good news is that your device is WiFi connected otherwise you could blow through your cellular data cap. And one thing I have learned is that the Particle cellular data caps only work if you go through them slowly. This is one of the reasons why I do not allow for a big backlog on my Borons.

The other reason was mentioned in your response. I used to have a metering step before each publish:

bool meterParticlePublish(void) {
  static unsigned long lastPublish = 0;                                 // Keep track of when we publish a webhook
  if(millis() - lastPublish >= 1000) {
    lastPublish = millis();
    return 1;
  }
  else return 0;
}

and then put a waitUntil(meterParticlePublish); before each event. This tied up the device for a second and a big web hook payload could tie it up for an additional time. With meterParticlePublish, I have decoupled the threads like you have.

I know that publishQueue can support other storage such as FRAM and SD cards but I have not played with this yet. I wonder if you could have two instances of this library with one sending retained messages to the SD card and the other being saved in retained memory for your small system message queue?

As for my example, all data that needs to be stored is sent via web hook to Ubidots. I do have a process where data goes into an “in-flight” state once sent and is only recorded as “sent” and zeroed once I get a “201” response from Ubidots. Otherwise, the counts continue with the “in-flight” so a missed reporting period does not result in missed counts. This is actually an area of active development for me as I am rethinking how I track this data.

Happy to keep the thread going as I always get new ideas from these exchanges - good luck with that Jira ticket!

Chip

@armor,

One more thing, I have this worry about using the PublishQueueAsync - what happens if I make a mistake and load up the queue with junk such as from a runaway loop. I need to build in a “clear the cache” function before I go too much farther down this path.

Chip