Smooth your analog input using a Digital Filter

Most of us know that the analog inputs drift too much. A solution for this is to use a digital filter. There are many out there, some are simple, and some are more complex. I found this one that smooths the input of my thermostat temperature sensor SHT31 very well.

Read more about the filter here: adc - How to smooth analog data? - Electrical Engineering Stack Exchange

y = (1-α) x + αy

y = output, x = input, α = 0.9 or 0.99

Here is the code that shows how I am using it:

void SHT31()

{
cTemp = sht31.readTemperature();
RH = sht31.readHumidity();
fTemp2 = (cTemp * 1.8) + 32);
dewPoint = ((243.12 * log(RH / 100) + 4283.8 * cTemp / (243.12 + cTemp)) / (17.62 - log(RH / 100) - 17.62 * cTemp / (243.12 + cTemp))) * 1.8 + 32;
Blynk.virtualWrite(V1, RH);
fTemp3 = (1 - 0.99) * fTemp2 + 0.99 * fTemp3; Digital Filter to smooth readings
fTemp = fTemp3;
Blynk.virtualWrite(V0, fTemp);

Here are some iPhone screenshots. The first one shows how smooth the temperature is when raised to 81 degrees, turns ON the thermostat, and brings the temperature down to 79 degrees, etc.

The thermostat works more or less this way. The 79 is the temperature what I want to cool down the house, the 2 is a dead zone, the thermostat will not respond until the temperature reaches 81.

5 Likes

I like it! Thank you.

I’ve been using this one here, called Analog Smooth, and the author explains how to use it on his blog, and it looks like this:

#include <AnalogSmooth.h>
 
int analogPin = 1;
 
// Defaults to window size 10
AnalogSmooth as = AnalogSmooth();
 
// Window size can range from 1 - 100
AnalogSmooth as100 = AnalogSmooth(100);
 
void setup() {
  Serial.begin(9600);
}
 
void loop() {
  // Regular reading
  float analog = analogRead(analogPin);
  Serial.print("Non-Smooth: ");
  Serial.println(analog);
   
  // Smoothing with window size 10
  float analogSmooth = as.smooth(analog);
  Serial.print("Smooth (10): ");  
  Serial.println(analogSmooth);
 
  // Smoothing with window size 100
  float analogSmooth100 = as100.analogReadSmooth(analogPin);
  Serial.print("Smooth (100): ");  
  Serial.println(analogSmooth100);
 
  Serial.println("");
  delay(1000);
}

Cheers!

2 Likes

I’m a DSP person in my day job.

The smooth library looks nice and is very convenient, but looking at the code it is just a simple averaging filter. This is OK but there are much better digital filter designs depending on what bad things you are trying to remove.

An averaging filter like this is what we call a “box car filter” since it is as if you using an FIR filter with coefficients that are all 1/N. This has about 13dB of stop band attenuation close to the pass band. In contrast, a similar length FIR filter designed by DSP methods could have 20dB to over 80dB of stop band attenuation for the cost of some multiplies before the addition.

I have also done similar averaging in javascript on the page that reads the raw data.

The OP’s filter is a one-pole recursive filter and is a classic. Recursive filters depend not just on previous inputs, but also on previous outputs. They can be hard to get stable in larger filters so we usually design second-order sections or SOS filters (also know as biquad filters).

6 Likes

@bko,

Thank you for sharing your expertise. It is amazing the talent and experience in this community!

Is this a reasonable way of thinking about how to approach filtering?

  1. See if you need any filtering. Collect a bunch of data points and see (not 100% sure how) if you can see “noise”. The simplest example is to measure something slow moving like temperature and see if in a 10 second period, you see variation bigger than you can tolerate. If not, life is good - no filtering needed.
  2. If you do see variation big enough to matter, try a simple “box car filter” as it is a lightweight and simple solution. If that does the trick, life is good! If not, go to step 3
  3. If you need more precision, you may need to look at one of the filter techniques you listed and will need to take on the challenge of collecting the needed data (input and output) and tuning it to avoid instability. This option would, I assume also bring a significantly higher computational costs especially if you need to do this on the microcontroller.

Am I thinking about this right?

Thanks,

Chip

1 Like

Hi @chipmc

Yes, I think that sounds like reasonable and practical advice with perhaps one or two additions:

Step 0: Decide how fast you really need to acquire the data. I would argue that some things, and temperature is good example, can’t change very fast most of the time and reading them too fast just adds noise. Now there are exceptions, of course, like heating liquids with large heater coils or flames etc. But think about the actual rate of change of the data and if you really need to see the 0.1° C or F values–maybe rounding would be enough to remove a lot of the noise, for instance.

Most DSP techniques are made for uniform sampling, that is, each sample is taken at the same time interval from the previous one, but sometimes sampling every 10 minutes and then every minute if you see a problem is a better approach, so your code may need to be flexible. We don’t always get uniformly sampled data in the IoT world and in most cases that doesn’t really matter. But if you need it, there are ways to resample data from non-uniform spacing to a uniform grid as well.

Step 1.5: If you do want or need to do filtering, decide where it is cost effective to do it for you. You can certainly do it on the device, but that is often the most expensive place to filter and you should also at least consider post processing in the software that presents the data to users or takes action based on that data.

To understand what kind of filtering you might need, we generally look at the frequency spectrum of the data using FFT and similar approaches. Just taking an FFT is not always helpful and there are a variety of ways of modifying the data before and after an FFT to reduce problems that might be added by the FFT itself. Welch’s power spectral density estimation algorithm is a good place to start.

The frequency spectrum tells you both amplitude (how much there is) and frequency (how fast that part of the signal is changing) but it really cannot perfectly tell you both at the same time, so there are techniques for amplitude measurement and different techniques for frequency measurement, plus some broad ways to balance the tradeoff between the accuracy of each like Welch’s algorithm. The frequency spectrum can also tell you other things like what is called the instantaneous phase, but you are not likely to need that. In addition to filtering, there are also techniques for outlier removal which are sometimes used.

It’s a pretty big field.

6 Likes

It is AWESOME to get to know more about the science behind all these filters.

This is teleporting me to my Univ days with Fourier :wink:

Thanks @bko for the insights!

3 Likes

@bko,

Ok, yes I do remember FFT from college and have a vague idea of what it is doing. Your mention and a little reading on the inter webs helped me recall more - not sure if these are happy memories…

So, let’s give your modified approach a try and see what we can learn. This is a super basic example so, some of this is likely overkill. Still good to try.

Step 0 - I am measuring three analog values (temp inside the enclosure, temp in the soil and soil moisture). All change slowly and I am not in a rush so, I will sample once a second.

Step 1- I took 12 samples over 12 seconds and look at what they would be without filtering. Looks like there is more noise than I thought so - filtering it is.

0000233651 [app] INFO: Internal: 25.91C, Soil 28.63C, Moisture 82.00 
0000234652 [app] INFO: Internal: 25.83C, Soil 26.63C, Moisture 82.08 
0000235653 [app] INFO: Internal: 26.23C, Soil 27.72C, Moisture 82.19 
0000236655 [app] INFO: Internal: 26.15C, Soil 28.53C, Moisture 82.13 
0000237656 [app] INFO: Internal: 25.99C, Soil 26.56C, Moisture 81.97 
0000238656 [app] INFO: Internal: 26.23C, Soil 28.74C, Moisture 81.95 
0000239657 [app] INFO: Internal: 25.99C, Soil 26.56C, Moisture 81.97 
0000240658 [app] INFO: Internal: 26.15C, Soil 28.70C, Moisture 82.21 
0000241660 [app] INFO: Internal: 26.15C, Soil 27.96C, Moisture 81.95 
0000242661 [app] INFO: Internal: 26.07C, Soil 27.40C, Moisture 81.97 
0000243661 [app] INFO: Internal: 26.15C, Soil 28.46C, Moisture 81.97 
0000244663 [app] INFO: Internal: 26.15C, Soil 27.30C, Moisture 82.24 

Step 1.5 - I want to be able to collect a large number of samples and only connect to Particle occasionally to send. This could result in significant savings in power consumption. For this reason, I would like to take the data I need to collect, perform the smoothing and then store the one smoothed value in memory for the next connection event.

Step 2.0 - Trying the box car approach. I took the average of the 12 data points and compared it to the value I would get by simply looking at the integer value. Looks like averaging does help smooth out the data and would result in a different integer value about 50% of the time.

	Internal	S-Temp	S-Moist	Does Int equal Avg Int		
	25.91	28.63	82.00	Bad	Bad	Good
	25.83	26.63	82.08	Bad	Bad	Good
	26.23	27.72	82.19	Good Good Good
	26.15	28.53	82.13	Good Bad Good
	25.99	26.56	81.97	Bad	Bad	Bad
	26.23	28.74	81.95	Good Bad Bad
	25.99	26.56	81.97	Bad	Bad	Bad
	26.15	28.70	82.21	Good Bad Good
	26.15	27.96	81.95	Good Good Bad
	26.07	27.40	81.97	Good Good Bad
	26.15	28.46	81.97	Good Bad Bad
	26.15	27.30	82.24	Good Good Good
						
Max	       26.23	28.74	82.24			
Min	       25.83	26.56	81.95			
Average	   26.08	27.77	82.05			
StDev	   0.12	     0.83.   0.11			

I believe that, this may be good enough. Simply taking the average seems to create smooth value.

I looked at the Particle libraries for calculating a moving average - no clear winner I could see. Any advice would be appreciated. @gusgonnet , I looked at the library you shared but, the code was last updated 7 years ago which is a red flag for me. I also prefer to use libraries that are in the particle library system and - ideally - are both popular and validated:

search for “Average”

Step 2 - I tried getting the FFT for this data but, I don’t know if this is appropriate / reasonable for data collected at a 1/sec rate for values that change slowly. I thought it would be fun to try the steps though.

I don’t have MatLab or anything, so I went to an FFT calculator site on the web here:

I loaded up the values above and got the following graphs but, if I am being honest, I don’t know what to make of this. Perhaps sampling a faster moving signal and at higher rates would give a more insightful picture.

For this example, it feels like taking a handful of samples and then looking at the average might be good enough.

Thank you @italex , @bko and @gusgonnet for this thread. It has been interesting.

Thanks, Chip

1 Like

Chip, I’d tend to think by sampling Soil Temp & Moisture at 1/sec, that your just effectively graphing the precision of your sensors.
I generally measure soil parameters once or twice per Hour for Industrial Foundation Monitoring, and that’s during a forensic investigation.

Your data might not give your procedures a fair shake ?

1 Like

This is a great worked-out example and you are right that averaging is good enough almost all the time for the things we measure. Your standard deviations are pretty small–all your data is in a fairly tight range except for soil temp which has slightly higher variability. BTW rounding should produce a higher value about 50% of the time. I am certainly not saying that you need complicated algorithms, just that they are available.

You still seem to sampling pretty fast, especially for soil. Maybe every 10 minutes would still be more than enough.

I think you would need a hundred to thousands of data points to do any real Fourier analysis. With more data you might see non-linearity in your sensors or electrical noise from nearby motors or thermal noise from a nearby fridge or air-conditioner or heater switching on/off or similar. I’m just speculating of course, but those are the kinds of effects you can track down. Fourier transforms are just a tool to help you look for patterns in the data that you might not otherwise see.

Funny story–when I worked on consumer electronics in the 90’s, I learned about a new kind of electrical noise that they didn’t teach about in school: “Hoover” noise. This is the commutator noise from a vacuum cleaner that shows up as lines on the screen and audio interference on your television. Test labs would have a rack-mounted cheap vacuum cleaner that they could switch on that was on the same power as the device under test to induce Hoover noise!

Sorry if I am enjoying nerding out in this thread a bit–I think what you are doing is clearly good enough…

5 Likes

@Rftop ,

I may have not been clear enough about my timings:

  • When I sample - take readings 1/sec for 10-12 seconds
  • Sample 4 times an hour
  • Connect and report once an hour

So, I guess the rate of taking readings is fairly arbitrary.

On how often I sample, you are right soil moisture generally changes slowly but with this new sleepHelper library, I was thinking about moving to an adaptive sampling rate. For example:

  1. As I look outside my window now, a heavy rain is starting and the soil moisture levels will change rapidly. Graphing this can give you data on the true “saturation” level of the soil.
  2. When the sun is out in full force, things can dry out more quickly.
  3. Finally, Overnight the soil moisture generally changes little at all.

So, I should set a sampling rate to capture interesting changes in moisture and collect enough data points when I sample to get an accurate reading.

What do you think? Chip

1 Like

@bko,

I see, if I get a chance, perhaps I will let this thing run and take a look at what the FFT looks like with a much bigger sample.

I love the nerding out part. Even if FFT is overkill in this use case, it may well come in handy in the future.

Chip

1 Like

10-4 Chip, I’m tracking with ya now.

Now we’re getting into Geotechnical Engineering… the fun stuff
You’re going to get me excited like you did @bko :slight_smile:

image

4 Likes

Hi there, I’ve used a bubble sort to remove the outliers, then average the middle numbers.

For me this works well, I’m PID controlling a peltier to maintain a consistant temperature at 1sec intervals.

raw_temp = (float)analogRead(AN1_InsideTemp);
// load values into array elemente
fInsideTempArray[iInsideTempPtr] = (0.000006 * (raw_temp * raw_temp)) - (0.0009 * raw_temp) - 17.171 + 0.5;
// move point on
++iInsideTempPtr;
if(iInsideTempPtr > 4)
{
  iInsideTempPtr=0;
  // then sort the array smallest to largets
  fsort(fInsideTempArray, 5);
  // then average the middle three elements, this elimitinates outliers
  fInsideTemp = (fInsideTempArray[1] + fInsideTempArray[2] + fInsideTempArray[3]) / 3.0;
  // transfer inside temp to PID
  PIDInput = fInsideTemp * 10; // for one decimal place
}

The fsort function I borrowed, cant remember where, but Google says here
//Bubble sort my ar*e

//Bubble sort my ar*e
void fsort(float *a, int n)
{
  for (int i = 1; i < n; ++i)
  {
    float j = a[i];
    int k;
    for (k = i - 1; (k >= 0) && (j < a[k]); k--)
    {
      a[k + 1] = a[k];
    }
    a[k + 1] = j;
  }
}

Anyway, might be useful. Also, like ChipMC said, great suggestions from the community here.

4 Likes

@zonared ,

Thank you for pointing this out and for sharing your code. Makes sense to me to throw out the outliers first, then average. Great suggestion.

Chip

@Rftop ,

This may merit its own thread but, since you mentioned “saturation”…

I have had a number of discussions with @micromet on this topic - he is a guru on all things soil. Here is the net of what I am thinking after these discussions:

  • The value from the soil moisture sensor is a simple measure of how much water is in contact with the probe - 0% when dry and 100% when submerged in water.
  • In an of itself, this reading is not that relevant - it is a proxy for what you are really interested in - the “saturation” of the soil. And this is dependent on the soil and on its drainage among other things I am sure but I am an amateur here.

So, here is a process I am considering to change my measurements from raw values from the sensor to some measure of “saturation” of the soil.

  • Place the sensor and start with the raw values - this is all you can do until you get more data
  • Wait for a “major rain event” - will need to figure out what defines this event but, to start, a raw value of over 80%.
  • When you see this event, increase sample frequency so you can see the “peak” and you can measure the “shape” of the moisture curve.
  • One the event is over, watch for the soil to drain for either a fixed time (say 2 hours) or when you see a “knee” in the soil. This will be your new 100% soil “saturation” value.
  • Reduce the sampling frequency back to normal values and scale (linearly using map() ?) readings to reflect saturation.
  • Repeat calibration process above whenever the soil saturation goes over 100% as soil and drainage conditions can change over time.

See the graph below which shows two “major Rain events”.

So, does this make sense? If so, I will try to write some code to carry this out using some of the smoothing ideas in this thread.

Chip

I love this topic! Because I work in viticulture with a focus on vine water stress, this discussion gets my juices flowing!

Chip, in addition to determining Field Capacity (FC) (the upper limit), you also need the Permanent Wilting Point (PWP). The PWP occurs when the volumetric water content in the soil is too low for the plant’s roots to extract water. This value sets the lower limit. I tend to be more concerned with the PWP - for obvious reasons we want to prevent our plants from getting to this dryness. However, calculating the PWP is much harder to do. Whereas your method to determine FC is plausible. I would caution you to give the soil time to stabilize. After a rain event, it can take at least a day or two and often longer depending on soil texture to reach FC. Looking at your example data above, I would estimate FC around 67-68%.

As for the PWP, it seems that you are trying to keep soil moisture from dropping to or below 50%. Is that based on observations where you saw wilting? If so, that might be a reasonable value to use for the lower limit.

You could then map your readings from 68 (->100) and 50 (->0).

Just my two cents…

2 Likes

@wineisgud4u ,

Thank you for your post. My wife and I love visiting vineyards and would agree with the sentiment in your user name!

I like the term “Field Capacity” - that is exactly what I am after. I also appreciate that you can estimate this from the graph I provided. What I need to figure out is a way to get to that number algorithmically. Perhaps it could be as simple as the value when two successive samples change by less than some small amount (1-2%). Then, as you suggest, this will be the upper limit.

I also like the term “Permanent Wilting Point” - and, in fact, this is what I used to set the 50% value. I can observe my garden and see the plants start to wilt - the leaves seem to loose their rigidity and begin to droop. When I saw that, I looked at the reading and set the 50% value as just above that. This one is harder to imagine setting when the sensors are installed remotely. I wonder if there is a “rule of thumb” based on the plant. For example, I know that the root systems in vineyards can run very deep - much deeper than the sensor can reach so perhaps that would be different than some garden vegetable that only grows for one season.

I will play with this for a bit and share what I come up with. Thank you all for your advice and please let me know if I am missing anything.

Chip

1 Like

Great Discussion !
If you want to learn more about the soil science of triggering irrigation, there is a nice extension article here Understanding Soil Water Content and Thresholds for Irrigation Management | Oklahoma State University

6 Likes

Hey people @italex @chipmc @bko @Rftop @micromet @zonared @wineisgud4u

I have a question for you all here:

1 Like

@chipmc , I’ve been away for several weeks… just now catching up.

I have a few thoughts you might consider.
In soil mechanics, the king is the Phase Diagram. You can google that and read material for days.
The short version is soil consists of all 3 Phases (Air, Water, Solids).
Many soil parameters are defined by these simple relationships, and they can change drastically (porosity, saturation, etc).
I mention the Phase Diagram simply to have you think about your actual Sensor Installation.
You will have a hard time placing any soil sensor in an Un-Disturbed fashion.

I don’t know what type of soil moisture sensor you selected.
Can you install several sensors as a trial to see how well they agree ?
I’d think the best case scenario is an installation depth near the mean mass of the root system, not the surface.

You might also enjoy reading about Cation Exchange Capacity (CEC). I not an Agronomist by any stretch of the imagination, but CEC tells you how “available” the nutrients are to your plants. The short version is that adding Organic Matter to garden soil (and a pH adjustment) will do more for your plants than anything else.

[Side note] - For the past 4 years I’ve been playing around with Super-Absorbent Polymers (SAP) as a soil amendment. I’ve tested/used SAP for drying Sludge, and had plenty on hand. I love growing tomatoes in containers, but I’m away from home for long periods during the summer. The SAP is a game changer. You can hold unbelievable amounts of moisture for long periods of time. And you can also place the SAP deep in the container (or soil profile) to encourage deeper root growth early in the season. It’s like cheating the Phase Diagram.

4 Likes