Smoothing out sensor spikes

I have a DHT22 that will occasionally send out erroneous data spikes (either negative or positive)
I’m looking for a way to ignore any readings that exceed +/- 5% since I read every 60 seconds.

Compare it to the moving average?

yea thats pretty much the idea, im just not too sure how to address it

Use a median filter. A median filter is a type of filter which picks the median value during the sample window. So if your data set is [1,2,3], then 2 is the median value. If it’s [1,3,2], the median value is still 2. If it’s [10,0,3000], then the median value will be 10.

Choose an order (i.e. size) which is sufficient for your spikes. If they are single data points, then a 3rd order filter is the smallest, fastest option.

The only concern with median filters is that they are non-linear. Your sample can be not delayed at all, or delayed by up to n-1 samples. If this isn’t a problem, e.g. because your sample rate is high relative to your process speed, then you’re golden.


Example:

Let’s follow a theoretical 1D, 3rd order median filter through a spike:

input | filter window  | output
42    | [42]           | 42
30    | [42,30]        | 36 (don't worry about this case, it has to do with the formal definition of the median when there are an even number of elements)
41    | [42, 30, 41]   | 41
25    | [30, 41, 25]   | 30
42    | [41, 25, 42]   | 41
39    | [25, 42, 39]   | 39
3000  | [42, 39, 3000] | 42
29    | [39, 3000, 29] | 39
24    | [3000, 29, 24] | 29
1 Like

thank you so much! this points me in the right direction, definitely.
I do have occasional issues with the DHT22 reporting back with no value, probably 1 in every 20. Would this cause an issue with the median filter?

So I used Rob Tillaarts RunningMedian library and its “working” but the results Im getting through my publish event are now ints even though I declared everything as a float.

{
 "d":[
    0:[
       0:29
       1:83
       2:24.82
       3:0
       4:3.69
    ]
  ]
}
// This #include statement was automatically added by the Particle IDE.
#include <RunningMedian16Bit.h>

// This #include statement was automatically added by the Particle IDE.
#include <ThingSpeak.h>

#include "Adafruit_DHT.h"

// Example testing sketch for various DHT humidity/temperature sensors
// Written by ladyada, public domain

#define DHTPIN 4     // what pin we're connected to

// Uncomment whatever type you're using!
//#define DHTTYPE DHT11		// DHT 11 
#define DHTTYPE DHT22		// DHT 22 (AM2302)
//#define DHTTYPE DHT21		// DHT 21 (AM2301)

// Connect pin 1 (on the left) of the sensor to +5V
// Connect pin 2 of the sensor to whatever your DHTPIN is
// Connect pin 4 (on the right) of the sensor to GROUND
// Connect a 10K resistor from pin 2 (data) to pin 1 (power) of the sensor

// Particle Core, Photon, and Electron the results are published to the Particle dashboard using events.
// Go to http://dashboard.particle.io, click on the logs tab, and you'll see the events coming in. 
 TCPClient client;

RunningMedian tempCsamples = RunningMedian(3);
RunningMedian tempFsamples = RunningMedian(3);
RunningMedian humidityPsamples = RunningMedian(3);


DHT dht(DHTPIN, DHTTYPE);

void setup() {
	Serial.begin(9600); 
	Serial.println("DHTxx test!");
    ThingSpeak.begin(client);
	dht.begin();
}

void loop() {

// Reading temperature or humidity takes about 250 milliseconds!
// Sensor readings may also be up to 2 seconds 'old' (its a 
// very slow sensor)
	float humidityP = dht.getHumidity();
// Read temperature as Celsius
	float tempC = dht.getTempCelcius();
// Read temperature as Farenheit
	float tempF = dht.getTempFarenheit();
  
// Check if any reads failed and exit early (to try again).
	if (isnan(humidityP) || isnan(tempC) || isnan(tempF)) {
		Serial.println("Failed to read from DHT sensor!");
		Particle.publish("Failed to read from DHT sensor");
		return;
	}
	
	
	tempFsamples.add(tempF);
	tempCsamples.add(tempC);
	humidityPsamples.add(humidityP);
    float tempFmedian = tempFsamples.getMedian();
    float tempCmedian = tempCsamples.getMedian();
    float humidityPmedian = humidityPsamples.getMedian();


// Compute heat index
// Must send in temp in Fahrenheit!

    float heatIndex = dht.getHeatIndex();
	float dewptF = (dht.getDewPoint() * 9/5) + 32;
	float tempK = dht.getTempKelvin();
	ThingSpeak.setField(1,tempC);
	ThingSpeak.setField(2,tempF);
	ThingSpeak.setField(3,humidityP);
	ThingSpeak.setField(4,dewptF);
	ThingSpeak.setField(5,tempK);
	ThingSpeak.setField(6,heatIndex);
    ThingSpeak.writeFields(xxx, "xxxx");
    
    float voltage = analogRead(BATT) * 0.0011224;
    

Particle.publish("app/output/data", String::format(
      "{"
        "\"d\":["
            "["
                "%5.2f,"
                "%5.2f,"
                "%4.2f,"
                "%4.2f,"
                "%4.2f"
            "]"
        "]"
        "}",
        tempFmedian,
        humidityPmedian,
        dewptF,
        tempCmedian,
        voltage
        ), 60, PRIVATE);



	delay(60000);
}






I’m a little confused, it looks like you’re getting floating point responses, no? Or am I looking at 2: 24.82 and 4: 3.69 wrong? I’m interpreting those as the ordered your filter for the various tempFmedian, humidityPmedian, dewptF, tempCmedian, voltage fields.


<rant>
BTW, is this the lib, https://github.com/RobTillaart/Arduino/blob/master/libraries/RunningMedian/RunningMedian.cpp? If so, ugh, that’s an obnoxiously commented and documented library. Don’t write code like this, boys and girls.

It’s poor coding practice to make people guess at what algorithms you’re using. Good math code always documents the algorithm and approach in extreme detail, so that if and when the inevitable bug sneaks in a reviewer can find it by referring to the original mathematical proof. It can be almost impossible to go from the implemented code to the source algorithm. (C.f. https://en.wikipedia.org/wiki/Fast_inverse_square_root#History_and_investigation, “It is not known precisely how the exact value for the magic number was determined…”)

Take for instance, the line at https://github.com/RobTillaart/Arduino/blob/master/libraries/RunningMedian/RunningMedian.cpp#L75: if (_cnt & 0x01) return _ar[_p[_cnt/2]];. Is it at all obvious what this is doing? Furthermore, is it obvious it’s doing the right thing? No, and no.

FWIW, what it’s doing is saying that if the number of elements is odd, then take the mid-point rounding down. It uses binary operators (_cnt & 0x01, because the final bit in an integer is 1 for odd numbers and 0 for even) and integer math (e.g. 5/2 = 2) to achieve this, as well as silently requiring that the array be 0-indexed.

It would have been much more readable to have used enums, such as #define IS_ODD(x) (x & 0x01).

That being said, the lib is probably safe to use by dint of it’s 10 year history. It’s just not doing us any favors by being needlessly cumbersome to review.

</rant>

Agree with your rant, and I figured out my issue. Turns out there were two versions of this in the Particle community library. I chose the most used, when I should have really read the example code on each first.
The RunningMedianST is the correct choice for floats

image

Ahhh… and I had found it by bing (googling on bing?), so when I inspected the code I could see it was faithfully handling floats. Glad you got it sorted out! Let us know if it’s a full solution to your problem

You could always inject a NaN when there’s no value. You can also simply not inject a number into the filter at all. In the latter case, your filter would continue to give you the same answer until you get new data.

Personally, if timing weren’t imperative then I’d go the second route. Then there’s nothing to program.

BTW, 1 failure out of 20 is crazy. I would be concerned even at 1 failure out of 1000 samples. Dollars to donuts, when there’s this high of a data loss there’s something very wrong either with the setup or driver.

Im using the Adafruit DHT library and an AM2302.
It seems pretty straight forward, so its possible the sensor aint no good no longer, yknow (its about 5 years old and was sitting in a drawer)

Possible. My first thought would be to investigate why there’s no report at all. The only part of a sensor which is going to decay in such a short time is the analog side. And when that starts failing, you still should get some kind of reading. My thoughts are the driver or the code which reads the driver.

Although maybe you should count yourself lucky to have such a clear indication of bad data! A lot easier to prune out a missed sample than a corrupted one.

you pointed suspicions in the right direction. I changed out the Adafruit library for PietteTech_DHT and its all working great now without NaN

cheers

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.