DHT Sensor data filtering method for photon - some logic mistake?


#1

Hi there, I’m trying to work out a greenhouse automation. It works pretty well except for the bumpy ride with noisy sensor data. My sensor for gathering temp and humidity is DHT 22. It works ok, for this project, but generates noise. Small noise, due to the instability of the sensor, and also some ridiculous (1546 °C) spikes from time to time.

I thought that it would be nice to fight both these noises with some filtering. I couldn’t find any serious library for that task, and so decided to write the filtering on my own. I prefer to write this on my own, so that I understand what is happening inside.

My idea is to handle the noises accordingly:
Small noise - average from multiple readings
Spikes - High-pass and Low-pass filter that works on a threshold based on the last GOOD temperature.

Also, that the system has to self calibrate when it starts and doesn’t have the “last good temperature” to look for. This is zero at start. Could use setup() to get the first reading but i never know will this be or not a spike, so I decided for self calibrating method.

The problem is that I can’t get the high-pass + low-pass filter to work - probably due to some stupid logical mistake I’ve made. What the code I paste below do is go into infinite loop. Could you point me in to the right direction?

What I would like to happen is as follows:

  1. Check the DHT sensor for temp and humidity
  2. Check are the readings within a reasonable range (2 * last reading; should be ok for this application). But if the readings are off from the range for 10 subsequent times - eg. at start when the last temp is 0, this checking stage should accept the 11 reading.
  3. If yes than wait for another reading and add this to the variable. If not - repeat the reading.
  4. When enough repetitions happened (different function in my code) - calculate the average, and post this to Blynk.

BTW - I tend to convert (or cast when the extra precision is not needed) the floats into int’s to make the math faster.

I’m sure the problem is somewhere in these lines:

int check = 0;
    float H;							// Read Humidity
    do{
        H = dht.getHumidity();
        check++;
    } while (((int)H >= ((int)h+(int)h) && (int)H <= ((int)h-(int)h)) || check <= 10);
    check = 0;
    temp_Humid = temp_Humid + (H * 10); 
    
    float T;							// Read temperature as Celcius
    do{
        T = dht.getTempCelcius();
    } while (((int)T >= ((int)t+(int)t) && (int)T <= ((int)t-(int)t)) || check <= 10);
    check = 0;
	temp_Temp = temp_Temp + (T * 10);

But probably this might be not enough to get my logic in the code, so here is the whole code boiled down just to the relevant functions and variables:

void na_start()
{
   WiFi.selectAntenna(ANT_EXTERNAL);
   System.enableFeature(FEATURE_RETAINED_MEMORY);
   pinMode(D0, OUTPUT);
   digitalWrite(D0, LOW);
}

#include "Adafruit_DHT/Adafruit_DHT.h"
#include "blynk/blynk.h"

#define DHTTYPE DHT22		// DHT 22 (AM2302)
#define DHTPIN D4 
DHT dht(DHTPIN, DHTTYPE);

// temp and humidity
float h;
float t;
float dirt;

unsigned long previousMillis = 0;        
const long interval = 15*1000;           // interval for timed event in a loop
uint32_t lastReset = 0;

int repetition = 0;						// counter for how many times the repeted function happened

int temp_Humid = 0;
int temp_Temp = 0;
int dirt_temp = 0;

void after_repetition() {  // Will be called automatically after enough repetitions

	h = ((float)temp_Humid/repetition)/10;
	 Blynk.virtualWrite(V5, h);
	 temp_Humid = 0;
	 
	t = ((float)temp_Temp/repetition) / 10;
	 Blynk.virtualWrite(V6, t);
	 temp_Temp = 0;
	 
	 dirt = ((float)dirt_temp/repetition) / 10;
	 Blynk.virtualWrite(V7, dirt);
	 dirt_temp = 0;
	repetition = 0;
}

void setup() {
    lastReset = millis();
	
    pinMode(dirt_humid_pwr, OUTPUT);
    delay(3000); // Allow board to settle
    
	dht.begin();

    Blynk.begin(auth);
}

void loop() {
	Blynk.run();

	unsigned long currentMillis = millis();

  if (currentMillis - previousMillis >= interval) {
    previousMillis = currentMillis;
    
    int check = 0;
    float H;							// Read Humidity
    do{
        H = dht.getHumidity();
        check++;
    } while (((int)H >= ((int)h+(int)h) && (int)H <= ((int)h-(int)h)) || check <= 10);
    check = 0;
    temp_Humid = temp_Humid + (H * 10); 
    
    float T;							// Read temperature as Celcius
    do{
        T = dht.getTempCelcius();
    } while (((int)T >= ((int)t+(int)t) && (int)T <= ((int)t-(int)t)) || check <= 10);
    check = 0;
	temp_Temp = temp_Temp + (T * 10);         
    
    digitalWrite(dirt_humid_pwr, HIGH);	// Read dirt humidity ( I don't power the dirt humidity constantly due to the heavy corrosion it suffers while constantly ON)
    delay(5);
    int D = analogRead(dirt_humid);    	
    dirt_temp = dirt_temp + map(D, 0, 4095, 0, 1000);
    digitalWrite(dirt_humid_pwr,LOW);
    
    repetition ++;
  }
  if (repetition >= 4) {
      after_repetition();
      repetition = 0;
  }
}

Any help will be greatly appreciated. Thanks


Multiple DS18B20 with Photon
#2

@ethelder, you may want to consider a Bayesian or simplified Kalyan filter. Here in a post where an analog sensor was used and is perfectly applicable:

http://www.magesblog.com/2014/12/measuring-temperature-with-my-arduino.html

Here is an arduino library which implements the Kalyan algorithm:


#3

Hi @peekay123 thanks for your response.

  1. Definitely will try the aproach you suggested. Is the library from Github the same as Kalman 0.0.6 on Web IDE? I never used, on Photon, libraries from somwhere else, than from the IDE library catalog, so preferably would like to use that one.

  2. While the usage of libraries is always a pleasure - I would really appreciate somebody taking a look into my code, and trying to figure out where I screwed it up, since when I write it from ground up, I learn more and know why something work or doesn’t:) As an example you can find i my code the timer “hand written”. Why I don’t use the SparkCorePolledTimer library? I did and all was fine:) But with my code, I always know where the system is in any given time and can debug it, and with library - things might go wild, and I will never figure this on my own:)


#4

This is always zero. I am not sure this is what you intended.

You should check if the variable check is 10 when you exit the loops and not update the H and T values (i.e. set h = H).

Philosophically I would do the test math in float (or double) for more accuracy, but that is up to you.


#5

@ethelder, adding to what @bko indicated, your do {} while() will call the DHT22 conversion in quick succession (ie, no delay). The minimum sampling time for the DHT22 is specified as 2 seconds. I still am not sure your sampling logic is sound.


#6

Why not use a better temp / humidity sensor that’s more accurate and stable?

I would checkout the Adafruit SHT31 temp / humidity sensor since there is a Particle library for it and it never gives me junk readings.


#7

@bko Thanks! Didn’t see that. No, that was not my intention. Although this is a greenhouse and it’s unlikely to get the temperature below 0, I will change this statement to something more “all-aroundish”.

I also wanted to do this like that but one sentence from documentation frightened me:
[“Floating point numbers are not exact, and may yield strange results when compared. For example 6.0 / 3.0 may not equal 2.0.”][1]

@peekay123 I’m aware of the DHT22 sensor needing this delay, but just didn’t bother to put the conditional delay in there, since the do {} while() loop should only repeat itseslf if the reading is a spike (bigger or smaller than the range). Now I’m thinking about writing a second function that would only check for the reading being in range, and resulting in bool which would be used as a flag, for the do {} while() loop. Also this could be used for the delay, if needed.

@RWB Because, that is what I had in my junk. Because the idea of taking a cheapo sensor and puting it to work with some software magic is interesting for me. Because in the long run, all the sensors might have some spikes and I would love to learn (this is also why I’m not using a ready made library) how to sort such data out:) But you are right, the SHT31 is a great sensor, that I used in one project and it was a breeze to work with:)

Conclusion: my code needs some work, and when I will introduce the changes You suggested - I’ll check it on my hardware, and get back to you with findings. Thanks and Happy New Year everybody.
[1]: https://docs.particle.io/reference/firmware/photon/#data-types


#8

That ’ what I came up with:

Checking for being in range:

bool inRange(int val, int comparison)
{
  return ((-5 <= val) && (val <= (comparison + comparison)));
}

And than using it:

int check = 0;
    bool range = 1;
    float H;
    do{
        if (flag == 0) delay(2000);
        H = dht.getHumidity();
        check++;
        flag = inRange((int)H, (int)h);
    } while (flag == 0 || check <= 10);
    check = 0;
    temp_Humid = temp_Humid + (H * 10); // Read Humidity

What you say?


#9

That was stupid.

The problem was, that the do {} while() loop was allways going round 10 times. I’ve rewritten the whole thing and also implemented some fixed min/max values (after some thought I decided that if the temp would be like 2 C than my thershold would be 4 C, and that is too small.
Now it looks like that, and runs ok on Photon without going into infinite loops.

Here goes the code as is now on Photon:

bool inRange(int val, int minimum, int maximum)
{
  return ((minimum <= val) && (val <= maximum));
}

And the Low-pass High-pass filter looking like that:

int check = 0;
    bool flag = 1;
    float H;
    do{
        if (flag == 0) delay(2000);
        H = dht.getHumidity();
        check++;
        flag = inRange((int)H, ((int)h - 40), ((int)h + 40));
        if (check >= 10) break;
    } while (flag == 0);

    temp_Humid = temp_Humid + (H * 10); // Read Humidity
    
    
    check = 0;
    flag = 1;
    float T;
    do{
        if (flag == 0) delay(2000);
        T = dht.getTempCelcius();
        check++;
        flag = inRange((int)T, ((int)t - 20), ((int)t + 20));
        if (check >= 10) break;
    } while (flag == 0);
    check = 0;
	temp_Temp = temp_Temp + (T * 10);         // Read temperature as Celcius

#10

After some time I confirm - it works and makes the DHT 22 much more accurate:)

BTW - this sensor is prone to failure after it has been exposed to high moisture (85% up). Many users report, that after such a “bath” it starts to give randomly close to 0% humidity readings, and that is exactly what happened to me. The fix, was changing in the filter the min value from a dependable on the last reading = (int)h - 40 to a fixed 10%. I will not have such low readings in the place after all:) This makes the High Pass/ Low Pass filter even more valuable:)


#11

@ethelder Here is some more info I came across about how bad the DHT 22 sensor is compared to others.


#12

@RWB Wow, it is worse, than I thought:)

But on the other hand, it makes me smile, when with just some code, I can turn a crappy, failing sensor to output a rather clean looking 0.1 precision chart curve along some weeks:) In the end, it’s just a High Pass and a Low Pass filter combined with an average of four readings:)

And yes, I will change the sensor eventually.


#13

:smile: I just hate to think of tons of living plants suffering because of a low-cost sensor that is known to be junk and fail early.


#14

@RWB Don’t start! I had about 10 dirt humidity sensors, before I understood that what is eating them is corrosion from electrolysis caused by the sensor being constantly pluged to power. Since than I only power my dirt sensor 50 millies before reading to stabilise it giving me 4 minutes of ON time per day, and this made the last sensor to run over a month but you can see the degradation of it by naked eye.

I wonder, you have so many of those sensors on the market. Did nobody check that with their “by the guide book” setup they will die after a week?


#15

I was planning to use some dht22 in a project with a high humidity concentration like my city. rain every other day, near sea…

Is there any kind of alternative to be used by a digital pin? I don’t want to use I2C because I need to connect more than one sensors at the same time and I have read that I can’t do that with I2C since they cannot change address.


#16

I’m sure there are i2c temp sensors that offer more than one I2c address.


#17

@Suriken, @Suriken, I could only find I2C sensors with 2 addresses. If you need more than 2 sensors, you may want to consider using an I2C multiplexer like this one from Adafruit:

:smiley:


#18

The BME-280 breakouts by Adafruit can (also) use SPI. Plus they have the benefit of being reasonably accurate compared to the DHT-22.

Admittedly (like all of the Adafruit pricing) a bit expensive, it is just some directional guidance

:grin:


#19

@BulldogLowell, the unstated little dirty bit here is “I was planning to use some dht22 in a project”. I would assume that would mean at a distance from the Photon. SPI is not really suited to that. Nonethless, the BME280 is a fantastic sensor!


#20

@peekay123, thank you for your idea. A multiplexer could fit perfect into my project. Also, I read about SoftI2C, it could be even better if it works fine. I want to order some i2c sensor to test that.

About distance… I am planning to use sensors with wires. About a meter or 2 at most. Would this be a problem?