Noise Cutting Algorithm

General
1

Hi Team,

I am using a module which reads the liquid flow in every 125ms. The issue is there can be some noise from the chip, even if there is no flow. We could easily detect the noise if we look at the readings manually.

Now I am planning to implement a queue with constant length and average the values, check with new readings, add or remove from the queue. However, there is a chance that I will miss a few readings even if it is actual water flow.

Is there anyone who could help with a better method to handle this challenge?

Thanks for your time.

2

Are you reading the flow meter over I2C, or is it interrupt driven?
I’m guessing you’ve already tried to eliminate the source of the noise, but it doesn’t hurt to ask :grin:

Since you are taking 8 readings a second, I’m also guessing this isn’t a positive displacement meter.

3

What makes my ears perk up and think that this is a solvable problem is the fact that you can easily visually see what is noise and what isn't. I've always found that if I can see the signal in the data, I can find a way to eliminate it in at least 90% of cases.

Can you post an example of the signal and the noise? There are countless techniques for filtering and decoloring noise, and choosing the right one requires understanding what kind of non-gaussian noise you're experiencing, as well as what kind of delays you can afford in the signal.

3 Likes
4

Hey guys sorry for the delay. Really caught up in something.

@Rftop I am reading this over SPI. We couldn’t eliminate it from the source. Yes, it is not a positive displacement meter.

@kubark42 We are getting the fluid movement in the form of Hexadecimal numbers after some conversions and calculations. The decimal representation of noise will be like, 0.1, 0.2, 0,1, 0.3, 2.0, 0.1 … .If there is no flow, we are getting like 0.15 gallons per minute. This is because sometimes there is a high spike in the readings, which cause this noise.

What I did was implementing a queue, with add and delete functions and the queue size is 8. I will take the average of all the data in the queue, check with the difference with new data. If it’s in a specified limit add to the total volume else not add to the total volume but to add it in the queue as it can be the start of the high flow, take average continue. I could eliminate the noise by half using this method still, I may miss some data.

Do any of you guys have a better way to do this?

5

If you’re looking to get rid of a very occasional spike, use a 1D median filter. It’s a very good filter which introduces minimal delay and is bog-simple to implement.

If it’s really just a single data point then you can get away with a 3 sample filter, which will introduce at most a 2 sample delay, but realistically the delay will hover around 0-1 samples.

So assuming that your original data series was:
0.1, 0.2, 0,1, 0.3, 2.0, 0.1, 0.2, 0,1, 0.3
then the output of a three sample median filter would be:
0.1, 0.2, 0.1, 0.2, 0.3, 0.3, 0.2, 0.1, 0.2

Afterwards, you can pass it through an 8-sample averaging filter:
0.1, 0.15, 0.133, 0.18, 0.24, 0.2, 0.2, 0.1875, .2

Which will give you a much better average now that you’ve thrown out the outliers.

Another approach is to average together a long window throwing out the max and min value as outliers, but in your case if your minimum values are reliable and only your maximum values are unreliable than you could do something like throw out the peak value (or peak two values, or three, or four, etc…) before taking your average. Either filter will work much better than simply averaging in known erroneous data.

1 Like
6

What if you sort the collected values, and take the median value (middle), instead of the mean?

1 Like
7

Since you mentioned that you can visually spot the noise, is it safe to assume that the actual flow characteristics don’t change quickly over a short period of time ?
In other words, when fluid does actually start flowing, it continues to flow for more than 8 samples ?
If so, you could fill an array each second and don’t actually totalize the flow until all of the elements are over your minimum threshold. You would know those 8 samples are valid and should be counted, and continue to count until the minimum threshold is no longer met.

Just curious, what’s the specified accuracy of the flow meter ?

8

Have you considered using a Kalman filter? I believe there is a library available in the web IDE.

1 Like
9

Kalman on a 1D system without a model is just a first-order low-pass filter with optimally chosen coefficients. Probably not worth the effort. And it would still be best to decolor the noise as much as possible before feeding to the KF.

10

Thanks for the input guys. I am working on IST so couldn’t reply all of you on time.

Let me go through all of them. I think the median is a good idea.

@Rftop The flow will be sure for more than 8 samples. I will update you with the accuracy. We could even identify very small flow like 0.08 gallons per minute.

@peekay123 I will check out the library too. Thanks

11

Finally went with the median. Took readings from the sensor at 250ms, added into an array upto 8 numbers, sorted using qsort. Took the median of it.

Thanks for the help guys …:clap::clap:

1 Like