With the announcement of the next generation particle devices and more importantly On-board additional 2MB SPI flash! It’s sparked my interest again in creating a data logger.
I’ve bought the Adafruit SPI FRAM Breakout, so I can start working on this before July using my Electron… The end goal is to locally record sensor data and then send all the logged data once a day to the particle cloud.
As a starting point, I would like to achieve the following:
Sample temp and humidity data from a DHT11 Sensor every 1 Second and then log the min, max and mean values from the samples every 6 Min.
How would you go about doing this? Create an array with 360 elements as a buffer for each sample before storing the logged values in another array utilising the RTC to time stamp the data…?
I did something similar using two Arrays that are the same size to log the sensor data and time of each reading at the same time. Then you can go back and process the data along find the time of each array entry.
There are some Arduino libraries for getting the Min, Max, and Mean from Arrays that may be helpful also that I was looking over recently.
What your trying to do is fairly simple and should be doable with the memory of the current Photon or Electron.
I don't consider your frequency high, compared to me taking 50 readings from 2 different sensors and the current time every 10 Milliseconds.
Thank you @RWB have you got any sample code you can share please?
I would like to store at least 30 days worth of data in the Particle device before the data either rolls over or stores until full (user configurable) I would also like to have the Sample rate and Logging rate user configurable too, which I guess adds further complexity to how the arrays are created and data processed and stored.
@Alph, I believe the recommended minimum sample rate for a DTH11 is once every two seconds. Also not that the DHT11 is not an accurate sensor and tends to drift over time.
As for storage requirements, you can create a structure that has an array of 360*2 floats (temperature, humidity) and the min, max and mean. The code stuffs the array with values and every six minutes (360 samples), you calculate your min, max and mean. Then you store the structure to flash and restart. You will need to consider wear-leveling for the flash so consider using SPIFFS or other wear-level SPI flash libraries.
@peekay123 is the array the correct way of doing it then?
I’ve been reading an old data logger protocol design document from the early 90s and In there, all the data is stored in tables I.e Analogue sensor data in one table location and digital counter data in another table location etc I presume this is also done using arrays (there’s no code example and I don’t have it at hand now).
Regarding the DHT11 sensor, I just picked it because it was cheap and cheerful for testing and gave me real world data rather than using test code to insert the data. I’ve also got a 3 axis gyroscope that I could use for logging vibration but would need to sample at 100hz and prob log faster too.
Any more thoughts about the best way of going about this, would be very much appreciated!
@Alph, the structure simply provides a “package” to collect your data and write to flash. How you organize your data is really up to you and what you are trying to achieve. You need to step back and analyze your data collection requirements and what resources you’ll need to fulfill those requirements. Sampling data at 100Hz and trying to store to SPI flash may be a challenge when you factor in wear leveling. Have you considered using a microSD? They are faster, do their own wear leveling and the Particle SDFat library supports DMA bulk writing which is really fast.
The reason why I looked at FRAM was because I’m sure I read somewhere it has something like a trillion read/writes which I don’t think I will ever reach over my life time.
Also I don’t really want to additional hardware and power consumption to the product. Once I get a logging proof of concept working the next stage will be how to achieve 5 year battery life
I will think harder about how I want the data structured and reply tomorrow, thank you for your help and guidance
You could cut back a lot on your data collection to make your life easier. If you’re measuring room or outdoor temps and humidity, taking a reading every second is really excessive; those values just don’t change that fast. Even with the six minute logging, I doubt that the min, max, and mean values will often be significantly different from each other, and to the extent that they are, you may just be looking at noise in the readings. I think that taking a reading once per minute and reporting the average value every 6 minutes (with some quality control to throw out any obviously bad numbers) would be more than sufficient for environmental monitoring (assuming that’s whaat you’re doing).
I’ve been logging weather data from a Davis weather station for years taking a sample every 10 minutes. The sample contains the instantaneous temperature, and the min and max temps over that 10 minutes; the differences I see between the min and max temps are almost always within the repeatability of the device.
Hi @Ric I only picked the DHT11 because it was a cheap real world sensor. I want the logger to be sensor agnostic (obviously will have to implement different libraries for different sensors though). One of the applications I would like to look at is water pressure transient logging for leak detection, hence the high frequency data collection.
@alp, the high rate of sampling is to catch an event. However, you may simply log the event as max amplitude, time, etc. and not all the sampled data. This is what @Ric was eluding to.
As far as I understood it; you would log normally 1sec or 10sec sampling with say a 10min logging rate and then under alarm conditions, switch to a high frequency sampling/logging of 100hz for X time to capture the event… I could have totally misunderstood though haha
Well, yes, if you’re only interested in recording a critical event, keep a rotating buffer of data and then on a trigger write your buffer to long-term storage. This should allow you to record data before and after your trigger event.
This is exactly what I was thinking, I need to take a step back like @peekay123 said and come up with a structure on how I want it to work, before I even start worrying about code I think.
I know enough to read sensor data, upload it to the cloud and store it in a database but trying to store it locally and working with arrays is all new to me.
I've recently gone down this road of using Arrays and on-device storage, it's easier than I expected. A few google searches and I found all the examples I needed to get up and running.
I’ve made a start, I think there is a lot more in my head than what I’ve written, if none of it makes sense then please shout up but, I would like to get your thoughts and ideas on wheres the best place to start code wise to make this a reality?
The Data logger firmware will be made up of 4 Tables/Arrays:
Master Definition
Data Definition
Data Store
Data Buffer
Master Definition
Index Number
Description
Element
Logger ID
0 - 65,535
Firmware Version (prefix)
0-255
Firmware Version (suffix)
0-255
Power Supply Alarm Status
Battery Low
Battery Normal After Low
Power Supply Alarm Mask
Alarm On / Alarm Off
Start / Stop / Clear Logging
Start Logging
Stop Logging (Standby)
Clear Data
Current Time (seconds)
0-59
Current Time (minutes)
0-59
Current Time (hours)
0-23
Current Date (day)
0-31
Current Date (month)
1-12
Current Date (year)
0-99
Current day of week
1-7 (Sunday = 1)
Running Time
0 - 65,535
Incremented every hour
Data Definition
Index Number
Description
Element
Logging Rate
0 - 1440 mins
Units
Temperature in degrees Centigrade, Electrical Current in Amps, Gauge Pressure in bars etc.
Lo Alarm Level
0 - 65,535
Hi Alarm Level
0 - 65,535
Current Sensor Value
Sampled on request
Data Count
0 - 65,535
Alarm Status
Hi
Lo
Alarm Mask
Alarm On / Alarm Off
Hi Alarm Time (Minutes)
0-59
Hi Alarm Time (Hours)
0-23
Hi Alarm Date (Day)
1-31
Hi Alarm Date (Month)
1-12
Hi Alarm Date (Year)
0-99
Lo Alarm Time (Minutes)
0-59
Lo Alarm Time (Hours)
0-23
Lo Alarm Date (Day)
1-31
Lo Alarm Date (Month)
1-12
Lo Alarm Date (Year)
0-99
Elements below can only be changed when the logger has stopped logging or the data has been cleared-
• Logging Rate
• Units
The Data Count element will increment every time a new value is added to the data store (e.g. once every 5 minutes for five minute logging).
Data Store
Data is stored consecutively in a Data Store table
The data store is a stack of values, the latest logged value will be pushed on to the stack
The existing data value index will be incremented
When the logger starts logging a data header is written:
Index Number
Description
Element
Start-of-Log Date (Year)
0-99
Start-of-Log Date (Month)
1-12
Start-of-Log Date (Day)
1-31
Start-of-Log Time (Hours)
0-23
Start-of-Log Time (Minutes)
0-59
Start-of-Log Time (Seconds)
0-59
The Data Store table will be a rotating data store i.e when the latest value is pushed on to the stack the oldest value will drop off (deleted)
Sensor Data is sampled up to 100ms and kept in a Data Buffer table, the maximum, minimum, mean and standard deviation over the logging rate period (set in the Data Definition table) is calculated and stored in the Data Store table stack.
I'm not sure this is acheivable. I'm still not clear on your sampling rate, collection time and total data per collection is. I'm also not clear on how often the data will be uploaded.
