Open Energy Monitor Port

Social Project Share
1

OK here goes. Here is a port of the monitoring part of the Open Energy Monitor - very little needed to be changed. I had an Arduino/Raspberry Pi energy monitor running for a couple of years but the Spark frees me from a long run of wires and gives me more processing power without going to the expense of Zigbee.

The Spark has only been running for a few days so doubtless it is still very buggy. I had some strange behaviour with the ADC readings as the Spark Core team have been going through the process of improving the firmware. With some recent firmware the ADC values have been odd but the power factor has remained correct so everything kept working. I will dig into this in a lot more detail when I next have time in a couple of weeks.

The system is based on a Spark that sits close to my two electricity meters (domestic and PV generation) and my gas meter.

The domestic meter is monitored by a clip on current sensor.
The voltage and relative phase is measured with an ac adaptor.
Both of these are straight from the Open Energy Monitor project http://openenergymonitor.org

The PV meter is inaccessible for a current sensor so I monitor it and also the domestic meter with photoresistors which detect the LED flashes which on my meters occur every Joule. A third digital input measures my gas consumption using a reed switch.

So the inputs to the Spark are 3 digital interrupts and two 50Hz low voltage analog signals.
The Spark acts as a UDP server and reports the energy consumption (actually the average power) since the last poll.

The Spark is polled by a Raspberry Pi which is sited conveniently close to my router and the energy display. It records the data on a USB hard drive, plots it to an image file used for a web page and sends it to an Arduino-driven 20x4 lcd display which also has a button to define whether my immersion heater should be on, off or controlled by the Pi depending on the solar PV. There are doubtless better web displays and better 20x4 displays but these work for me. The other reason for keeping the Pi is that it drives a Telldus USB stick that remotely switches my immersion heater and other electrical equipment.

The photos show the Spark on a Veroboard - there is very little external circuitry. The ribbon connector goes to the sensors.

There is a magnet built into the 1/100 m3 wheel of the gas meter and this closes a reed switch (attached with blu tack) once per revolution.
The photoresistors are poked down the tube of an old pen and attached to the meter with blutack.
The display consists mainly of a bar which has a length proportional to actual electricity use with the horizontal position showing the import/export balance. Rather busily is also shows the gas consumption and the energy cost for the past 24 hrs.
The web display shows a running 24 hrs generation and use on a Ā± 4kW scale superimposed on the gas unfortunately for my bills on a 0 - 40kW scale.

Iā€™ll fill in more details in a week or two but in the meantime there are several potentially helpful snippets:

ADC measurements faster than the standard loop() will give you
A simple port of Open Energy Monitorā€™s emon.lib
Spark interrupts triggered by flashes and a reed relay
UDP communications with a Raspberry Pi
Serial communications between Raspberry Pi and an Arduino controlled display.

It may seem a little odd that I use 3 different devices. It is mainly historical - I originally had a 4 core telephone cable from the meters to the Arduino display which then delivered data to the Pi for storage and to act as Web server and immersion controller. The Spark does away with the wires and allows me to collect more information but I still need a ā€œrealā€ computer to control the spare PV power and since the Arduino already does a good job controlling the display I havenā€™t reworked it.

The Spark, Pi and Arduino code is at: https://gist.github.com/phec/9254793

The Spark

Photoresistor on generation meter

Reed switch on gas meter

Simple display

Busy display

Web image

11 Likes
2

Wow this is really cool! Shall learn more from you since my project is somehow related to power monitoring :slight_smile:

3

What a cool project! Thanks for sharing @phec.

What was odd about the ADC values? Anything we can help debug?

4

This is great. To clarify, is the spark core taking all the measurements and the other devices are just for logging and control?

1 Like
5

The Open Energy Monitor Project is a good one that I can see the Spark Core being used in.

There are many features that @phec is not using that the Open Energy Monitor Project is capibable of that are pretty cool.

There is a wireless display that shows the system and weather status. http://openenergymonitor.org/emon/emonglcd



And then they have a killer online interface that you can customize as you like. The spark core can be used to get this data from your system to the internet for logging and graphing.

I personally would like to explore using the Spark Core to push data to this free online interface: http://emoncms.org/site/home

Here is an example of their customize able online interface. http://www.emoncms.org/jb79/Solar1

Just wanted to expand on what is possible with the Open Energy Monitor Project. Iā€™m looking forward to seeing the Spark Core being used in this open source system.

@phec thanks for being the first to share this and how your using the Spark Core instead of the Arduino equipment.

2 Likes
6

@kareem613 Yes thatā€™s right. The sensors all connect to the Spark and the Spark provides a UDP service to the other devices, essentially sending them the three meter readings.

7

@Zach thank you for the offer. The symptoms are of sampling too fast and not allowing sufficient settling time (too low a reading) on the part of my software, but it came and went with revisions to the core firmware.
In a few days I will separate out purely the ADC part of the software and collect some unambiguous data and plot some graphs then there will be some concrete evidence to pursue.

Sampling a 50Hz 0.6 volt ac signal.
This is what I have at the moment which looks good to me. The ADC is working well with the latest core firmware. You can see that the 41.5uS limit on sample time is kicking in so there is no significant difference between the 5uS delay and the 25uS delay in my measuring loop. This is fine for monitoring 50Hz but it limits sampling to about 10kHz signals if my sums are right (1 sample per 2*46uS). That messes up my plans to listen to insects in the ultrasonic range but I can see that for static measurements it is a good idea.
What do I need to edit in my local copy of spark_wiring.h or spark_wiring.cpp to get back the faster sampling?
The messy waveform may come from sampling on the same power block as several beefy switch mode power supplies.
@BDub This uses a 0.1uF decoupling capacitor.

/* 
 * File:   PCtestADC
 * Author: Pete
 * This waits for a ready signal and sends a string to the client
 * Created on 08 February 2014, 11:36
 * Uses Serial
 * 
 */

/* Includes ------------------------------------------------------------------*/

#include "application.h"


/* Function prototypes -------------------------------------------------------*/

/* Variables -----------------------------------------------------------------*/
int LED = D7;
int ADC = A0;
//
//char UDPincomingData[256];
//char UDPoutgoingData[256];
unsigned long lastRead;
unsigned long t1;
unsigned long t2;
unsigned long t3;
unsigned long t4;
int lag;
int netTime;
int i;
int j = 0;
unsigned int ADC0[512];

/* This function is called once at start up ----------------------------------*/
void setup() {
    pinMode(D7, OUTPUT); // Turn on the D7 led so we know it's time
    pinMode(A0, INPUT);
    digitalWrite(D7, HIGH); // to open the Serial Terminal.
    Serial.begin(9600); // Open serial over USB.
    while (!Serial.available()); // Wait here until the user presses ENTER in the Serial Terminal

    digitalWrite(D7, LOW);
    Serial.println("Ready");
    do{
        ADC0[0] = analogRead(A0);
    }
    while ((ADC0[0]<2036) && (ADC0[0]>2060));
    lastRead = millis();
    t1 = millis();
    t2 = millis();
    digitalWrite(D7, HIGH);
    for (int i = 0; i < 500; i++) {
        ADC0[i] = analogRead(A0);
        //digitalWrite(D7, !digitalRead(D7));//delay  
        delayMicroseconds(25);
    }
    t3 = millis();
    delay(500);
    digitalWrite(D7, HIGH);
        Serial.println(((t3-t2)-(t2-t1)));
        Serial.println(((t3-t2)-(t2-t1))/500.0);
    for (int i = 0; i < 500; i++) {
        Serial.println(ADC0[i]);
        delay(10);
    }
}

/* This function loops forever --------------------------------------------*/
void loop() {
}
8

Thanks @RBW yes the Open Energy Monitor project has some great stuff and it looks really smart. I guess the Spark could simplify EmonTx quite a bit.

9

Here is the DIFF between DMA ADC and the way it used to be: Dual Slow Interleaved ADC Mode with DMA for High Impedence & Accuracy Ā· particle-iot/device-os@7e2b448 Ā· GitHub

Before you revert the DMA stuff, simply try changing the ADC_SAMPLING_TIME ADC_SampleTime_41Cycles5 to the fastest setting ADC_SAMPLING_TIME ADC_SampleTime_1Cycles5

1 Like
10

This is one of the best projects i have seen for the spark! and may help me overcome a problem i can foresee with the next project i buildā€¦ plus give me heaps and heaps of extra features.

How sensitive is the current clamp? will you see a difference when a 100w globe gets turned on? what about a 15w compact fluro?

The reason i ask is because i wanted to control all (well the 8 remote control ones anyway) the lights in my house with the spark using a 433mhz module, one problem i can see is i donā€™t know the current state of the light, and i donā€™t fancy putting ldrā€™s in each room. if i could see the change in power consumption then i could determine its state.

@phec have you looked at using a 433mhz module direct from the spark instead of the Telldus usb stick? they are so cheap on ebay only a few bucks for a setā€¦

11

Here is a good thread for on RF radios. https://community.spark.io/t/nrf24l01-library-ported/2286

12

I was actually looking at these radios http://dx.com/p/433mhz-rf-transmitter-module-receiver-module-link-kit-for-arduino-arm-mcu-wl-green-220194

As my lights and fans are on this 433mhz frequencyā€¦ ill use the receiver to grab/learn the data sent by the original remote and then just sent the data from the core using the transmitter. hopefully its fairly simple and there are a few arduino projects doing it this way.

13

Cheaper is better as long as it works how you want it to.

Be sure to show us what you come up with when you get your idea up and running. I'm interested in seeing more of this type of stuff.

The link below is talking about the sensitivity of the current clamp sensor and if I'm reading it right its saying the wattage has an error rate of up to 30w. 30w being the smallest measurement it can make wattage wise.

http://openenergymonitor.org/emon/buildingblocks/measurement-implications-of-adc-resolution-at-low-current-values

To see what numbers you might expect in practice, say the voltage is 240 and the current sensor is the standard 100 A CT with the standard burden resistor (18Ī©). 100 A rms represents a peak-to-peak count of about 800, so our wave with a peak-to-peak amplitude of 1 count has an rms value of about 0.125 A. This represents a true power of 30 W. Therefore, in the second case we might read 13.5 W and 0.09 A, and even though we have unity power factor, we read an apparent power of 21.6 VA and a power factor of 0.625.

But you will get exactly the same result if the current is the smallest imaginable whilst still being just sufficient to flip the analog to digital converter between two adjacent values. So in practice, you might read anything from zero power and current to a power of 13.5 W and a current of 0.09 A.
On the other hand, you might read zero power and zero current for any load up to 30 W.

Up to this point, we've assumed the Arduino ADC input comes from the current sensor. It may also come from digital noise radiated from the Arduino itself, or outside noise picked up by the PCB or the wiring.

14

Looking at the EmonTx v3 there is a high accuracy CT input. basically they change the burden resistor which reduces the max power from 23kw to 4.6kw but gives higher resolution. If i did this on my light circuit, which will hopefully not go anywhere near 4.6kw (even with the security flood lights).

Also looking at the changes to port this to the core, it appears the core has a higher resolution ADC? which will improve this even more?

Just checked my gas and water meters, both allow a reed switch to be connected!

Not sure how i will implement it into the final project, i might develop a ā€˜dual coreā€™ system. one core that monitors the power, gas and water and uploads to the web mounted somewhere near the meter box, and a second core with a display that reads from the web and controls the lights using the 433mhz transmitter

15

@Hootie81 I already had the Tellstick so thatā€™s why I followed that route. Telldus were looking at an Arduino interface but they got sidetracked into more consumer oriented developments - very sensible commercially.
If you dig around you will find people have decoded some common transmitter signals using the transmitter/receiver you found. Thereā€™s quite a bit on the Arduino Forum - for example: http://forum.arduino.cc/index.php?topic=188027.0 If you try this Iā€™d be very interested to hear how you get on and what works best.

So far as detecting small changes in power is concerned I think that external noise will be the limiting factor. I prefer counting meter flashes because they give a precise answer - the same one the power company gets - but of course you canā€™t separate out individual circuits as you are doing (neat idea!).
While the clip on current sensor I use is not as accurate as flash counting it does have much better resolution so at low powers I use the CT sensor value and at higher powers I count the flashes and just use the CT and voltage measurement to detect whether I am importing or exporting. The resolution of the sensor with a 12 bit ADC should be just a few watts but I get around a 30 watt variation between spot readings taken at 15 second intervals. You shouldnā€™t get anything like this variation if you are just looking at your lighting circuit. I have fridge, freezer, air circulator and solenoid valves all confusing the measurements.
Flash counting over 15 seconds can only resolve to 120 watts (if my sums are right) which is why I put in the code that averages the value if there is only a one flash per 15 secs change. While the flash counting resolution is poor, the values for the total and average power consumed will agree with the power company measurements.

I used a slightly different approach with the original Arduino (pre CT sensor) version of this software. I had the flash interrupt function on the Arduino update the average power every time the meter flashed until it was reset by being polled. This gave nice smooth values that were easy to correlate to determine whether I was importing or exporting.

You could do something similar with the CT values - instead of taking a spot reading when polled you could take readings every second or two and report the average when polled. In fact I think that is the better way to do it. Have you taken a look at what the Open Energy Monitor guys do? They have a large community and have given a lot of thought to the best way to measure things. I see that the later versions of their emonlib cater for 12bit ADCs so Iā€™m probably reinventing the wheel.

I think your two core solution sounds excellent. If I hadnā€™t already built the Arduino display thatā€™s the route Iā€™d follow. Iā€™d consider having the server provide a UDP data stream to a PC or Pi too for logging.

16

Im still trying to get my head around the way they do it on the open energy setupsā€¦ i think i will go with the PLL method and see how i go. i only have single phase coming into the house and 3 circuits covers everything. power point circuit, light circuit, oven and A/C. the way the meterbox is wired makes it even easier, i can use a high res CT for the light circuit and a medium res for the power circuit and a low res for the A/C and oven. 3 different burden resistor values! That way i can get an idea of where iā€™m using the most power too.

I came up with an idea the other day that may tie in well with this projectā€¦ i have an old ups that has a blown charging circuit but the inverter part works fine, and a very large battery bank that was donated by my dad. i may get a few panels for the roof and charge the battery bank with them, and have the core monitor the battery voltage and control a power point the UPS is plugged into, when the battery charge goes over 75% - switch off the power point so the inverter powers everything, the batteries will be getting charged from the solar still but discharged from the inverter. if the batteries drop to 20% (ie at night) then switch the power point back on, the mains power will pass straight through the UPS seamlessly. The 20% cutoffon the batteries could be kept in reserve for those times when the power does go out.

1 Like
17

@Hootie81 - Given the clean 3*3 supply pin on the Spark from which to generate the 1.65v reference I think you have the better solution with a low pass filter to adjust an dc voltage offset rather than the high pass filter to bring the mean down to zero that I used.
Looking at my ADC values the offset from the expected 2048 mid range value in a raw 1v ac signal is only 5 so the low pass filter wonā€™t have much work to do. What is more you wonā€™t get the phase shift problems that the high pass approach gives when the time constant is too low.
Nevertheless, unless you slow down the Spark sampling rate you may need to change the time constant quite substantially from the Energy Monitor project value since the Spark makes many more measurements per mains cycle than an Arduino. I changed the .996 to .9996 which reduced the phase shift and got the power factor with a pure resistive load (my oven) back to 1.0. I guess that with the high pass filter youā€™d need to go from .000122 to .00001 or even lower. As you note, the 14bit ADC means that the rounding error correction needs to be 1(left shift)14 rather than 1(left shift)12 and all references to 512 become 2048 (half the ADC range).
Please post how you get on. Itā€™s quite likely that Iā€™ll jump to your approach.

18

I have been running the Spark Energy Monitor for a couple of days now and it seems to be behaving itself.

Measurement speed

I have just checked how many measurements it makes each 50Hz mains cycle - the Spark makes 182 measurements of each pair of V and I compared with the Arduinoā€™s 53. That is just as expected. It corresponds with 110uS per measurement loop and when I tested the current Spark DMA/ADC with two channels it took 90uS per pair of measurements leaving 20uS for the fairly simple high pass filter sums.

High Pass Filter Constant

When I was using the original 1_5 (8uS) ADC conversion speed I found that I needed a filter constant of 0.9996 rather than the Arduino 0.996 to avoid introducing a large lag in the signal. This can now be reduced much closer to the Arduino value - say to 0.998 which seems the best compromise between settling reasonably quickly and not introducing a large phase shift. The Open Energy Monitor guys donā€™t seem concerned about this phase shift - maybe it is the same for V and I so doesnā€™t matter.

Phase Calibration

Iā€™ve fiddled with the PHASECAL value in the call to emon1.voltage(). It has little effect on the reported power factor which is 0.98 for a predominantly resistive load. I suspect this is because the Spark samples are measured every 2 degrees vs the Arduinos 7 degrees so they donā€™t span a wide enough phase range to correct for any difference between the current clamp and the voltage source. The calibration value for the Arduino example is 1.7 which represents about 3 degrees additional correction beyond the 2 degree sampling lag and outside the range I can adjust. The simplest fix would be to introduce a 250uS delay in the sampling loop to increase the separation of the samples close to the Arduino value. In this case the delay of 45uS between measuring V and I would need a PHASECAL value of 1.1 to correct for the 0.8 degree lag and an additional 0.4 to match the Arduio example value. I donā€™t plan to do this anytime soon as the important thing is to detect the power factor changing sign and this seems to be pretty accurate.

Next Steps

Apart from waiting for something to go wrong, Iā€™ll see whether I can make the display a little more friendly by cutting out as much clutter as possible. While the Open Energy Monitor display is great my 20x4 lcd is already screwed to the wall and shows all I need. Ditto the graphic for a web page - I can see all I want at a glance. But neither of those is a Spark project. As far as the Spark is concerned Iā€™ll implement the low pass filter if @Hootie81 finds it works better than the high pass filter. I had been thinking about a Tinker like phone app but I can already see the web page so that will go on hold. Otherwise Iā€™m waiting for delivery of two more Sparks - one for a beehive monitor and the other to play with. Iā€™m afraid that Iā€™ve cooked my current spare.

19

Spark Energy Monitor Stripboard

Here is the stripboard layout I used. Hand drawn in PowerPoint so no guarantees Iā€™m afraid.

20

All Code now on Gist

I set up the Gist in a rush before I went away and see that I missed posting the Spark Core energy monitoring code which is the most interesting bit.
It is all now up there together with a minor modification to allow access to another emon variable - the number of samples collected per call to emon1.calcVI() and an additional try/except check in the python code to trap stray non ascii characters in the UCF stream.
https://gist.github.com/phec/9254793