Webhook to move data from Particle to another source (Thingspeak at this time, probably moving to Azure)
I require higher-resolution data measurement than most Electron projects that I’ve read on here so I haven’t been able to locate any similar projects on this forum.
Problem
I have tried to measure AND push data at the same time (once per 1 minute, once every 2 minute), but the battery level is reducing too fast, especially on cloudy days. I’m assuming this is because the Cell modem is on and drawing a lot of current from the battery. Also, each time I push data to the cloud, that is more current draw.
I’ve also tried the following scenario: Measure & Publish Data > Sleep Device for 1min > Wake Device (rinse repeat)
This seems to draw just as much current from the battery and it also misses a publish some times because I have to reconnect to the cell tower and Particle cloud.
Solution?
My idea is to.
Turn off the cell modem.
Measure sensors and collect data once per minute, store data & timestamp (or time elapsed) in an array.
Once 20 minutes has elapsed, turn the cell modem on, connect to Particle cloud, and push the data.
(rinse, repeat)
All sounds good in theory, but is this feasible to send a large chunk of data that includes timestamps and/or time-elapsed between measurements?
Is there any Webhook or post application that can parse the data correctly so that the timestamps are correct?
@ScruffR I guess that’s what I meant by time elapsed. If I have the ONE timestamp of the major Publish event, then I can assign each of the data points a timestamp AFTER they are published to the cloud (based off the fact I know the interval between measurements in fixed). Is that what you mean by a fix raster array? I didn’t know that such a technique had a name.
Based off the Particle console, a single data set has this string:
So with the new code, major publish events (which occur every 20 minutes) have 20 measurements in them. That might increase the total data publish event to be…1700-1800 bytes?
I understand how to send this data to the Cloud and all of that. I guess my question is more aimed towards the fix raster array - can that be done in Webhooks or any other Particle tool?
If you reduce the data overhead by not using JSON but a standardized data frame with one char delimiters and only send significant decimal places you might be able to pack that in one or two Particle.variable()s or if you don’t need a secure transport send binary data via UDP/TCP.
Hmm, I never thought to use Particle variables and have never used them before, but supposed I should give it a shot. I do need secure transport so forget about sending binary data over UDP/TCP.
Once I have the Particle variable data, where do I perform the fix raster array? I don’t see anything specifically in the Console > Integrations. I suppose some third party can do it once the data is in their cloud - Google Cloud IOT or Azure.
@fishmastaflex How much battery SOC are you losing per day?
How large of a solar panel do you have attached to the Electron?
I’m running off solar power and have been burning up about 25% of the 2000 mAh battery capacity from sun down to up when publishing to Ubidots every 2 mins. I put the Electron into deep sleep once the batery SOC hits 20%. Then it wakes up every hour to see if the solar panel has charged the battery up past 20% then it starts sending data every 2 mins again.
@RWB Mine are about 2W as well, though I am using a different solar panel than you are. I’m actually using solar cells, which have a very low Voc of about 0.6V and Isc of 9A, which have to be boosted up to 5V using a chip. The cells are the core of my project, so unfortunately they cannot be swapped out with higher voltage ones. I still need to tweak the boost circuit a bit for maximum efficiency, but as I explain below about battery loss without sun, it’s concerning.
Here is an export of the SOC level for today (which rained and didn’t get much sunshine). From about 11am - 4pm I had some sunshine, but it was very cloudy today. When there is no sun and the Electron is just pushing data @ 5min intervals, the battery loss is about 3.6% per hour. So after half a day, the loss is almost 50%.
I would say that my target is about 20% SOC loss per day (if there was no sun) so that the Electron could last 4 days without charging the battery.
Looking at your graphs (which I love the look of!) on Friday 06 you ran outta juice and the Electron was almost oscillating in and out of low battery states
Try pushing your data to Ubidots using the same code I’m using below and see if you get better performance. You should since your only publishing at half the rate I am.
Your probably just going to need more solar if possible and a bigger battery to get longer run times.
If you can go with a 5v solar panel so you don’t loose the power in the DC to DC boost conversion to charge the battery at 4v.
SYSTEM_MODE(SEMI_AUTOMATIC);
//SYSTEM_THREAD(ENABLED);
// This #include statement was automatically added by the Particle IDE.
#include "Ubidots/Ubidots.h"
#define TOKEN "Your-Token" // Put here your Ubidots TOKEN
#define DATA_SOURCE_NAME "ElectronSleepNew"
SerialLogHandler logHandler(LOG_LEVEL_ALL); //This serial prints system process via USB incase you need to debug any problems you may be having with the system.
Ubidots ubidots(TOKEN); // A data source with particle name will be created in your Ubidots account
int button = D0; // Connect a Button to Pin D0 to Wake the Electron when in System Sleep mode.
int ledPin = D7; // LED connected to D1
int sleepInterval = 60; // This is used below for sleep times and is equal to 60 seconds of time.
void setup() {
//Serial.begin(115200);
pinMode(button, INPUT_PULLDOWN); // Sets pin as input
pinMode(ledPin, OUTPUT); // Sets pin as output
ubidots.setDatasourceName(DATA_SOURCE_NAME); //This name will automatically show up in Ubidots the first time you post data.
PMIC pmic; //Initalize the PMIC class so you can call the Power Management functions below.
pmic.setChargeCurrent(0,0,1,0,0,0); //Set charging current to 1024mA (512 + 512 offset)
pmic.setInputVoltageLimit(4840); //Set the lowest input voltage to 4.84 volts. This keeps my 5v solar panel from operating below 4.84 volts.
}
void loop() {
FuelGauge fuel; // Initalize the Fuel Gauge so we can call the fuel gauge functions below.
if(fuel.getSoC() > 20) // If the battery SOC is above 20% then we will turn on the modem and then send the sensor data.
{
float value1 = fuel.getVCell();
float value2 = fuel.getSoC();
ubidots.add("Volts", value1); // Change for your variable name
ubidots.add("SOC", value2);
Cellular.connect(); // This command turns on the Cellular Modem and tells it to connect to the cellular network.
if (!waitFor(Cellular.ready, 600000)) { //If the cellular modem does not successfuly connect to the cellular network in 10 mins then go back to sleep via the sleep command below. After 5 mins of not successfuly connecting the modem will reset.
System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); //Put the Electron into Sleep Mode for 2 Mins + leave the Modem in Sleep Standby mode so when you wake up the modem is ready to send data vs a full reconnection process.
}
ubidots.sendAll(); // Send fuel gauge data to your Ubidots account.
digitalWrite(ledPin, HIGH); // Sets the LED on
delay(250); // waits for a second
digitalWrite(ledPin, LOW); // Sets the LED off
delay(250); // waits for a second
digitalWrite(ledPin, HIGH); // Sets the LED on
delay(250); // waits for a second
digitalWrite(ledPin, LOW); // Sets the LED off
System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); //Put the Electron into Sleep Mode for 2 Mins + leave the Modem in Sleep Standby mode so when you wake up the modem is ready to send data vs a full reconnection process.
}
else //If the battery SOC is below 20% then we will flash the LED 4 times so we know. Then put the device into deep sleep for 1 hour and check SOC again.
{
//The 6 lines of code below are needed to turn off the Modem before sleeping if your using SYSTEM_THREAD(ENABLED); with the current 0.6.0 firmware. It's a AT Command problem currently.
//Cellular.on();
//delay(10000);
//Cellular.command("AT+CPWROFF\r\n");
//delay(2000);
//FuelGauge().sleep();
//delay(2000);
digitalWrite(ledPin, HIGH); // Sets the LED on
delay(150); // Waits for a second
digitalWrite(ledPin, LOW); // Sets the LED off
delay(150); // Waits for a second
digitalWrite(ledPin, HIGH); // Sets the LED on
delay(150); // Waits for a second
digitalWrite(ledPin, LOW); // Sets the LED off
delay(150); // Waits for a second
digitalWrite(ledPin, HIGH); // Sets the LED on
delay(150); // Waits for a second
digitalWrite(ledPin, LOW); // Sets the LED off
delay(150); // Waits for a second
digitalWrite(ledPin, HIGH); // Sets the LED on
delay(150); // Waits for a second
digitalWrite(ledPin, LOW); // Sets the LED off
System.sleep(SLEEP_MODE_DEEP, 3600); //Put the Electron into Deep Sleep for 1 Hour.
}
}
Thanks for the response @RWB. The solar cells are the core of my project and they can’t be changed because they measure solar irradiance with precision. Going to another panel with higher voltage, I totally lose that precision!
From your graphs, doesn’t it concern you that you are losing 2-3% of battery per hour? I tried to calculate your battery loss per hour from your graphs (visually guessing the SOC levels) and it seems like it’s about 2-3%. If you didn’t have any sun for even 2 days, your Electron would be dead and going into the low battery oscillations like you had on Friday 06. This is what I don’t want.
That being said, I think I still want to measure data with the cell modem turned off, store the values in a string, and send a big chunk of it every 20 minutes to the Particle cloud. This way I can power save a lot more through the day. If this practice doubles battery life, then that’s awesome!
I just don’t know of the best integration platform that will parse the rather large string to an IoT service with graphing capabilities. I’m going to create an account in Azure to play around. I’ll have to update everyone and let them know if this helps battery life!
@fishmastaflex What are you building exactly? I’m in the solar business so maybe I would be interested in
What type of solar cells are you using? I’m using Sunpower cells.
No, I’m not concerned with power usage levels because all I’m doing now is making sure the Electron is reliable and figuring out what the power consumption baseline is so I can size the solar panel and battery to meet the runtimes I’m looking for.
The Photon and Electron will end up in portable systems with large 12v batteries so there will be plenty of battery to keep them online daily.
Azure IOT Hub and Microsoft Power BI allows for the Ultimate custom dashboard experience but it kinda has a steep learning curve. Ubidots has the easiest to setup, use, and share interface that I have come across so far.
Azure IOT Hub allows you to send 512kb messages and it’s considered as 1 message but I’m not sure if the Particle Webhooks can send a single 512kb message to Azure IOT Hub or not.
A little update here…I was able to find some time to implement the 1min data / 20min cloud push code you gentlemen so kindly helped me with. A new firmware has since released so I had some additional questions that browsing through other threads didn’t answer exactly.
Because i need the MCU to run while taking measurements every minute, I am using:
// Turn off cellular for seconds.
// Keep application running.
System.sleep(200);
Then after 20 minutes I initiate a cloud push.
Particle.publish("dataChunk",dataChunk,PRIVATE);
Except I think this is using up a lot of data because of the handshake that occurs when turning on the cell modem back again.
I guess my question is, how do I put the cell modem into a STANDBY state (not an OFF state) for those 20 minutes so that no handshake is needed?
I can’t use SLEEP_NETWORK_STANDBY because that pauses the microcontroller. I am using SEMI_AUTOMATIC mode and SYSTEM THREAD ENABLED.
I thought I read somewhere that the code was optimized in the latest version.
