Electron data usage problem redux

BDub · December 21, 2016, 4:20am

Hi @bioagbob I replied over here about the data usage question:
Monthly Electron Data Usage

RWB · December 21, 2016, 5:46am

@bioagbob Using the new 0.6.1-rc firmware try this new improved code out. I’m seeing improvements on my end

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED); 

int button = D0; 
int ledPin = D7;              // LED connected to D1
int sleepInterval = 60;

    
void setup()
{
  //Serial.begin(115200);
 pinMode(button, INPUT_PULLDOWN);    // sets pin as input
 pinMode(ledPin, OUTPUT);    // sets pin as output
 
}
 
void loop()
{
    FuelGauge fuel;
    float value;
    bool success;

if(fuel.getSoC() > 20)
  {

  Particle.connect();
  
  if (!waitFor(Particle.connected, 60000)) {  //If we do not connect to the Particle Cloud in 5 mins then go back to sleep and try again. 
      
    System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY);
}
  
  value = fuel.getVCell();
  String output = "{\"batt-value\": \"" + String(value) + "\"}";
  success = Particle.publish("fuel-level1", output);


  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(250);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(250);                  // waits for a second
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(250);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  
  System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); 
  }
  else
  {
  //Put Cellullar Modem into Deep Sleep until battery charged above 20%   
  Cellular.on();
  delay(10000);
  Cellular.command("AT+CPWROFF\r\n");
  delay(2000);
  //FuelGauge().sleep();
  //delay(2000);
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(150);                  // waits for a second
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(150);
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(150);                  // waits for a second
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  System.sleep(SLEEP_MODE_DEEP, 3600); //Wake Up Every Hour to check if battery is over 20% yet. Can do this for 20 Days with no charge input before depleting battery. 
  }   
}

ScruffR · December 21, 2016, 6:00am

Thanks @BDub, that was the thread I was looking for but couldn’t find anymore
Electron strange behavior with sleeping

bioagbob · December 21, 2016, 6:57am

You’re too quick @RWB! I had other things to attend to, but I’m back and just re-tested my original code with and without the extra 5 second delay. (I’ll get to the new firmware soon).

Results with my original code and 0.6.0-rc.2:

because I can’t find a facepalm or head-bangingin-into-wall gif. I’ve been missing part of the data when looking at getDataUsage(). I had been resetting the counter early in code and therefore missed the usage associate with network connection. Duh. When looking at all usage after board reset, it adds up! Thanks to Hologram MVNO session usage stats, I can confirm that getDataUsage() matched the carrier usage. It’s about the same as Particle. Every publish was burning 953 bytes of data. Good news overall, but I’m still sorry I didn’t have that straight.
In previous tests, I did in fact use much less data. It’s still not clear what changed between then and now, but I’m done trying to solve that puzzle.
If I add the 5 second delay to my original code, the first connection uses 941 bytes and subsequent connections use only 288 bytes. My data string is about 85 bytes of that. 288 bytes @ 10 minute interval is about 1.24 MB per month, which is on par with my previous results (before this whole mess). That’s still more than @RWB claimed here Best Methods for Conserving Electron Current and Data Usage, but still much better.

Kudos to @RWB and @ScruffR for helping me figure this out!

RWB · December 22, 2016, 3:49pm

@bioagbob I measured my data usage for sending voltage and SOC reading to Ubidots every 2 mins using the latest code and 0.6.1-RC1 firmware and it works out to 0.004 MB per hour, or 0.096 per 24 hours.

How does that compare with yours?

bioagbob · December 22, 2016, 5:05pm

@RWB Are you measuring data use with getDataUsage() or does Ubidots provide something? I’m using getDataUsage(). Just checking. I can provide complete code if that helps compare.

I ran 0.6.0-rc.2 with 5sec delay and my original code again to confirm everything. It pushes {"key":"9d45cc63bba86499b860230","body":"HOLOSPARK58&3&M0:3.937700"} to the Particle cloud and then uses a webhook to a private server. First couple connections used 4916, 2002, and 2009 bytes (I push more data in those cycles) and then it used 289 bytes thereafter. I’m publishing every 10 minutes (Note that’s 10 minute sleep + delays, so it actually pushed only 67 packets in 12 hours). Over 24 hours, that’s about 0.047 MB. At a 2 minute rate that would be roughly 0.20 MB.

I may have longer packets. My payload is 68 bytes, plus publish name of 7 bytes = 75 bytes per packet. That means my overhead per publish is approx 289 - 75 = 214 bytes.

I’m just started a test with 0.6.1-rc.1. I removed the 5 sec delay after publish but use a 500 ms delay (as I did when it was not working with 0.6.0) to flush the serial buffer. First 3 cycles used 5062, 2157, 2157 bytes. Next cycle used 1099 bytes. Next few used 288 bytes each, so it appears similar. Test will run for the day while I’m away.

I can run your test code for comparison if it’s helpful for comparison between us. Did you ever finish your ‘crude’ test from a couple days ago to compare Ubidots to Particle?

RWB · December 22, 2016, 5:38pm

I’m just using the Console data usage report reading as my source for how much data I’m consuming.

Ubidots has a lower data transfer option for the Electron that uses UDP but I have not tried that yet so I have no idea how much data that saves over the current method.

I stopped the first crude test because I updated to the new firmware and made some code changes so started over.

Share your code for measuring data usage and I’ll test it using Ubidots MQTT vs UDP transfer methods.

It looks like Ubidots MQTT method is more efficient than your Particle Publish method so it may be a good reason to switch over to Ubidots, especially if the UDP method saves even more data.

bioagbob · December 22, 2016, 10:55pm

Forgive me. This code is not cleaned up for public consumption (except for bogus api keys), but I’m not going to spend the time right now. Lots to do better, but I’m shooting for function, not pretty right now.

//Based on PUBLISHBATT_HOLO_SERIALUSAGE.INO
//rev1 10/28/16: Adds .ready(), .publish return, localIP, RSSI, and QUAL
//rev2 10/29/16: Streamline output to use csv format (all data on single line)
//rev3 11/1/16: Updated firmware to 0.6.0-rc2 during rev2.  Added delay after Publish to allow UDP flush? 
//rev4 11/14/16: Reduce delays to reduce batt consumption.  Push API key for .stream HoloSpark58.
//rev4.0.1 12/16/16: Add \" before body value (syntax error in json packet)
//rev4.0.2 12/16/16: Must use M0, not bat in streams{}
// 10 min, NO_ACK, PARTICLE SIM
//rev4.0.3 12/20/16: Testing with 5 sec delay before sleep (supposed fix in forthcoming firmware 0.6.1-rc.1)
//rev4.0.4 12/20/16: Change getDataUsage tracking to include initial usage after init (connection overhead). Print total usage.
//rev4.0.5 12/20/16: Add additional case to send metadata packet after init.  Added print of metadata.
//rev4.0.6 12/21/16: Removed delay after publish and updated firmware to 0.6.1-rc.1
//
// v4.0.4 <,Timestamp,Init_CID,T,R,T,R,.ready,localIP,Reset_CID,T,R,T,R,batt_value,pubSuccess,RSSI,QUAL,Publish_CID,T,R,T,R,deltaT,deltaR,totalusage>

//KNOWN BUGS:
// Time() might not sync properly and gives a 1969 date.  Need to check and Particle.syncTime().  May also need to sync for drift.
// Metadata prints out in middle of csv string

//#include "cellular_hal.h"
//STARTUP(cellular_credentials_set("apn.konekt.io", "", "", NULL));
STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));

String DeviceKey = String("9d05cc635ba90494a8d023a");
String DeviceID = String("HOLOSPARK58");
String DeviceName = String("HS58");
retained int PktCount = 0;


void setup()
{
    bool readyOK;
    bool success;
    Serial.begin(9600);
    delay(1000);    //delay to allow user serial connection
    //Serial.print("******** Init **********");
    Serial.print("<,");
    Serial.print(Time.format(Time.now(), TIME_FORMAT_ISO8601_FULL));
    Serial.print(",");
    PrintCellUsage();   //values on init
    Serial.print(",");
    //Cellular.resetDataUsage();  //can't reset or we lose info about tower connection costs
    //Serial.print("Reset: ");
    //PrintCellUsage();   //check that reset
    //Serial.print(",");
    if(Cellular.ready())
        Serial.print("true");
    else
        Serial.print("false");
    //Serial.print("localIP = ");
    Serial.print(",");
    Serial.print(Cellular.localIP());
    //CellularSignal sig = Cellular.RSSI();
    //Serial.print("RSSI,QUAL = ");
    //Serial.println(sig);  //printed later
    
    //CONFIG: 40c06a19&nn:N1346.2_TD&pr:M0;xn:INT_VLT;nx:2;#xm:BattVolts;mty:volts;unm:volts;#xm:SolrVolts;mty:volts;unm:volts;
    //one packet per port
   if (PktCount%200 == 0 || PktCount < 3) {
        String output = "{\"key\":\"" + DeviceKey + "\",\"body\":\"";
        output.concat(DeviceID + "&");
        output.concat("nn:" + String(DeviceName) + "&");
        output.concat("pr:" + String("M0") + ";");
        output.concat("xn:" + String("DEV") + ";");
        output.concat("nx:" + String("1") + ";");
        output.concat("#xm:" + String("BattV") + ";");
        output.concat("mty:" + String("V") + ";");
        output.concat("unm:" + String("V") + ";");
        output.concat("\"}");
        success = Particle.publish("metadata", output);
        Serial.println(output);		//Note prints out in middle of csv string
        delay(5000);
   }
} //end setup


void loop()
{
    FuelGauge fuel;
    float value;
    bool success;
    int tx1, rx1, tx2, rx2, deltatx, deltarx, totalusage;
    
    //Serial.print("Before Publish: ");
    Serial.print(",");
    //Cannot call function becuase need 'data' in this scope for delta calculation
    CellularData data;
    if (!Cellular.getDataUsage(data)) {
        Serial.print("-1,-1,-1,-1,-1");
    }
    else {
        Serial.print(data); // printed as CID,TX,RX,TX,RX
    }
             
    tx1 = data.tx_total;  //v4.0.4 - usage after connection to tower
    rx1 = data.rx_total;  //v4.0.4 - usage after connection to tower
                
    value = fuel.getVCell();
    
    if (PktCount > 999)
        PktCount = 0;
    else
        PktCount++;
    
    //String output = "{\"key\":\"9d05cc635ba90494a8d023a\",\"body\":\"bat\":\"" + String(value) + "\"}";
    //JSON DATA:   40c06a19&0&M0:4.168:1.7&M1:2.203:2.199
    String output = "{\"key\":\"" + DeviceKey + "\",\"body\":\"";
    output.concat(DeviceID + "&");
    output.concat(String(PktCount) + "&");
    output.concat("M0:" + String(value) + "\"}");
    
    Serial.print(",");
    Serial.print(value,4);
    success = Particle.publish("streams", output, NO_ACK);
    //Serial.print("publish success = ");
    Serial.print(",");
    Serial.print(success);
    CellularSignal sig = Cellular.RSSI();
    // Serial.print("RSSI,QUAL = ");
    Serial.print(",");
    Serial.print(sig);
    //delay(1);
    //delay(5000);    //flush serial buffer (and UDP buffer?)
    //Serial.print("delay - publish success = ");
    //Serial.println(success);
    //Serial.print("After Publish: ");
    Serial.print(",");
    if (!Cellular.getDataUsage(data)) {
        Serial.print("-1,-1,-1,-1,-1");
    }
    else {
        Serial.print(data); // data usage counter after publish
    }
             
    tx2 = data.tx_total;  //v4.0.4 - usage after publish
    rx2 = data.rx_total;  //v4.0.4 - usage after publish
    
    deltatx = tx2 - tx1;  //v4.0.4 - usage from publish, not including connection
    deltarx = rx2 - rx1;  //v4.0.4 - usage from publish, not including connection
    //Serial.print("Delta: ");
    Serial.print(",");
    Serial.print(deltatx);
    Serial.print(",");
    Serial.print(deltarx);
    totalusage = tx2 + rx2;
    Serial.print(",");
    Serial.print(totalusage);
    Serial.println(",>");    //END OF CSV LINE
    Serial.println(output);
    // delay(5000);    //allow serial flush before sleep and rev4.0.3 delay before sleep
    delay(500);    //allow serial flush before sleep
     
    //NOTE: This bricks the device for OTA... 
    System.sleep(SLEEP_MODE_DEEP, 600);    //1200 = 20 min
    
} //end loop

void PrintCellUsage() {
            CellularData data;
            if (!Cellular.getDataUsage(data)) {
                Serial.print("-1,-1,-1,-1,-1");
            }
            else {
                Serial.print(data); // printed as CID,TX,RX,TX,RX
            }
}

Serial output with 0.6.1-rc.1 gives this. Note that last value before “>” is the total Tx and Rx usage as reported by getDataUsage().

<,1999-12-01T00:00:32Z,31,1924,1940,1924,1940,true,10.40.224.162{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&nn:HS58&pr:M0;xn:DEV;nx:1;#xm:BattV;mty:V;unm:V;"}
,31,2855,2067,2855,2067,3.9413,1,-77,37,31,2995,2067,2995,2067,140,0,5062,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&1&M0:3.941300"}
<,2016-12-22T16:08:08Z,31,521,438,521,438,true,10.42.27.154{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&nn:HS58&pr:M0;xn:DEV;nx:1;#xm:BattV;mty:V;unm:V;"}
,31,1452,565,1452,565,3.9377,1,-77,37,31,1592,565,1592,565,140,0,2157,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&2&M0:3.937700"}
<,2016-12-22T16:18:49Z,31,521,438,521,438,true,10.40.224.261{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&nn:HS58&pr:M0;xn:DEV;nx:1;#xm:BattV;mty:V;unm:V;"}
,31,1452,565,1452,565,3.9377,1,-79,37,31,1592,565,1592,565,140,0,2157,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&3&M0:3.937700"}
<,2016-12-22T16:29:31Z,31,521,438,521,438,true,10.42.27.142,31,321,438,521,438,3.9377,1,-77,37,31,661,438,661,438,140,0,1099,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&4&M0:3.937700"}
<,2016-12-22T16:40:09Z,31,74,61,74,61,true,10.40.224.151,31,74,61,74,61,3.9377,1,-79,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&5&M0:3.937700"}
<,2016-12-22T16:50:43Z,31,74,61,74,61,true,10.40.224.151,31,74,61,74,61,3.9377,1,-79,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&6&M0:3.937700"}
<,2016-12-22T17:01:18Z,31,74,61,74,61,true,10.40.224.151,31,74,61,74,61,3.9377,1,-79,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&7&M0:3.937700"}
<,2016-12-22T17:11:52Z,31,74,61,74,61,true,10.40.3.134,31,74,61,74,61,3.9377,1,-75,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&8&M0:3.937700"}
<,2016-12-22T17:22:27Z,31,74,61,74,61,true,10.40.3.134,31,74,61,74,61,3.9377,1,-75,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&9&M0:3.937700"}
<,2016-12-22T17:33:02Z,31,74,61,74,61,true,10.40.3.134,31,74,61,74,61,3.9377,1,-75,37,31,228,61,228,61,154,0,289,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&10&M0:3.937700"}

Note that (for me anyway) the serial port gets closed on sleep and does not automatically reconnect on my laptop (Macbook). I wrote a python script to brute force a workaround. It just polls the port until it opens when Electron wakes up each cycle. Now I can leave it running without manually have to reconnect the serial port.

import serial
import time
#electron dumps about 250 char.  .ser(500) should timeout

while True:
    try:
        serialport = serial.Serial("/dev/cu.usbmodem1411", 9600, timeout=5)  #electron takes about 20 sec in wake
        #print "Port open"
        output = serialport.read(500)
        if output:
            print output,
        serialport.close()
        #print "Port closed"
    except:
        #print "Error port cannot be opened"
        time.sleep(2)   #want in except so that only delays when port not available

fishmastaflex · November 8, 2017, 9:46pm

If you use TeraTerm for reading/sending commands instead, it will do the same thing as your python script - automatically reconnect when the serial port is available again.

Topic		Replies	Views
Particle Console Data Usage? General	4	423	July 4, 2019
Electron Data Usage Investigation General	34	8780	March 9, 2018
Electron data usage General	26	10201	February 25, 2016
Recent electron data usage General	0	966	May 6, 2016
Using lots of data to connect? Troubleshooting	11	1001	April 11, 2017

Electron data usage problem redux

Related topics