Electron data usage problem redux

Cool. Thanks for watching. I wish Particle had a better way to track carrier charges during testing.

Results of new code without SYSTEM_THREAD. Different pattern, but still strange and not reliable timing.

i) t = 0, fuel-level1, breathe cyan for 60 sec, sleep;
ii) t = 3 min, fuel-level1, breath cyan for 5 sec, sleep;
iii) t = 5.2 min, NOTHING in cloud, cyan on < 1 sec, sleep;
iv) t = 7.2 min, Online AND fuel-level1 packets, cyan blinks 3 sec, rapid flash 5 sec, sleep;
v) repeat cycle 2 min later.


That disconnect does remind me of this open issue

And another one I reported in conjunction with publish and sleep but can’t find right now :blush:

Hmm… except I’m publishing every 2 minutes. It seems that since yours worked at 1.1 sec, I should be okay at 2 min. :smile: I guess I can try a longer interval anyway. I can rule nothing out at this point.

Yes, but the exact same behaviour with a full connect handshake after a fix count of wakes was seen by another member and I was able to confirm that and back then I reported the similarity between that and the fix count of “rapid” publishes issue to Particle.
As far as I remember adding a delay(5000) between the publish and sleep instructions did help getting around the issue.
Could you try that?

The minimal wait with immediate sleep may be just that tad too quick for a reliable reconnect on next wake as the expected 1000ms was that bit too short.

I’ll try adding the delay after I wait for my console to reach steady state (Electron turned off).

I ran RWB’s code (part with and part without threads) for about 3 hours over which there were about 70 fuel-level or ‘online’ connections. The console started at 0.28 MB (had not run all night before this test) and is now at 0.40 MB (about 2 hours after Electron turned off). So, 0.12 MB is about 1.7 kB per connection. :dizzy_face:

@bioagbob Try that for sure.

When I was Particle Publishing and going to sleep I was seeing the Electron flash Green or Cyan for up to 60-90 seconds before it would actually go to sleep for some reason. Never tried the 5-second delay before calling sleep I don’t think. It would be nice if it fixed the issue though.

Testing now with 5 sec delay before sleep. I may have done something similar way back, but can’t recall now since things had been working okay (My most recent code did have a 1 sec delay to allow the serial port to flush before sleep, so perhaps that helped - but doesn’t explain why I started seeing problems now).

Console billing was stable at 0.40 MB at start of test. Updated code (with delay 5) took about 6 minutes before it reached a repeatable pattern of publishes. First 6 minutes: ONLINE, pin reset, update, fuel: breathing cyan for 60 sec, fuel: [missed flashes], ONLINE. Since then it’s been publishing every 2.1 minutes, each time breathing cyan for about 7-9 seconds. So, I’d say that fixed (worked around) the re-connection bug. :+1:

Will let it run to see what happens with data usage.

1 Like

Looks like you might be hitting this bug (Fixed in 0.6.1-rc.1 will be releasing soon):

1 Like

Thanks @BDub. Glad to know there’s a fix. How soon is “soon”? :wink:
And while you’re here, what are the odds of getting better tools to track carrier usage charges (per session or hour or anything timestamped!)? Monthly Electron Data Usage

2 Likes

How about right now :smile: (check the Web IDE… haven’t even finished the release post yet)

I’ll come back for your other question in a bit…

2 Likes

Wow! That’s what I call service! :stuck_out_tongue:

2 Likes

Hi @bioagbob I replied over here about the data usage question:
Monthly Electron Data Usage

@bioagbob Using the new 0.6.1-rc firmware try this new improved code out. I’m seeing improvements on my end :smiley:

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED); 

int button = D0; 
int ledPin = D7;              // LED connected to D1
int sleepInterval = 60;

    
void setup()
{
  //Serial.begin(115200);
 pinMode(button, INPUT_PULLDOWN);    // sets pin as input
 pinMode(ledPin, OUTPUT);    // sets pin as output
 
}
 
void loop()
{
    FuelGauge fuel;
    float value;
    bool success;

if(fuel.getSoC() > 20)
  {

  Particle.connect();
  
  if (!waitFor(Particle.connected, 60000)) {  //If we do not connect to the Particle Cloud in 5 mins then go back to sleep and try again. 
      
    System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY);
}
  
  value = fuel.getVCell();
  String output = "{\"batt-value\": \"" + String(value) + "\"}";
  success = Particle.publish("fuel-level1", output);


  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(250);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(250);                  // waits for a second
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(250);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  
  System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); 
  }
  else
  {
  //Put Cellullar Modem into Deep Sleep until battery charged above 20%   
  Cellular.on();
  delay(10000);
  Cellular.command("AT+CPWROFF\r\n");
  delay(2000);
  //FuelGauge().sleep();
  //delay(2000);
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(150);                  // waits for a second
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(150);
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  delay(150);                  // waits for a second
  digitalWrite(ledPin, HIGH);   // sets the LED on
  delay(150);                  // waits for a second
  digitalWrite(ledPin, LOW);    // sets the LED off
  System.sleep(SLEEP_MODE_DEEP, 3600); //Wake Up Every Hour to check if battery is over 20% yet. Can do this for 20 Days with no charge input before depleting battery. 
  }   
}

Thanks @BDub, that was the thread I was looking for but couldn’t find anymore
Electron strange behavior with sleeping

You’re too quick @RWB! I had other things to attend to, but I’m back and just re-tested my original code with and without the extra 5 second delay. (I’ll get to the new firmware soon).

Results with my original code and 0.6.0-rc.2:

  1. :scream: :hankey: because I can’t find a facepalm or head-bangingin-into-wall gif. I’ve been missing part of the data when looking at getDataUsage(). I had been resetting the counter early in code and therefore missed the usage associate with network connection. Duh. When looking at all usage after board reset, it adds up! Thanks to Hologram MVNO session usage stats, I can confirm that getDataUsage() matched the carrier usage. It’s about the same as Particle. Every publish was burning 953 bytes of data. Good news overall, but I’m still sorry I didn’t have that straight.

  2. In previous tests, I did in fact use much less data. It’s still not clear what changed between then and now, but I’m done trying to solve that puzzle.

  3. If I add the 5 second delay to my original code, the first connection uses 941 bytes and subsequent connections use only 288 bytes. My data string is about 85 bytes of that. 288 bytes @ 10 minute interval is about 1.24 MB per month, which is on par with my previous results (before this whole mess). That’s still more than @RWB claimed here Best Methods for Conserving Electron Current and Data Usage, but still much better.

Kudos to @RWB and @ScruffR for helping me figure this out! :muscle:

2 Likes

@bioagbob I measured my data usage for sending voltage and SOC reading to Ubidots every 2 mins using the latest code and 0.6.1-RC1 firmware and it works out to 0.004 MB per hour, or 0.096 per 24 hours.

How does that compare with yours?

@RWB Are you measuring data use with getDataUsage() or does Ubidots provide something? I’m using getDataUsage(). Just checking. I can provide complete code if that helps compare.

I ran 0.6.0-rc.2 with 5sec delay and my original code again to confirm everything. It pushes {"key":"9d45cc63bba86499b860230","body":"HOLOSPARK58&3&M0:3.937700"} to the Particle cloud and then uses a webhook to a private server. First couple connections used 4916, 2002, and 2009 bytes (I push more data in those cycles) and then it used 289 bytes thereafter. I’m publishing every 10 minutes (Note that’s 10 minute sleep + delays, so it actually pushed only 67 packets in 12 hours). Over 24 hours, that’s about 0.047 MB. At a 2 minute rate that would be roughly 0.20 MB.

I may have longer packets. My payload is 68 bytes, plus publish name of 7 bytes = 75 bytes per packet. That means my overhead per publish is approx 289 - 75 = 214 bytes.

I’m just started a test with 0.6.1-rc.1. I removed the 5 sec delay after publish but use a 500 ms delay (as I did when it was not working with 0.6.0) to flush the serial buffer. First 3 cycles used 5062, 2157, 2157 bytes. Next cycle used 1099 bytes. Next few used 288 bytes each, so it appears similar. Test will run for the day while I’m away.

I can run your test code for comparison if it’s helpful for comparison between us. Did you ever finish your ‘crude’ test from a couple days ago to compare Ubidots to Particle?

I’m just using the Console data usage report reading as my source for how much data I’m consuming.

Ubidots has a lower data transfer option for the Electron that uses UDP but I have not tried that yet so I have no idea how much data that saves over the current method.

I stopped the first crude test because I updated to the new firmware and made some code changes so started over.

Share your code for measuring data usage and I’ll test it using Ubidots MQTT vs UDP transfer methods.

It looks like Ubidots MQTT method is more efficient than your Particle Publish method so it may be a good reason to switch over to Ubidots, especially if the UDP method saves even more data.

Forgive me. This code is not cleaned up for public consumption (except for bogus api keys), but I’m not going to spend the time right now. Lots to do better, but I’m shooting for function, not pretty right now. :grin:

//Based on PUBLISHBATT_HOLO_SERIALUSAGE.INO
//rev1 10/28/16: Adds .ready(), .publish return, localIP, RSSI, and QUAL
//rev2 10/29/16: Streamline output to use csv format (all data on single line)
//rev3 11/1/16: Updated firmware to 0.6.0-rc2 during rev2.  Added delay after Publish to allow UDP flush? 
//rev4 11/14/16: Reduce delays to reduce batt consumption.  Push API key for .stream HoloSpark58.
//rev4.0.1 12/16/16: Add \" before body value (syntax error in json packet)
//rev4.0.2 12/16/16: Must use M0, not bat in streams{}
// 10 min, NO_ACK, PARTICLE SIM
//rev4.0.3 12/20/16: Testing with 5 sec delay before sleep (supposed fix in forthcoming firmware 0.6.1-rc.1)
//rev4.0.4 12/20/16: Change getDataUsage tracking to include initial usage after init (connection overhead). Print total usage.
//rev4.0.5 12/20/16: Add additional case to send metadata packet after init.  Added print of metadata.
//rev4.0.6 12/21/16: Removed delay after publish and updated firmware to 0.6.1-rc.1
//
// v4.0.4 <,Timestamp,Init_CID,T,R,T,R,.ready,localIP,Reset_CID,T,R,T,R,batt_value,pubSuccess,RSSI,QUAL,Publish_CID,T,R,T,R,deltaT,deltaR,totalusage>

//KNOWN BUGS:
// Time() might not sync properly and gives a 1969 date.  Need to check and Particle.syncTime().  May also need to sync for drift.
// Metadata prints out in middle of csv string

//#include "cellular_hal.h"
//STARTUP(cellular_credentials_set("apn.konekt.io", "", "", NULL));
STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));

String DeviceKey = String("9d05cc635ba90494a8d023a");
String DeviceID = String("HOLOSPARK58");
String DeviceName = String("HS58");
retained int PktCount = 0;


void setup()
{
    bool readyOK;
    bool success;
    Serial.begin(9600);
    delay(1000);    //delay to allow user serial connection
    //Serial.print("******** Init **********");
    Serial.print("<,");
    Serial.print(Time.format(Time.now(), TIME_FORMAT_ISO8601_FULL));
    Serial.print(",");
    PrintCellUsage();   //values on init
    Serial.print(",");
    //Cellular.resetDataUsage();  //can't reset or we lose info about tower connection costs
    //Serial.print("Reset: ");
    //PrintCellUsage();   //check that reset
    //Serial.print(",");
    if(Cellular.ready())
        Serial.print("true");
    else
        Serial.print("false");
    //Serial.print("localIP = ");
    Serial.print(",");
    Serial.print(Cellular.localIP());
    //CellularSignal sig = Cellular.RSSI();
    //Serial.print("RSSI,QUAL = ");
    //Serial.println(sig);  //printed later
    
    //CONFIG: 40c06a19&nn:N1346.2_TD&pr:M0;xn:INT_VLT;nx:2;#xm:BattVolts;mty:volts;unm:volts;#xm:SolrVolts;mty:volts;unm:volts;
    //one packet per port
   if (PktCount%200 == 0 || PktCount < 3) {
        String output = "{\"key\":\"" + DeviceKey + "\",\"body\":\"";
        output.concat(DeviceID + "&");
        output.concat("nn:" + String(DeviceName) + "&");
        output.concat("pr:" + String("M0") + ";");
        output.concat("xn:" + String("DEV") + ";");
        output.concat("nx:" + String("1") + ";");
        output.concat("#xm:" + String("BattV") + ";");
        output.concat("mty:" + String("V") + ";");
        output.concat("unm:" + String("V") + ";");
        output.concat("\"}");
        success = Particle.publish("metadata", output);
        Serial.println(output);		//Note prints out in middle of csv string
        delay(5000);
   }
} //end setup


void loop()
{
    FuelGauge fuel;
    float value;
    bool success;
    int tx1, rx1, tx2, rx2, deltatx, deltarx, totalusage;
    
    //Serial.print("Before Publish: ");
    Serial.print(",");
    //Cannot call function becuase need 'data' in this scope for delta calculation
    CellularData data;
    if (!Cellular.getDataUsage(data)) {
        Serial.print("-1,-1,-1,-1,-1");
    }
    else {
        Serial.print(data); // printed as CID,TX,RX,TX,RX
    }
             
    tx1 = data.tx_total;  //v4.0.4 - usage after connection to tower
    rx1 = data.rx_total;  //v4.0.4 - usage after connection to tower
                
    value = fuel.getVCell();
    
    if (PktCount > 999)
        PktCount = 0;
    else
        PktCount++;
    
    //String output = "{\"key\":\"9d05cc635ba90494a8d023a\",\"body\":\"bat\":\"" + String(value) + "\"}";
    //JSON DATA:   40c06a19&0&M0:4.168:1.7&M1:2.203:2.199
    String output = "{\"key\":\"" + DeviceKey + "\",\"body\":\"";
    output.concat(DeviceID + "&");
    output.concat(String(PktCount) + "&");
    output.concat("M0:" + String(value) + "\"}");
    
    Serial.print(",");
    Serial.print(value,4);
    success = Particle.publish("streams", output, NO_ACK);
    //Serial.print("publish success = ");
    Serial.print(",");
    Serial.print(success);
    CellularSignal sig = Cellular.RSSI();
    // Serial.print("RSSI,QUAL = ");
    Serial.print(",");
    Serial.print(sig);
    //delay(1);
    //delay(5000);    //flush serial buffer (and UDP buffer?)
    //Serial.print("delay - publish success = ");
    //Serial.println(success);
    //Serial.print("After Publish: ");
    Serial.print(",");
    if (!Cellular.getDataUsage(data)) {
        Serial.print("-1,-1,-1,-1,-1");
    }
    else {
        Serial.print(data); // data usage counter after publish
    }
             
    tx2 = data.tx_total;  //v4.0.4 - usage after publish
    rx2 = data.rx_total;  //v4.0.4 - usage after publish
    
    deltatx = tx2 - tx1;  //v4.0.4 - usage from publish, not including connection
    deltarx = rx2 - rx1;  //v4.0.4 - usage from publish, not including connection
    //Serial.print("Delta: ");
    Serial.print(",");
    Serial.print(deltatx);
    Serial.print(",");
    Serial.print(deltarx);
    totalusage = tx2 + rx2;
    Serial.print(",");
    Serial.print(totalusage);
    Serial.println(",>");    //END OF CSV LINE
    Serial.println(output);
    // delay(5000);    //allow serial flush before sleep and rev4.0.3 delay before sleep
    delay(500);    //allow serial flush before sleep
     
    //NOTE: This bricks the device for OTA... 
    System.sleep(SLEEP_MODE_DEEP, 600);    //1200 = 20 min
    
} //end loop

void PrintCellUsage() {
            CellularData data;
            if (!Cellular.getDataUsage(data)) {
                Serial.print("-1,-1,-1,-1,-1");
            }
            else {
                Serial.print(data); // printed as CID,TX,RX,TX,RX
            }
}

Serial output with 0.6.1-rc.1 gives this. Note that last value before “>” is the total Tx and Rx usage as reported by getDataUsage().

<,1999-12-01T00:00:32Z,31,1924,1940,1924,1940,true,10.40.224.162{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&nn:HS58&pr:M0;xn:DEV;nx:1;#xm:BattV;mty:V;unm:V;"}
,31,2855,2067,2855,2067,3.9413,1,-77,37,31,2995,2067,2995,2067,140,0,5062,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&1&M0:3.941300"}
<,2016-12-22T16:08:08Z,31,521,438,521,438,true,10.42.27.154{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&nn:HS58&pr:M0;xn:DEV;nx:1;#xm:BattV;mty:V;unm:V;"}
,31,1452,565,1452,565,3.9377,1,-77,37,31,1592,565,1592,565,140,0,2157,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&2&M0:3.937700"}
<,2016-12-22T16:18:49Z,31,521,438,521,438,true,10.40.224.261{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&nn:HS58&pr:M0;xn:DEV;nx:1;#xm:BattV;mty:V;unm:V;"}
,31,1452,565,1452,565,3.9377,1,-79,37,31,1592,565,1592,565,140,0,2157,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&3&M0:3.937700"}
<,2016-12-22T16:29:31Z,31,521,438,521,438,true,10.42.27.142,31,321,438,521,438,3.9377,1,-77,37,31,661,438,661,438,140,0,1099,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&4&M0:3.937700"}
<,2016-12-22T16:40:09Z,31,74,61,74,61,true,10.40.224.151,31,74,61,74,61,3.9377,1,-79,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&5&M0:3.937700"}
<,2016-12-22T16:50:43Z,31,74,61,74,61,true,10.40.224.151,31,74,61,74,61,3.9377,1,-79,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&6&M0:3.937700"}
<,2016-12-22T17:01:18Z,31,74,61,74,61,true,10.40.224.151,31,74,61,74,61,3.9377,1,-79,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&7&M0:3.937700"}
<,2016-12-22T17:11:52Z,31,74,61,74,61,true,10.40.3.134,31,74,61,74,61,3.9377,1,-75,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&8&M0:3.937700"}
<,2016-12-22T17:22:27Z,31,74,61,74,61,true,10.40.3.134,31,74,61,74,61,3.9377,1,-75,37,31,227,61,227,61,153,0,288,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&9&M0:3.937700"}
<,2016-12-22T17:33:02Z,31,74,61,74,61,true,10.40.3.134,31,74,61,74,61,3.9377,1,-75,37,31,228,61,228,61,154,0,289,>
{"key":"9d05cc635ba90494a8d023a","body":"HOLOSPARK58&10&M0:3.937700"}

Note that (for me anyway) the serial port gets closed on sleep and does not automatically reconnect on my laptop (Macbook). I wrote a python script to brute force a workaround. It just polls the port until it opens when Electron wakes up each cycle. Now I can leave it running without manually have to reconnect the serial port.

import serial
import time
#electron dumps about 250 char.  .ser(500) should timeout

while True:
    try:
        serialport = serial.Serial("/dev/cu.usbmodem1411", 9600, timeout=5)  #electron takes about 20 sec in wake
        #print "Port open"
        output = serialport.read(500)
        if output:
            print output,
        serialport.close()
        #print "Port closed"
    except:
        #print "Error port cannot be opened"
        time.sleep(2)   #want in except so that only delays when port not available
1 Like

If you use TeraTerm for reading/sending commands instead, it will do the same thing as your python script - automatically reconnect when the serial port is available again.

1 Like