Abnormal SIM Card data volume usage on Electron

Troubleshooting
1

We are building a product based on the Electron and currently we are trying to figure out what is the best way to safe data volume.

The product reads out a sensor every five minutes and sends its value to a backend using publish / subscribe methods.

After the sensor has been read out the device goes back into deep sleep to safe battery since the product will be solar powered.

In the documentation I read the following about optimising data consumption (link to quote):

From 0.6.0, the device determines that a full reinitialization isn't needed and reuses the existing session ...

as well as:

A key requirement for the device to be able to determine that the existing session can be reused is that the functions, variables and subscriptions are registered BEFORE connecting to the cloud.

There is also an example that shows how to register functions and subscriptions before connecting to the cloud:

// EXAMPLE USAGE
// Using SEMI_AUTOMATIC mode to get the lowest possible data usage by
// registering functions and variables BEFORE connecting to the cloud.
SYSTEM_MODE(SEMI_AUTOMATIC);

void setup() {
    // register cloudy things
    Particle.function(....);
    Particle.variable(....);
    Particle.subscribe(....);
    // etc...
    // then connect
    Particle.connect();
}

So in my code I combined the information I got from the documentation and came up with this:

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

FuelGauge fuel;
PMIC pmic; 

long lastPublish = 0;

void setup() {
    
    pmic.begin();
    pmic.setChargeCurrent(0,0,1,0,0,0);
    pmic.setInputCurrentLimit(1500);
    
    Cellular.on();
    Cellular.connect();
    waitUntil(Cellular.ready);
    
    Particle.connect();
    waitUntil(Particle.connected);
    
    Particle.publish("B",
        "v:" + String::format("%.2f",fuel.getVCell()) +
        ",c:" + String::format("%.2f",fuel.getSoC()),
        60, PRIVATE
    );    
    
    lastPublish = 0;
}

void loop() {
    
    if (millis() - lastPublish > 5000) {
        System.sleep(SLEEP_MODE_DEEP, 240);
    }   

}

But using a Particle.io SIM card this leads almost 0.6 MByte / hour data consumption! Given the fact that a reconnection to the cloud needs almost 4,4KB and by recalculating how often it will reconnect I am pretty sure that the above code always does a handshake with the cloud and after every wake up a full reconnection is performed !!!

So I am currently having problems to set up Deep Sleep and Re-Connection as stated in the documentation!!!

2

I’ve made some test with and without Deep Sleep:

  1. Deep Sleep: 3900 Bytes per Hour
  2. No Sleep: 2300 Bytes per Hour

By waking up 12 times during an hour this is: 12 x 135 Bytes (Session re-use) = 1620 Bytes

2300 Bytes (No Sleep) + 1620 Bytes = 3920 Bytes which is exactly the difference I measured between 1) Deep Sleep and 2) No Sleep! So these facts are real!

But why does sending 12 times per hour a publish of about 15 bytes lead to 191 Bytes? I have upgraded the code to use a NO_ACK method so even with 60 bytes of CoAP Overhead it’s still more than 131 Bytes of data!!! Just for a single Publish!

I am pretty sure that this is due to encryption since I think that all data is being encrypted, right?

There are people claiming that with sending two values every five minutes they stay below 1 Megabyte per month. - Which cannot be true since 2.3 KB x 24h x 31d = 1.7 Megabyte!!!

3

Setup a Ubidots account and use this code below to post every 2 mins.

Let me know how much more efficient it is compared to the Particle Publish Method.

It’s way less than the numbers your seeing.

SYSTEM_MODE(SEMI_AUTOMATIC);
//SYSTEM_THREAD(ENABLED);
// This #include statement was automatically added by the Particle IDE.
#include "Ubidots/Ubidots.h"

#define TOKEN "Your Ubidots Token Here"  // Put here your Ubidots TOKEN
#define DATA_SOURCE_NAME "ElectronSleepNew"

SerialLogHandler logHandler(LOG_LEVEL_ALL);  //This serial prints system process via USB incase you need to debug any problems you may be having with the system.

Ubidots ubidots(TOKEN); // A data source with particle name will be created in your Ubidots account.
//

int button = D0;         // Connect a Button to Pin D0 to Wake the Electron when in System Sleep mode. 
int ledPin = D7;         // LED connected to D1
int sleepInterval = 60;  // This is used below for sleep times and is equal to 60 seconds of time. 

ApplicationWatchdog wd(660000, System.reset); //This Watchdog code will reset the processor if the dog is not kicked every 11 mins which gives time for 2 modem reset's. 

void setup() {
 //Serial.begin(115200);
 pinMode(button, INPUT_PULLDOWN);  // Sets pin as input
 pinMode(ledPin, OUTPUT);          // Sets pin as output

 ubidots.setDatasourceName(DATA_SOURCE_NAME); //This name will automatically show up in Ubidots the first time you post data. 
 
 PMIC pmic; //Initalize the PMIC class so you can call the Power Management functions below. 
 pmic.setChargeCurrent(0,0,1,0,0,0); //Set charging current to 1024mA (512 + 512 offset)
 pmic.setInputVoltageLimit(4840);   //Set the lowest input voltage to 4.84 volts. This keeps my 5v solar panel from operating below 4.84 volts.  
}

void loop() {
    
FuelGauge fuel; // Initalize the Fuel Gauge so we can call the fuel gauge functions below. 
 
    
if(fuel.getSoC() > 20) // If the battery SOC is above 20% then we will turn on the modem and then send the sensor data. 
  {
   
   float value1 = fuel.getVCell();
   float value2 = fuel.getSoC();
   
  ubidots.add("Volts", value1);  // Change for your variable name
  ubidots.add("SOC", value2);    

  Cellular.connect();  // This command turns on the Cellular Modem and tells it to connect to the cellular network. 
  
   if (!waitFor(Cellular.ready, 600000)) { //If the cellular modem does not successfuly connect to the cellular network in 10 mins then go back to sleep via the sleep command below. After 5 mins of not successfuly connecting the modem will reset.  
    
    System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); //Put the Electron into Sleep Mode for 2 Mins + leave the Modem in Sleep Standby mode so when you wake up the modem is ready to send data vs a full reconnection process.  
    
}  
  
     ubidots.sendAll(); // Send fuel gauge data to your Ubidots account. 

     digitalWrite(ledPin, HIGH);   // Sets the LED on
     delay(250);                   // waits for a second
     digitalWrite(ledPin, LOW);    // Sets the LED off
     delay(250);                   // waits for a second
     digitalWrite(ledPin, HIGH);   // Sets the LED on
     delay(250);                   // waits for a second
     digitalWrite(ledPin, LOW);    // Sets the LED off
  
     System.sleep(D0, RISING,sleepInterval * 2, SLEEP_NETWORK_STANDBY); //Put the Electron into Sleep Mode for 2 Mins + leave the Modem in Sleep Standby mode so when you wake up the modem is ready to send data vs a full reconnection process.  
    
  }
  else //If the battery SOC is below 20% then we will flash the LED 4 times so we know. Then put the device into deep sleep for 1 hour and check SOC again. 
  {
      
  //The 6 lines of code below are needed to turn off the Modem before sleeping if your using SYSTEM_THREAD(ENABLED); with the current 0.6.0 firmware. It's a AT Command problem currently. 
  //Cellular.on();
  //delay(10000);
  //Cellular.command("AT+CPWROFF\r\n");
  //delay(2000);
  //FuelGauge().sleep();
  //delay(2000);
  
  
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, HIGH);   // Sets the LED on
  delay(150);                   // Waits for a second
  digitalWrite(ledPin, LOW);    // Sets the LED off
  
  System.sleep(SLEEP_MODE_DEEP, 3600);  //Put the Electron into Deep Sleep for 1 Hour. 
  
  }
}
4

Thanks! Ubidots is pretty cool but I’ve taken a look at their library for Particle and (I already knew it before) they are sending values / request without any encryption so that means that the Auth-Token is transmitted human-readable and can be read out with a simple packet sniffer.

There is currently no third-party TLS/SSL support for Particle and Ubidots Library isn’t using any encryption at all!

Without any encryption the data being send will always be much smaller than same data (or even less) which is encrypted.

For a commercial product you should always use encrypted data transmission so that no one can easily steal your auth-token and post wrong values or fire callbacks, etc!

I am still making test between third-party and particle SIM to see wether there are some differences between providers.

But I am pretty sure that encryption adds up to every byte transferred…

5

This seems to pretty much carry on from this other thread (which I closed for that reason)

What does Particle Console | Build your connected product say?
I recall an issue where after four wakes a full handshake happened for some obscure reason
https://github.com/spark/firmware/issues/1133

6

Yes that's true and I agree that we should close the other.

I've made sure that there is no handshake happening and debugged it on device and in the console as well. Everytime a handshake is being made you see DEVICE-ONLINE Message appearing in the logs - which is not the case with my devices.

Also you have to make sure that between publish and deep sleep there's some idle time - but not with a delay - just to give the system thread some time to run Particle.process() a bit.

There is just a single publish happening with 16 bytes of user data.

But see my latest info on that:

When waking from deep sleep 135 bytes are send to reuse current session. So 135 Bytes x 12-Times during an hour = 1620 Bytes - which is exactly the difference between Deep Sleep and No Sleep!!! (and not calculated - these are measured facts)

This exactly behaves as written in the docs:

Connecting to the cloud after reset or wake-up sends just a reconnect message, using 135 bytes of data. (Source)

I also read about how the communication is build up and found this one here:

So - Let's say a single Publish with NO_ACK (just like in my case) takes about 70 Bytes overhead - this makes:

12 Times per Hour x 70 Bytes = 840 Bytes overhead per Hour

and we have to add:

16 Bytes payload x 12 Times per Hour = 192 Bytes per Hour

this makes a total of 1.032 Bytes - or 1 KB per Hour

But I am having 2300 Bytes / 2,3 KB per Hour - So there is a difference - and which cannot be because of a handshake (which really does not take place). It has to be something that happens underneath - or (more realistic) - It's because of encryption of payload since you cannot simply take 16 Bytes payload and encrypt it without adding up to the overall size!

7

This should have been fixed in a previous update. Is this a regression?

8

No I guess not. It was just an issue with my code since I am using SYSTEM_THREAD(ENABLED) in conjunction with SYSTEM_MODE(SEMI_AUTOMATIC) because the System-Thread speeds up my cellular connection from more than 30s to just 3-5 seconds! But it also leaded to the fact that setting deep sleep took place way to early so I added a small idle phase!

BTW: Not pretty sure if this is important but as mentioned: System Thread allows me to reconnect to cellular after deep sleep within just a few seconds - without it, the electron often needs almost 30s+ to reconnect!

9

@simmikolon @RWB you're right; Ubidots' libraries are meant to help you send data to the cloud in the least amount of steps with maximum connectability, accelerating your IoT prototyping and design efforts. But they are not optimized for security.

Ubidots does support SSL encryption in its API, so here is my 2 cents regarding security:

1- Use Particle Webhooks to secure Device to Cloud Communication

If your data uploads fit into the time/size constraints of event publishing, you can use a web hook to simply Particle.publish from your code and have the web hook convert it into an SSL request to your server. (The data communication between your device and the cloud is also encrypted.)
https://docs.particle.io/guide/tools-and-features/webhooks/8

2- Send data to Ubidots from Particle Cloud using HTTPS
Particle Webhooks have an "Enforce SSL" option which you can use to send data securely to Ubidots.

Connect your Particle device to Ubidots using Particle Webhooks | Ubidots Help Center

10

I would be very much interested in this topic, but find the discussion break without solution. @simmikolon did you resolve it? I am preparing myself to deploy many sensors in the field and the idea of speeding up the cellular after deep sleep together with low but encrypted data usage would be sweet. Could you post your code that you rested on in this issue?