Unusually High Idle Boron Cellular Data Usage

I have a Boron that is actively sitting on idle and started consuming approx. 1.2 MB per day unexpectedly. I installed the device inside my motor control box on 7-09 and a large jump in data consumption can be observed the next day. The following two days the device performed as expected when sitting idle, before drastically spiking the following four days consistently. I have used Particle.function() and Particle.publish exactly five times each since 7-09, which should require drastically less data than is currently being consumed. Over the past week signal quality and strength has been consistent so I am unsure what the cause might be. Below is my code:

#include <Particle.h>
//sets relay connection pin to D4 on board:
const int relay = D4;
const int address = 0;
const String device_name = "DAD_SEC_54_SW_CORNER";
SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

void setup() {
    
    noInterrupts();
    //sets relay pin as output:
    pinMode(relay, OUTPUT);
    pinMode(boardLED, OUTPUT);
    
    //Checks previous well state:
    int wellState = EEPROM.read(address);
    if (wellState == 1) {
        delay(4min);
        digitalWrite(relay, HIGH);
        digitalWrite(boardLED, HIGH);
    } else {
        digitalWrite(relay, LOW);
        digitalWrite(boardLED, LOW);
    }
    interrupts();
    //Enables web API control:
    Particle.function("webControl",controlWell);
    Particle.connect();
    
    

}

void loop() {

}

int controlWell(String webInput) {
    if (webInput == "start") {
        digitalWrite(relay, HIGH);
        digitalWrite(boardLED, HIGH);
        EEPROM.put(address, 1);
        String wellState = "Well Started";
        String status = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
        Particle.publish("well_status", status, PRIVATE, WITH_ACK);
        //Particle.publish("well_start", PRIVATE, WITH_ACK);
        
        return 1;
    } else if (webInput == "stop") {
        digitalWrite(relay, LOW);
        digitalWrite(boardLED, LOW);
        EEPROM.put(address, 0);
        String wellState = "Well Stopped";
        String status = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
        Particle.publish("well_status", status, PRIVATE, WITH_ACK);
        Particle.disconnect();
        delay(4min);
        Particle.connect();
        //Particle.publish("well_stop", PRIVATE, WITH_ACK);
        
        return 0;
    } else {
        return -1;
    }
}

Usage and connectivity data:


There are a bunch of things which may or may not be an issue:

  • You cannot call delay() within a noInterrupts block. It shouldn't be necessary there.
  • It's probably not a good idea to have a delay of 4 minutes in setup. You probably should instead move the relay control to loop, controlled by a flag that can be set either in setup or from the function.
  • Also don't use delay() in the loop implementation, instead use millis() and a state machine.
  • You should not call delay() in a function handler because it will cause the function to time out from the cloud side.
  • The most likely reason you're using a lot of data is the Particle.disconnect(). You should never do that, because every time you reconnect you will use 5K of data.
  • Instead keep track of the last time to filled and when you handle that in loop, if you've recently filled, ignore the request.
2 Likes

Thank you for the suggestions! I made the suggested revisions and will provide an update on the data usage over the next few days. New code below:

#include <Particle.h>

//sets relay connection pin to D4 on board:
const int relay = D4;
//Onboard LED:
const int boardLED = D7;
const int address = 0;
const String device_name = "DAD_SEC_54_SW_CORNER";
SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

bool startWell = false;
bool stopWell = false;
unsigned long lastStopTime = 0;

void setup() {
    //sets relay pin as output:
    pinMode(relay, OUTPUT);
    pinMode(boardLED, OUTPUT);
    
    //Checks previous well state:
    int wellState = EEPROM.read(address);
    if (wellState == 1) {
        startWell = true;
    } else {
        stopWell = true;
    }
    
    //Enables web API control:
    Particle.function("webControl",controlWell);
}

void loop() {
    if (startWell && millis() - lastStopTime > 4 * 60 * 1000UL) {
        startWell = false;
        digitalWrite(relay, HIGH);
        digitalWrite(boardLED, HIGH);
        EEPROM.put(address, 1);
        String wellState = "Well Started";
        String status = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
        Particle.publish("well_status", status, PRIVATE, WITH_ACK);
            if (Particle.connected() == false) {
                Particle.connect();
                }
    } else if (stopWell) {
        stopWell = false;
        digitalWrite(relay, LOW);
        digitalWrite(boardLED, LOW);
        EEPROM.put(address, 0);
        String wellState = "Well Stopped";
        String status = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
        Particle.publish("well_status", status, PRIVATE, WITH_ACK);
        lastStopTime = millis();
        if (Particle.connected() == false) {
                Particle.connect();
                }
    }
}

int controlWell(String webInput) {
    if (webInput == "start") {
        startWell = true;
        return 1;
    } else if (webInput == "stop") {
        stopWell = true;
        return 0;
    } else {
        return -1;
    }
}
2 Likes

That looks good. Just a few minor changes:

At the end of setup() add Particle.connect() because you're using SEMI_AUTOMATIC mode.

Remove the two places where Particle.connect() is called if not connected. This is not necessary as it's done automatically in SEMI_AUTOMATIC mode once you've connected once.

The last change may vary based on what you want the desired behavior to be. I think you probably want to only act on startWell and stopWell if Particle.connected() is true. For a function call, it will already be, but if you set startWell during setup you won't be connected yet.

The other option is to just check for Particle.connected() before Particle.publish(). This will cause the well to run even if not connected, but you will be missing the well_status events. Both will work, it's just a matter of how you want it to behave.

I'd probably also add a safety so if you don't get a stopWell after a period of time after startWell, it will stop even if not commanded to, in case the device goes offline with it running and can't receive the stop.

I made a few changes to the program's logic because the well needs to resume its previous state first regardless of internet connection. I'd rather not be stuck waiting for a cellular connection for an unknown amount of time before the well starts. The well runs 99% of the time so I never want it to turn itself off.

#include <Particle.h>

//sets relay connection pin to D4 on board:
const int relay = D4;
//Onboard LED:
const int boardLED = D7;
const int address = 0;
const String device_name = "DAD_SEC_54_SW_CORNER";
SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

bool startWell = false;
bool stopWell = false;
unsigned long lastStopTime = 0;

void setup() {
    //sets relay pin as output:
    pinMode(relay, OUTPUT);
    pinMode(boardLED, OUTPUT);
    
    //Checks previous well state:
    int wellState = EEPROM.read(address);
    if (wellState == 1) {
        startWell = true;
    } else {
        stopWell = true;
    }
    
    //Enables web API control:
    Particle.function("webControl",controlWell);
}

void loop() {
    if (startWell && millis() - lastStopTime > 4 * 60 * 1000UL) {
        startWell = false;
        digitalWrite(relay, HIGH);
        digitalWrite(boardLED, HIGH);
        EEPROM.put(address, 1);
            if (Particle.connected() == false) {
                Particle.connect();
                } else {
                    String wellState = "Well Started";
                    String status = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
                    Particle.publish("well_status", status, PRIVATE, WITH_ACK);
                }
    } else if (stopWell) {
        stopWell = false;
        digitalWrite(relay, LOW);
        digitalWrite(boardLED, LOW);
        EEPROM.put(address, 0);
        lastStopTime = millis();
        if (Particle.connected() == false) {
                Particle.connect();
                } else {
                    String wellState = "Well Stopped";
                    String status = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
                    Particle.publish("well_status", status, PRIVATE, WITH_ACK);
                }
    }
}

int controlWell(String webInput) {
    if (webInput == "start") {
        if (millis() - lastStopTime < 4 * 60 * 1000UL) {
            unsigned long elapsedTime = millis() - lastStopTime;
            unsigned long remainingTime = 4 * 60 * 1000UL - elapsedTime;
            unsigned long seconds = remainingTime / 1000UL;
            unsigned long minutes = seconds / 60UL;
            seconds = seconds % 60UL;
String timeStr = String(minutes) + " minutes " + String(seconds) + " seconds";
            String notReady = "Well start in: " + timeStr;
            String notReadyFormat = String::format("{\"wellState\": \"%s\", \"device_name\": \"%s\"}", notReady.c_str(), device_name.c_str());
            Particle.publish("well_status", notReadyFormat, PRIVATE, WITH_ACK);
            startWell= true;
        } else {
            startWell = true;
        }
        return 1;
    } else if (webInput == "stop") {
        stopWell = true;
        return 0;
    } else {
        return -1;
    }
}

You should still check for connected before publish, as in:

if (Particle.connected()) {
  Particle.publish("well_status", status, PRIVATE, WITH_ACK);
}

The reason is that if you in the process of reconnecting, Particle.publish can block until connected, which can lead to unexpected behavior. It probably won't adversely affect your code since you only control the pump from the cloud, but it can cause unexpected behavior if you also have local control (like a button) and it's good practice to get into.

2 Likes

Ok, thank you so much! I will have to keep an eye on the data usage over the next few days and see if these changes resolve the issue.

I now have multiple particles deployed with widely varying data usage from 0.027 to 0.303 MB. These particles are all running the same device OS version, same firmware, and the hardware is configured the same for all of them. Sometimes I have been able to negate this high data usage temporarily by first using the device restore tool to flash tinker, letting the Boron sit a day, then flashing my firmware back onto the device. However, if I follow up with a firmware update oftentimes the devices will return to wildly consuming data. Attached below is an image of devices configured the same way. The four staggered bars in the middle are the result of me using the device restore tool to flash tinker on them and allowing them to sit idle the remainder of the day. I also went and checked to see if disconnect events may have been the cause for this data overuse, but at most they only had 1 cloud disconnect event, while the majority of them had 0. See photo attached below:

As I mentioned above, these devices remain idle and breathing 99% of the time, and are not actively publishing any information. I also figured I could go with SYSTEM_MODE(AUTOMATIC) and leave the cellular connection management to the device since I have no need to ever disconnect after startup. None of the devices have ever been stuck inside the loop because of Particle.publish, and they are quite responsive to cloud commands. Attached below is my code:

#include <Particle.h>

PRODUCT_VERSION(10);

const int relay = D4;
const int boardLED = D7;
const int address = 1000;
const int nameSaveAddress = 1001;
const int maxNameLength = 100;
String device_name = "N/A";
bool nameUpdate = false;
bool readNameAtStart = true;
const int isNameSavedAddress = 1002;
String wellStateVariable = "N/A";
char eeprom_device_name[maxNameLength];

SerialLogHandler logHandler;
SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(AUTOMATIC);
bool startWell = false;
bool stopWell = false;
bool nameSet = true;
unsigned long lastStopTime = 0;

void subscriptionHandler(const char *topic, const char *data)
{
    Log.info("Started subscriptionHandler");
    String receivedName = String(data);
    Log.info("Received Name: " + receivedName);

    saveNameToEEPROM(receivedName);
    device_name = receivedName;
    Log.info("Confirming name save to EEPROM...");
    readNameFromEEPROM();
    int nameSaveIntTest = EEPROM.read(isNameSavedAddress);
    if (nameSaveIntTest != 1)
    {
        Log.error("isNameSavedAddress was not equal to one... fixing...");
        EEPROM.put(isNameSavedAddress, 1);
    }
}

void setup()
{
    pinMode(relay, OUTPUT);
    pinMode(boardLED, OUTPUT);

    if (EEPROM.read(address) == 1)
    {
        startWell = true;
    }
    else
    {
        stopWell = true;
    }

    readNameFromEEPROM();
    
    if (EEPROM.read(isNameSavedAddress) != 1 || device_name == "N/A" || device_name == "DA" || device_name == "RU")
    {
        readNameAtStart = false;
        nameUpdate = true;
        EEPROM.put(isNameSavedAddress, 1);
    }
    Particle.function("webControl", controlWell);
    Particle.function("onlineReset", remoteReset);
    Particle.variable("state", wellStateVariable);
    Particle.subscribe("particle/device/name", subscriptionHandler);
}

void loop()
{
    if (Particle.connected() && nameUpdate)
    {
        nameUpdate = false;
        Particle.publish("particle/device/name");
        
        readNameFromEEPROM();
    }

    //Well Start Code: MAKE SURE TO FIX TIME DELAY AFTER TESTING
    if (startWell && millis() - lastStopTime > 4 * 60 * 1000UL)
    {
        Log.info("Well On and name: " + device_name);
        startWell = false;
        digitalWrite(relay, HIGH);
        digitalWrite(boardLED, HIGH);
        int checkEeprom = EEPROM.read(address);
        if (checkEeprom != 1)
        {
            EEPROM.put(address, 1);
        }
        if (Particle.connected())
        {
            String wellState = "Well Started";
            String status = String::format("{\"status\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
            Particle.publish("pushoverStatus", status, PRIVATE, WITH_ACK);
        }
        wellStateVariable = "On";
        System.disableUpdates();

        //Well Stop Code:
    }
    else if (stopWell)
    {
        Log.info("Well Off and name: " + device_name);
        stopWell = false;
        digitalWrite(relay, LOW);
        digitalWrite(boardLED, LOW);
        int checkEeprom = EEPROM.read(address);
        if (checkEeprom != 0)
        {
            EEPROM.put(address, 0);
        }
        lastStopTime = millis();
        if (Particle.connected())
        {
            String wellState = "Well Stopped";
            String status = String::format("{\"status\": \"%s\", \"device_name\": \"%s\"}", wellState.c_str(), device_name.c_str());
            Particle.publish("pushoverStatus", status, PRIVATE, WITH_ACK);
        }
        wellStateVariable = "Off";
        System.enableUpdates();
    }
    if (readNameAtStart)
    {
        readNameFromEEPROM();
        readNameAtStart = false;
    }
}

//Online API Call Method:
int controlWell(String webInput)
{
    if (webInput == "start")
    {
        if (millis() - lastStopTime < 4 * 60 * 1000UL)
        {
            unsigned long elapsedTime = millis() - lastStopTime;
            unsigned long remainingTime = 4 * 60 * 1000UL - elapsedTime;
            unsigned long seconds = remainingTime / 1000UL;
            unsigned long minutes = seconds / 60UL;
            seconds = seconds % 60UL;
            String timeStr = String(minutes) + " minutes " + String(seconds) + " seconds";
            String notReady = "Well start in: " + timeStr;
            String notReadyFormat = String::format("{\"status\": \"%s\", \"device_name\": \"%s\"}", notReady.c_str(), device_name.c_str());
            Particle.publish("pushoverStatus", notReadyFormat, PRIVATE, WITH_ACK);
            startWell = true;
        }
        else
        {
            startWell = true;
        }
        return 1;
    }
    else if (webInput == "stop")
    {
        if (startWell)
        {
            String startCancel = "Well start cancelled";
            String startCancelFormat = String::format("{\"status\": \"%s\", \"device_name\": \"%s\"}", startCancel.c_str(), device_name.c_str());
            Particle.publish("pushoverStatus", startCancelFormat, PRIVATE, WITH_ACK);
            startWell = false;
        }
        else
        {
            stopWell = true;
        }
        return 0;
    }
    else
    {
        return -1;
    }
}

void saveNameToEEPROM(const String &data)
{
    int nameLength = data.length();

    if (nameLength >= maxNameLength)
    {
        Log.error("Name length exceeds maximum size. Not saving to EEPROM.");
        return;
    }

    // Save the name to EEPROM, including the null terminator
    data.toCharArray(eeprom_device_name, maxNameLength);
    EEPROM.put(nameSaveAddress, eeprom_device_name);

    Log.info("Name saved to EEPROM: " + data);

    // Update the global String variable
    device_name = data;
}

void readNameFromEEPROM()
{
    // Read the name from EEPROM
    EEPROM.get(nameSaveAddress, eeprom_device_name);
    eeprom_device_name[maxNameLength - 1] = '\0'; // Ensure null-termination

    // Check if the savedValue is valid (not an empty string)
    if (eeprom_device_name[0] != '\0')
    {
        // Replace non-alphanumeric characters (excluding underscores) with 'A' until the null-terminator
        for (int i = 0; i < maxNameLength; i++)
        {
            if (eeprom_device_name[i] == '\0')
            {
                break; // Stop replacing characters at the null-terminator
            }
            if (!isAlphaNumeric(eeprom_device_name[i]) && eeprom_device_name[i] != '_')
            {
                if (eeprom_device_name[0] == 'D')
                {
                    eeprom_device_name[i] = 'A';
                }
                else if (eeprom_device_name[0] == 'R')
                {
                    eeprom_device_name[i] = 'U';
                }
            }
        }

        // Update the global String variable
        device_name = eeprom_device_name;

        // Display the saved string
        Log.info("Saved Name from EEPROM: " + device_name);
    }
    else
    {
        // If no valid data is found, set the default string
        Log.error("No valid name found in EEPROM!");

        // Update the global String variable
        device_name = "N/A";
    }
}

int remoteReset(String textInput)
{
    if (textInput == "sr")
    {
        EEPROM.put(isNameSavedAddress, 1);
        delay(1000);
        System.reset();
        return 0;
    }
    else if (textInput == "hr")
    {
        EEPROM.put(address, 0);
        EEPROM.put(isNameSavedAddress, 0);
        delay(1000);
        System.reset();
        return 1;
    }
    else if (textInput == "nr")
    {
        EEPROM.put(isNameSavedAddress, 0);
        nameUpdate = true;
        return 2;
    }
    else if (textInput == "nt")
    {
        readNameFromEEPROM();
        return 3;
    }
    else
    {
        return -1;
    }
}

You should pull the data operations report, available in the billing page of the console, to make sure that it's only cellular data that is high on those devices, and not also data operations. That will help isolate the cause.

It don't see anything obvious from a quick look at the code that should cause large data usage if the devices are staying connected to the cloud.

I'm pulling the report, but the Data operations for all the products is about where I would expect it to be.

Ok, I pulled the report and data operations are exceptionally low for the devices included in the chart above. The highest data operations I observed on any of these devices was 12.

Could you DM me the device ID of a device with high usage? I'll see if there's anything noticeable on the cloud side.

Keeping in mind I don't normally examine the cloud logs so I might be missing something. And the logs I'm looking at only go back 3 days. However both of the devices had OTA updates at the oldest point in the log. I think something is causing the OTA update to not be applied, which is causing the cloud to retry the update multiple times, which is causing the elevated data usage.

However I'm not 100% sure that's what's happening, and I don't know what that would be happening, if that's the cause.

If this were true, wouldn't they be running the incorrect firmware? They are currently running the most recent and correct firmware sent through an OTA update as far as I can tell.

I can't see enough back into the logs, because near the beginning (August 1) they had a successful flash. Possibly there were failures before that? I'm not particularly confident of this being the reason, but it would explain high data usage with low data operations usage.

After that, things look pretty normal. The devices are not repeatedly rebooting or reconnecting, everything looks pretty normal in the last day or two.

I've had this issue for the past two weeks since I started using these particles. I've never had any failures with flashing, however, sometimes the data usage would increase dramatically after a flash, which could only be remedied by following the steps I detailed above. This unusual data usage and spiking has been making it difficult to deploy devices with confidence that they will perform as intended. Sometimes the Borons would settle down with data usage after a reset and an OTA update, before resuming their irregular usage of data afterwards. Is there somebody that might be more familiar with this issue?

I'm wondering if it's related to System.disableUpdates(). It's probably timing-sensitive. One of the things about disableUpdates is that it's a cloud operation that sends a message to the cloud to prevent updates from being sent down and the reverse to enable them.

But from looking at your code, you really want to System.disableReset(). Since you're using SYSTEM_THREAD(ENABLED) the downloading will occur in the background and if reset is currently disabled, it will hold the updates until System.enableReset() at which point the device will reset and apply the update. The disableReset/enableReset are entirely local and only affect whether the device resets after an OTA update.

Okay, I will make the change and deploy this software to the two misbehaving devices. It will also reduce data operations so might as well implement it regardless.

Unfortunately, that change did not fix the high data usage. I flashed the new firmware to one of the devices while leaving the other one unchanged and they have both continued to consume data at a similar rate.