Free Sandbox Data Operations Limit being reached without explanation

Hello,

Like many others, it seems that my free sandbox data operation limit is being reached well before the monthly billing cycle refreshes. I only have three devices running right now. Two Photons control lights for a couple of aquariums and one Photon controls a hydroponic garden. I have a raspberry pi running Node-Red which calls functions and variables on the photons and displays them on a dashboard.

I have tried everything within reason to reduce the amount of data operations, however, no matter how much I reduce the calls, the amount I see in the usage data does not change. I have changed the refresh rate of every variable from 15 min to 1 hour and have greatly reduced the number of variables being called to the bare minimum. For one light, I have removed the particle.variables and functions altogether. But it’s usage report is not changing.

Here is the code currently running on that Photon, which is identical to my other Photon minus a temperature sens. Yet the other photon has no data usage being incurred.

#include <spark-dallas-temperature.h>

#define ONE_DAY_MILLIS (24 * 60 * 60 * 1000)
unsigned long lastSync = millis();//variable for syncing clock to Particle cloud 

#define timeOnSummer       6   //start fading at 6, fully on by 7
#define timeOffSummer      19  //start fading at 7, off by 8       *13 hours of full light
#define timeOnSpringFall   7   //start fading at 7, fully on by 8
#define timeOffSpringFall  18  //start fading at 6, off by 7       *11 hours of full light
#define timeOnWinter       8   //start fading at 8, fully on by 9
#define timeOffWinter      17  //start fading at 5, off by 6       *8 hours of full light

#define TIMER_LENGTH 10000// ms 

int pwmVal;
int pwmPin = A5;

int ACPowerPin = D5;

int counter = 0;
bool debug = 0;

STARTUP(WiFi.selectAntenna(ANT_EXTERNAL));//Use external Antenna

#define ONE_WIRE_BUS D4
#define TEMPERATURE_PRECISION 12
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

DeviceAddress inWaterThermometer = { 0x28, 0xC3, 0xD1, 0x5F, 0x07, 0x00, 0x00, 0x73 };

double waterTempC = -1;
double watertempf = -1;

long tempCheckTimer = 0;

// --------------------------------------------------------------------
void setup() 
{ 
  pinMode(pwmPin, OUTPUT);
  analogWrite(pwmPin, 0);
  
  pinMode(ACPowerPin, OUTPUT);
  digitalWrite(ACPowerPin, LOW);
  
  //Particle.variable("watertempf", watertempf);
  //Particle.variable("pwmVal", pwmVal);
  //Particle.function("ACPower", ACPower);
    
  if(debug)  
  Serial.begin(9600);
  
  RGB.control(true);
  RGB.color(0, 0, 0);//Turn RGB LED Off
  
  Time.zone(-6);//Set Time zone to MDT
  Particle.syncTime();

  // DS18B20 initialization
  //Initialize the I2C sensors and ping them
  sensors.begin();
  sensors.setResolution(inWaterThermometer, TEMPERATURE_PRECISION);


  checkDateTime();
  getTemp();
  
  if(debug)
  Serial.println(Time.timeStr());
}

// --------------------------------------------------------------------
void loop() 
{
  checkDateTime();
  
  if((millis() - tempCheckTimer) > TIMER_LENGTH)
  {
    getTemp();
    //getWeather();
    tempCheckTimer = millis();
  }
  
  // DST adjstment
  //bool daylightSavings = IsDST(Time.day(), Time.month(), Time.weekday());
  //Time.zone(daylightSavings? -6 : -7);

  if (millis() - lastSync > ONE_DAY_MILLIS) {
    // Request time synchronization from the Particle Device Cloud
    Particle.syncTime();
    lastSync = millis();
  }
  
}


//---------------------------------------------------------------
// int ACPower(String command)
// {
//   if(command == "on")
//   {
//     digitalWrite(ACPowerPin, HIGH);
//     return 1;
//   }
//     if(command == "off")
//   {
//     digitalWrite(ACPowerPin, LOW);
//     return 2;
//   }
//   else return -1;
// }
//---------------------------------------------------------------
void update18B20Temp(DeviceAddress deviceAddress, double &tempC)
{
  tempC = sensors.getTempC(deviceAddress);
}
//---------------------------------------------------------------
void getTemp()
{
    //get temp from DS18B20
    sensors.requestTemperatures();
    update18B20Temp(inWaterThermometer, waterTempC);

    if(waterTempC < -100)
    {
      watertempf = watertempf;//push last value so data isn't out of scope
    }
    else
    {
      watertempf = (waterTempC * 9)/5 + 32;//convert to F
    }
}
//---------------------------------------------------------------
void getWeather()
{
  // Measure Relative Humidity from the HTU21D or Si7021
//   humidity = sensor.getRH();

//   // Measure Temperature from the HTU21D or Si7021
//   tempf = sensor.getTempF();
//   // Temperature is measured every time RH is requested.
//   // It is faster, therefore, to read it from previous RH
//   // measurement with getTemp() instead with readTemp()

//   //Measure the Barometer temperature in F from the MPL3115A2
//   baroTemp = sensor.readBaroTempF();

//   //Measure Pressure from the MPL3115A2
//   pascals = sensor.readPressure();

  //If in altitude mode, you can get a reading in feet with this line:
  //float altf = sensor.readAltitudeFt();
}

////////////////////////////////////////////////////////////////////////////////
void checkDateTime()
{
  counter++;
  if(counter == 6000)//check every ~30 seconds
  {
    int month = Time.month();
    
    if(month == 12 || month == 1 || month == 2)
    winter();//if == the months above, run the winter routine
    
    if(month == 3 || month == 4 || month == 5 || month == 9 || month == 10 || month == 11)
    springFall();//if == the months above, run the spring/fall routine
    
    if(month == 6 || month == 7 || month == 8)
    summer();//if == the months above, run the summer routine

    if(debug)
    Serial.println(Time.timeStr());
    
    //For AC power control
    if(pwmVal == 255)
    {
        digitalWrite(ACPowerPin, HIGH); 
    }
    else
    {
        digitalWrite(ACPowerPin, LOW);
    }

    counter = 0;
  }
}
////////////////////////////////////////////////////////////////////////////////
void summer()
{
  //Fade On Gradually   
  if(Time.hour() == timeOnSummer)
  {
    pwmVal = map(Time.minute(), 1, 59, 1, 255);
    analogWrite(pwmPin, pwmVal);
  }
    
  //Fully On
  else if(Time.hour() > timeOnSummer && Time.hour() < timeOffSummer)
  {
    pwmVal = 255;
    analogWrite(pwmPin, pwmVal);
  }
  //Fade Off Gradually
  else if(Time.hour() == timeOffSummer)
  {
    pwmVal = map(Time.minute(), 1, 59, 255, 1);
    analogWrite(pwmPin, pwmVal);
  }
  //Fully Off
  else
  {
    pwmVal = 0;
    analogWrite(pwmPin, pwmVal);
  }
  
  
}
////////////////////////////////////////////////////////////////////////////////
void winter()
{
  //Fade On Gradually   
  if(Time.hour() == timeOnWinter)
  {
    pwmVal = map(Time.minute(), 1, 59, 1, 255);
    analogWrite(pwmPin, pwmVal);
  }
    
  //Fully On
  else if(Time.hour() > timeOnWinter && Time.hour() < timeOffWinter)
  {
    pwmVal = 255;
    analogWrite(pwmPin, pwmVal);
  }
  //Fade Off Gradually
  else if(Time.hour() == timeOffWinter)
  {
    pwmVal = map(Time.minute(), 1, 59, 255, 1);
    analogWrite(pwmPin, pwmVal);
  }
  //Fully Off
  else
  {
    pwmVal = 0;
    analogWrite(pwmPin, pwmVal);
  }
}
////////////////////////////////////////////////////////////////////////////////
void springFall()
{
  //Fade On Gradually   
  if(Time.hour() == timeOnSpringFall)
  {
    pwmVal = map(Time.minute(), 1, 59, 1, 255);
    analogWrite(pwmPin, pwmVal);
  }
    
  //Fully On
  else if(Time.hour() > timeOnSpringFall && Time.hour() < timeOffSpringFall)
  {
    pwmVal = 255;
    analogWrite(pwmPin, pwmVal);
  }
  //Fade Off Gradually
  else if(Time.hour() == timeOffSpringFall)
  {
    pwmVal = map(Time.minute(), 1, 59, 255, 1);
    analogWrite(pwmPin, pwmVal);
  }
  //Fully Off
  else
  {
    pwmVal = 0;
    analogWrite(pwmPin, pwmVal);
  }
}
/////////////////////////////////////////////////////////////
bool IsDST(int dayOfMonth, int month, int dayOfWeek)
{
  if (month < 3 || month > 11) 
  {
    return false;
  }
  if (month > 3 && month < 11) 
  {
    return true;
  }

  int previousSunday = dayOfMonth - (dayOfWeek - 1);
  if (month == 3)
  {
    return previousSunday >= 8;
  }
  return previousSunday <= 0;
}

That code was uploaded several days ago, commenting out the particle.variables and functions, however, as the report downloaded from the billing page shows, it’s still incurring data usage (Photon_B5 Turtles). I checked on the Particle Console, and the variables and function that were on that Photon are no longer available, so I know that this version of the code is running on it.

What am I missing? I deleted the objects from the Node-Red flow altogether, but that hasn’t changed the amount of data being used either.

Any help is appreciated. These limits placed on the free plan are very restrictive and are causing headaches for long-time, loyal Particle customers unnecessarily. It would be very helpful if these usage reports from the billing page had more information such as which operations were the ones incurring an operation charge rather than just usage by device. It feels like there is an issue in how these are calculated to force people to upgrade to the paid plans. It makes no sense that I should be reaching my limit with two devices in less than a month when there are up to 100 devices allowed on the free plan.

Hello,

If you go to the Billing page in your account, you’ll see an option to generate a report based on a Device or Product with an option to export based on a specific duration.

Hi lanceseidman,

Thanks for the reply. The spreadsheet screenshot is the device report from the billing page of my account for my only three active devices. That is the last week of usage by device. I changed my code three days ago (Sun, Sept, 11). However, as you can see in my spreadsheet. The data usage for that particular device (Photon_B5 Turtles) is hanging around ~50,000 Data Operations a day no matter what I change. Do you see anything in my code that could be causing those data operations other than the particle.variables and particle.function that I removed? Is there a way to generate a more detailed report than just by devices or products?

The following incur a data operation:

  • Publish, and you don’t have any
  • Subscribed events that you receive, and you don’t have any
  • Function calls, depends on how often the function is called from the cloud
  • Variables, depends on how often the value is retrieved from the cloud
  • Manual requests for things like time synchronization, sending vitals, etc.

Your normal usage should be dependent mostly on the number of times you call the functions or retrieve variables, but since the data operations continued to go up when you commented out those lines, that only leaves one thing.

Is the device continuously rebooting? It’s unnecessary to synchronize the time at boot because it happens automatically, and you’re not charged for the automatic one. But if you do it manually, you are, so remove that code.

Monitoring the event log in the console for events from the device, or the serial monitor locally, may help track that down.

Unless you need accuracy within a few seconds, you don’t need to synchronize the clock every day, either. I have dozens of Photons using no data operations. I have code that reboots the device once a week. I do that so the clock gets synchronized, and in case I have a memory leak so the memory will be freed.

1 Like

Hi rickkas7, thank you for the reply.

The device is not restarting as far as I know. I would see the lights turn on and off constantly if it was, so I think it’s safe to assume that rebooting is not the issue. I was thinking the time sync code could be the issue as well. However, here is the code for my other Photon, listed as Photon_A4 FrogLightYellow in the spreadsheet in my first post:

#define ONE_DAY_MILLIS (24 * 60 * 60 * 1000)
unsigned long lastSync = millis();//variable for syncing clock to Particle cloud 

#define timeOnSummer       6   //start fading at 6, fully on by 7
#define timeOffSummer      19  //start fading at 7, off by 8       *13 hours of full light
#define timeOnSpringFall   7   //start fading at 7, fully on by 8
#define timeOffSpringFall  18  //start fading at 6, off by 7       *11 hours of full light
#define timeOnWinter       8   //start fading at 8, fully on by 9
#define timeOffWinter      17  //start fading at 5, off by 6       *8 hours of full light

int pwmVal;
int pwmPin = D0; 
int counter = 0;
bool debug = 0;

  
// --------------------------------------------------------------------
void setup() 
{ 
  pinMode(pwmPin, OUTPUT);
  analogWrite(pwmPin, 0);
  
  if(debug)  
  Serial.begin(9600);
  
  RGB.control(true);
  RGB.color(0, 0, 0);//Turn RGB LED Off
  
  Time.zone(-6);//Set Time zone to MDT
  Particle.syncTime();

  Particle.variable("pwmVal", pwmVal);

  checkDateTime();
  
  if(debug)
  Serial.println(Time.timeStr());
}

// --------------------------------------------------------------------
void loop() 
{
  checkDateTime();
  
  // DST adjstment
  //bool daylightSavings = IsDST(Time.day(), Time.month(), Time.weekday());
  //Time.zone(daylightSavings? -6 : -7);

  if (millis() - lastSync > ONE_DAY_MILLIS) {
    // Request time synchronization from the Particle Device Cloud
    Particle.syncTime();
    lastSync = millis();
  }
  
}
////////////////////////////////////////////////////////////////////////////////
void checkDateTime()
{
  counter++;
  if(counter == 6000)//check every ~30 seconds
  {
    int month = Time.month();
    
    if(month == 12 || month == 1 || month == 2)
    winter();//if == the months above, run the winter routine
    
    if(month == 3 || month == 4 || month == 5 || month == 9 || month == 10 || month == 11)
    springFall();//if == the months above, run the spring/fall routine
    
    if(month == 6 || month == 7 || month == 8)
    summer();//if == the months above, run the summer routine

    if(debug)
    Serial.println(Time.timeStr());

    counter = 0;
  }
}
////////////////////////////////////////////////////////////////////////////////
void summer()
{
  //Fade On Gradually   
  if(Time.hour() == timeOnSummer)
  {
    pwmVal = map(Time.minute(), 1, 59, 1, 255);
    analogWrite(pwmPin, pwmVal);
  }
    
  //Fully On
  else if(Time.hour() > timeOnSummer && Time.hour() < timeOffSummer)
  {
    pwmVal = 255;
    analogWrite(pwmPin, pwmVal);
  }
  //Fade Off Gradually
  else if(Time.hour() == timeOffSummer)
  {
    pwmVal = map(Time.minute(), 1, 59, 255, 1);
    analogWrite(pwmPin, pwmVal);
  }
  //Fully Off
  else
  {
    pwmVal = 0;
    analogWrite(pwmPin, pwmVal);
  }
}
////////////////////////////////////////////////////////////////////////////////
void winter()
{
  //Fade On Gradually   
  if(Time.hour() == timeOnWinter)
  {
    pwmVal = map(Time.minute(), 1, 59, 1, 255);
    analogWrite(pwmPin, pwmVal);
  }
    
  //Fully On
  else if(Time.hour() > timeOnWinter && Time.hour() < timeOffWinter)
  {
    pwmVal = 255;
    analogWrite(pwmPin, pwmVal);
  }
  //Fade Off Gradually
  else if(Time.hour() == timeOffWinter)
  {
    pwmVal = map(Time.minute(), 1, 59, 255, 1);
    analogWrite(pwmPin, pwmVal);
  }
  //Fully Off
  else
  {
    pwmVal = 0;
    analogWrite(pwmPin, pwmVal);
  }
}
////////////////////////////////////////////////////////////////////////////////
void springFall()
{
  //Fade On Gradually   
  if(Time.hour() == timeOnSpringFall)
  {
    pwmVal = map(Time.minute(), 1, 59, 1, 255);
    analogWrite(pwmPin, pwmVal);
  }
    
  //Fully On
  else if(Time.hour() > timeOnSpringFall && Time.hour() < timeOffSpringFall)
  {
    pwmVal = 255;
    analogWrite(pwmPin, pwmVal);
  }
  //Fade Off Gradually
  else if(Time.hour() == timeOffSpringFall)
  {
    pwmVal = map(Time.minute(), 1, 59, 255, 1);
    analogWrite(pwmPin, pwmVal);
  }
  //Fully Off
  else
  {
    pwmVal = 0;
    analogWrite(pwmPin, pwmVal);
  }
}
/////////////////////////////////////////////////////////////
bool IsDST(int dayOfMonth, int month, int dayOfWeek)
{
  if (month < 3 || month > 11) 
  {
    return false;
  }
  if (month > 3 && month < 11) 
  {
    return true;
  }

  int previousSunday = dayOfMonth - (dayOfWeek - 1);
  if (month == 3)
  {
    return previousSunday >= 8;
  }
  return previousSunday <= 0;
}

It uses almost the exact same code to sync the time once a day, and as you can see from the spreadsheet, it is not incurring any data operations to do so. Do you see any differences in the two sketches that could explain why one Photon is at 50,000 operations a day and the other is at 0?

I downloaded a usage report for the past 6 months and discovered that this Photon (Photon_B5 Turtles) was pulling about 50,000 daily operations before I setup the Node Red server. So I’m convinced that it is something with the code and not how frequently I am pinging any variables or functions in Node Red.

The only difference in the code for the Photon_B5 Turtles is that I’m checking the temperature every 10 seconds. I’m going to try and decrease that rate to every 15 min. But I’m curious if using millis() causes a data operation that goes against my monthly usage rate?

#define TIMER_LENGTH 10000// ms 

  if((millis() - tempCheckTimer) > TIMER_LENGTH)
  {
    getTemp();
    tempCheckTimer = millis();
  }

I don’t see why it would since all of that is happening locally and shouldn’t need to reach out to the cloud.

The only other difference is the Dallas DS18B20 temp sensor. Does including external libraries in a sketch count against monthly data usage?

Thank you for your time.

I’d probably just comment out the time sync code temporarily as that’s most likely the cause. If the data operations are still increasing, then some more digging will be necessary.

Assuming the library doesn’t do a publish, it won’t generate data operations. That’s where checking the Event Log in the console can help, in case something is publishing unexpectedly.

This is not the bug that is causing data operations, but it is a bug:

long tempCheckTimer = 0;

That needs to be unsigned long. As it is, after about 25 days tempCheckTimer will go negative and getTemp() will be called on every loop. That should not generate data operations, but it’s still not right.

Ok, thank you for the information. I will comment out that line and see if that decreases the data usage. I’ll check back with the outcome.

Hi @rickkas7

The verdict is in after yesterday’s changes, and it looks like commenting out the Particle.synctime has had no effect on my data usage. I changed the code yesterday afternoon, but as you can see there are no changes in the amount of data that was “consumed” yesterday. It’s still siting at ~50000 operations and it in fact consumed more data yesterday that any other day this week.

I am only 8 days into this billing period and yet I have already somehow consumed 35% of my monthly allotted data with just two devices.

Screen Shot 2022-09-16 at 8.30.02 AM

What else could possibly be eating up the data?!?

Did you check the event log for the device to see if it’s generating any published events?

And how often are the variables queried from the Pi? Each variable request will use a data operation. If you query both variables, changing to a data format like comma-separated or JSON and putting all of the data into a single variable would reduce the number of data operations.

I have been watching the log. The only time anything prints in there is when I reset the device or upload new firmware. Other than that, there are no events showing up in the log. It just sits saying waiting for events. I am not publishing in my code, so I do not expect to see anything to show up there.

As of right now, I am not querying anything on that Photon from the Pi. I removed all the particle.variables and particle.functions to see if the data consumption was reduced at all. But so far no changes to the code or removal of any particle.____() calls has had any effect on data usage.

Do the time related functions possibly have anything to do with this (ie Time.hour(), Time.minute())? Does calling a time function use data from the cloud?

That is mysterious!

The Time functions like .hour() and .now() are all entirely on-device and don’t use data operations.

Could you DM me the Device ID of the device without the functions and variables?

I will DM you that shortly. Here is a snapshot of my device’s data use back in June of this year, well before I had set the Pi up with Node Red and got the third Photon back up and running. As you can see, it was consuming the same amount of data even before the Pi was in use. Thus, I’m convinced that the code is somehow consuming the data and not Node Red.

Turns out it’s 100% variable requests. It appears that even though you removed the variable handler from the code, the request is still counted as a data operation. Basically, every variable request is counted, even if it ultimately fails to get the data from the device. I forgot that it works that way.

2 Likes

Can you see where the variable requests are coming from?

I suspected this might be the issue, so I had disconnected the timestamp from the variables in Node Red to hopefully prevent requests from being made.

And, at one point I had even deleted the particle blocks from the Node Red Flow altogether.

Neither seemed to affect the data. And, as pointed out in my last post. The variable requests were still coming in to the Photon before the Node Red Pi was ever setup.

I don’t have any other devices that are requesting variables or functions other than the Pi.

Is it possible that someone else is pinging my device?

I’ve been following this thread and am somewhat disconcerted that requests for non-existent variables (and probably function calls likewise) are counted as the cloud should be well aware that there would be absolutely no point to even try to communicate with the device against a non-existent endpoint.

So strictly speaking I’d not see this as failing to get the data from the device but attempting a futile endeavor (which could be ignored by the cloud or be treated as a rogue requester and be throttled and possibly blacklisted).

4 Likes

Ok, the someone else turned out to be me. I had forgot that I had an instance of Node Red running on a personal website. Turns out it was requesting some of the variables from both Photons that have the high data rate every 5 seconds. That has to be the culprit. I deleted that node red flow, so I’m assuming that will drop the data being consumed significantly. Thank you very much for the help @rickkas7!

I agree with @ScruffR, that a request to a variable that doesn’t exist should not count against the data usage.

5 Likes

Not to be too critical of Particle, but I think this also highlights just how limited the “developer/home tinkerer” plan is. While I understand limiting the cellular in such a way, it is insane imo to have this set of limits on the wifi side. An individual with a handful of photons should never come even close to a reasonable limit just automating things around the house. Personally, I was extremely disappointed when Particle announced these changes and unfortunately had to simply move away from the platform as many others did since there was no reasonable “dev/tinkerer” option even with payment. When you compare this with the alternatives (ex: cost with AWS services @ $2-4/month), it can be quite frustrating.

2 Likes