Best practices for Spark Variable update frequency?

Hi,

I have built a temperature sensor that pushes data to the cloud including LED display status (on/off), temp and uptime. It works fine and I currently have it setup where all this data is put into one Spark variable which is parsed by Python.

My question comes down to Spark variable update frequency.Part of the design allows for the remote control of the LED and the page when loading shows whether it is on or off. It is nice to have real time updates so that the page immediately reflects a change when the LED is turned on or off. In order to achieve, this the spark variable needs to update instantly (e.g. with every run through the sketch.) This works fine, but then I wonder if this is abusing the cloud too much since I am theoretically send cloud updates more than once a second.

Are there are any best practices about frequency of Spark Variable updates? Will I cause problems if I keep it constantly updated versus updating at a specific interval? How do other folks handle this?

Thank you,

@JL_678, the rule is one cloud request per second with a burst up to 4/sec. If you sustain higher than 1/sec then the cloud will actually slow its response accordingly. If you keep it to 1/sec you will not have problems. :smiley:

2 Likes

To be clear, If you're making data available via Spark.variable() then this isn't pushed to the cloud, but rather it's pulled each time the variable value is requested. So this isn't pushing, but rather pulling!

You can change the value of the variable as often as you like in the sketch - the cloud is involved only when the variable value is requested. There's no enforced limit to how often you can request the variable from the cloud.

3 Likes

@mdma, I just learned something! :relaxed:

1 Like

Wow, @mdma this is really interesting. So in fact I can update the variable in the sketch as much as I want as long as I don’t request it too much? Since requests are driven from a web page, I will have no issues updating every clock cycle since requests are way under the limit.

1 Like

@JL_678, what @mdma said is that you can update in your sketch as often as you like AND request from the cloud as often as you like as there are no enforced limits on those requests!

Yes, I agree. However, for me, the biggest a-ha was the “pull model.” I had always assumed that Spark boards pushed data to the cloud whenever a Spark Variable was updated. Hence millisecond variable updates would result in increased network traffic and run against update limits. Clearly I was wrong and this clarification really helps.

In my code, there is clearly no downside in updating the Spark Variable in every loop because the data is only accessed infrequently. (e.g. when the related web page loads or is refreshed.)

1 Like

Although it’s true that there’s no network penalty for updating every cycle on the board, you should ask yourself whether or not it’s necessary. The DHT22 temp/humi sensor, for example, needs 2 seconds to measure. Depending on what sensors you’re using, it might have a negative effect if you try to update it too frequently.
If you need a push system, you could use the Particle.publish(). A motion sensor is an excellent example case for this. Rather than polling the board whether or not motion has been seen, the board will notify you immediately, and only if something happened. If you want the push type of interaction, but the 63 bytes of the publish() don’t suffice, you could make the webpage listen for an event, and upon receiving one, fetch the matching variable. This way you have the instantaneous benefits of a pushing system, with the greater capacity of variables.

1 Like

Hi,

Yes, I am familiar with the limitations that you describe although I am using a MPL3115A2 vs a DHT22 which allows for 1 second temperature measurements. However, to your point, I do not need to be measuring that rapidly and so I only samples the temperature every 10 seconds.

The real issue from my view is updating LED Status when turning on/off a LED remotely. When the page loads, it polls a spark variable to get whether the LED is currently on or off. If I sample less frequently then the LED status (and as a result the web page) will not change its state for up to 10 seconds. A continuous sample will resolve the situation.

Can you suggest a better way to do this? It does work fine.

Hi @JL_678

Have you seen my tutorial on Spark functions and variables here?

Using a very simple reload window interval of 1 second, I was able to get real-time control over a servo with a web page. You couldn't fly a helicopter with this but you could make a lot of control systems work just fine.

You could update the temperature every 10 seconds and other sensors every second and still combine the data into one string to send back to the web.

1 Like

If you want to reduce the sampling of the variable, you could only poll the variable on page load. Once the page is loaded, use Spark.publish() to post any changes of the variable and use SSE on the page to monitor those publishes. Just to be safe, you might want to poll the variable every 60 seconds or so (or less if you're extra paranoid).

@bko wrote an excellent tutorial on Spark.publish() and SSE:

3 Likes

I'm not entirely sure I understand the setup; You've got an LED hooked up to your Core, which you can toggle over the web? Or is it an LED which you monitor with your Core? Getting the LED status on the Core can be done as often as you like. You could make a bool changed variable, which you toggle if you notice the LED is toggled.
You can then update the variable accordingly, to contain the current state. To make sure the Status is displayed on the website correctly, within the polling time of 10s, you could use the Particle.publish(). If you're using the website to toggle the LED in the first place, you could have the return value indicate what the current State is. If you're toggling the LED from somewhere else, you could have the page subscribed to a state changed publish event. If the aforementioned bool changed is true, then publish an event indicating the LED State, after which you set the bool changed = false again.

tl;dr: do what the two guys above said, specifically @wgbartley SSE setup would be useful here.


@wgbartley, @bko: stop being so fast! :cry:

1 Like

Oh boy, you guys are giving me a ton to think about. Funny enough, the uptime parsing code I am using actually came from @bko’s tutorial on publishing uptime. The only difference is that things changed from spark.publish to a spark.variable along the way. I need to go back and look at how I handle this.

To be clear, I have built is a temperature sensor that publishes the temperature data on a web page and it has an attached LED display to show the temperature locally. The web page has a simple form that allows you to turn the temperature display on or off and it also shows you whether the LED is currently on or off. It does the latter by including a LED status indicator inside my comma delimited Spark Variable. It is the LED status indicator that I want to get updated in real time so that when you turn the LED on/off that the web page properly shows LED status when the page is reloaded after form submission.

I think that the key takeaway is that I need to explore spark.publish for this. Truthfully, I have not and instead have relied on a comma delimited spark variable which has worked well. When I say comma delimited, I mean that I am concatenating multiple sketch variables into one spark string variable and then using Python to parse out each variable on the web server. This is much faster because it reduces 3 REST API calls to one.

Okay, I think I get the picture now :wink:
Some random ideas that popped up, since I've got a similar project (LED lightning control). I've made a page with HTML/JavaScript, which allows me to control an RGB strip over the web. But since I can control them from multiple devices at once, I'd still like to know the current color.
In order to do that, I've made a function which publishes my current color values using SSEs. When I initially load the page it will call that function, and the Core will trigger the SSE. I've subscribed my webpage to those events, so as soon as it receives it, it parses the colors, and shows them accordingly. The same thing happens when I change a color from the web: a function gets called to change the colors, and publish them afterwards. Since all opened webpages are subscribed to these events, they will all update, keeping them in sync with the actual color.

I guess the reloading is necessary since you're using a form? If it has to reload anyhow, there's a really easy fix: make the function on the Core update the State. Once the page reloads, it will fetch the new variable, and the State will have changed accordingly.
I personally like 'Single page applications' better, since they don't require reloading, and are thus MUCH faster. If what you've described is the only thing it needs to do, you don't need Python, of forms. Some basic HTML/JavaScript should suffice, and might actually be faster. Also, it will run anywhere, since JavaScript is the standard scripting language in browsers.


Okay: I've come up with some (absolutely horrible) pseudocode, hoping to give you a general idea of what I'm trying to explain. It actually doesn't use any variables, but that can be added easily, should you require the extra data.

Particle sketch:

int variable;
bool ledStatus = true;

setup(){
  //pinSetup, etc.
  Particle.function(measureSensor);
  Particle.function(toggleLED);
}

loop(){
  measureSensor();
  delay(10000);
}

int measureSensor(String message){

  if (timeSinceLastMeasurement > minimumTimeRequired){	//You could make this time variable using the function argument.
    variable = sensorValue(); 	//your measurements with some library
  }
  Particle.publish(variable);
}

int toggleLED(String message){
  //Toggle LED off;
  ledStatus = !ledStatus;
  Particle.publish(ledStatus);
  return ledStatus ? 1 : 0;
}

HTML stuff:

<html>
  <head>
    <!-- include all libraries -->
  </head>
  <body>
    <text id="temperature">
      <!-- temperature will show up here -->
    </text>
    <text id="ledState">
      <!-- Led State will show up here -->
    </text>
    <button id="updateTemp"> click here to update the temperature </button>
    <button id="toggleLED"> click here to toggle the LED display </button>
    <script>
      if (pageLoaded){
      	Particle.subscribe(temperature);
      	Particle.subscribe(ledState);
      	Particle.function(measureSensor);
      }
      
      temperature(data){	//this will get triggered on a Particle.publish();
      	if (data){
      		updateTextWithData(temperature);
      	}
      }
      
      ledState(data){		//this will get triggered on a Particle.publish();
      	if (data){
      		updateTextWithData(ledState);
      	}
      }
      
      if (buttonPushed(id)){
      	if (id == updateTemp){
      		Particle.function(measureSensor);
      	}
      	if (id == toggleLED){
      		Particle.function(toggleLED);
      	}
      }
    </script>			
  </body>
</html>

Again, the code above is horrible, it's just there to visualise the 'flow' of how I'd imagine it to work.
Let me know if you need any further explanation, or if I can help :smile:

2 Likes

@Moors7 - This is awesome! Thank you. My challenge is that I am illiterate when it comes to Javascript hence my use of Python. (I also happen to have a few Raspberry Pis floating around which further reinforces the use of Python.)

All that said, it does look like JS would be better especially since a page reload is not required as you point out. I will spend some time with your code, and it looks like I need to go learn Javascript.

Thank you again.

Okay, I might have been a bit too enthusiastic… I was like:“how hard can it be? Just write some lines of code. piece of cake…”
It’s 06:00 now, birds are chirping, and it’s light outside…

Core code:

unsigned long lastMeasurement;
int ledState = 0;
unsigned long minimumTime = 2000;
long previousMillis = 0;
int sensorData = 0;
int led = D7; 
unsigned long currentMillis = 0;

void setup() {
  lastMeasurement = millis();
  pinMode(led, OUTPUT);
  Spark.function("getTemp", getTemp);
  Spark.function("toggleLed", toggleLed);
}

void loop() {
  currentMillis = millis();

  // non-blocking delay
  if(currentMillis - previousMillis > 10000) { 
    // save the last time you blinked the LED 
    previousMillis = currentMillis;   

    getTemp("bla");
  }
}


int getTemp(String message){      
  //checks if there's at least minimumTime between measurements
  if (millis() - lastMeasurement > minimumTime){

    // implement your measurement here.

    // set 'sensorData' here

      // -------------------------------

    RGB.control(true);  
    RGB.color(0,255,0);
    delay(100);
    RGB.color(0,0,0);
    delay(100);
    RGB.color(0,255,0);
    delay(100);
    RGB.control(false);

    lastMeasurement = millis();

    Spark.publish("temperature", String(sensorData));

    // Sneaky trick to get the ledState from the temperature function.
    // Since the getTemp is requested when loading the page, this will
    // fetch the ledState without having to do an additional request,
    // or create a separate function.
    return ledState; 
  }
  // If you measure too quickly, return -1
  else {
    RGB.control(true);  
    RGB.color(255,0,0);
    delay(100);
    RGB.color(0,0,0);
    delay(100);
    RGB.color(255,0,0);
    delay(100);
    RGB.control(false);
    return -1;
  }
}

int toggleLed(String command){
  if (ledState == 1){
    digitalWrite(led, LOW);
    ledState = 0;
  } 
  else {
    digitalWrite(led, HIGH);
    ledState = 1;
  }

  Spark.publish("ledState", String(ledState));
  return ledState;
}

HTML/JavaScript code:

<html>
  <head>
    <title>Spark temperature</title>
  </head>
  <script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>
  <script src="http://cdn.jsdelivr.net/sparkjs/0.4.2/spark.min.js" type="text/javascript" charset="utf-8"></script>  
  <!-- Latest compiled and minified CSS -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
  <!-- Latest compiled and minified JavaScript -->
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
  <body>
    <div class="container">
      Temperature:
      <div class="input-group input-group-sm"> 
        <input type="text" class="form-control" placeholder="No data received yet" readonly id="temperature">
        <span class="input-group-btn">
        <button class="btn btn-default" type="button" onclick="getTemp()">get!</button>
        </span>
      </div>
      Led state:
      <div class="input-group input-group-sm"> 
        <input type="text" class="form-control" placeholder="No data received yet" readonly id="ledState">
        <span class="input-group-btn">
        <button class="btn btn-default" type="button" onclick="toggleLed()">toggle!</button>
        </span>
      </div>
    </div>
    <script>
      var coreID = "53ff6d065000035140000687";							// fill in your Device ID
      var accessToken = "e7c24c4aaaadca0d258ad27aaaa946970fa2193e"		// fill in your accesstokem
      var relevantData;													// Parsed data from the SSE
      var temperatureText = document.getElementById('temperature');		// the input text object where the text will be
      var ledStateText = document.getElementById('ledState');				// the input text object where the text will be
      
      
      
      var openStream = function() {											// 'starts' the function
          //Get your event stream
          var stream = spark.getEventStream(false, coreID, function(data) {	// open stream, listening to a specific device
      	      console.log("Event: " + JSON.stringify(data));				// log stuff to console
      	      relevantData = data.data;										// get the relevant data from the JSON			      
      	      if (data.name == 'temperature'){								// check for the right name		
      	      	temperatureText.value = relevantData;						// update the temperature box with the value from the JSON
      	      }
      	      else if (data.name == 'ledState'){							// check for the right name		
      	      	ledDisplay(relevantData);
      	      }
      	      else {
      	      	return;
      	      }
      	    });
      	
      
          jQuery(stream).on('end', function() {							// ckeck if the stream ends (ya know, shit happens...)
              console.log("ended!  re-opening in 3 seconds...");			// print stuff to console
              setTimeout(openStream, 3 * 1000);							// open the stream again after 3s
          });
      };
      
      
      function ledDisplay(relevantData){								// function to display the ledState
      	if (relevantData == 1 || relevantData == 'true'){			// check for the state
           		ledStateText.value = "on";								// display state in ledState box
           		ledStateText.style.backgroundColor="rgb(0,255,0)";				// change ledState box background color
           	}
           	if (relevantData == 0 || relevantData == "false"){			// check for the state		
           		ledStateText.value = "off";								// display state in ledState box
           		ledStateText.style.backgroundColor="red";				// change ledState box background color
           	}
      }
      
      var getTemp = function(){
      	spark.callFunction(coreID, 'getTemp', false, functionCallback);			// call the 'getTemp' function without arguments
      }
      
      var toggleLed = function(){
      	spark.callFunction(coreID, 'toggleLed', false, functionCallback);		// call the 'toggelLed' function without arguments
      }
      
      var functionCallback = function(err, data) {								// function response callback
      	if (err) {			  
      	  console.log('An error occurred while getting core attrs:', err);		// print error message to console
      	} else {
      	  console.log('Core attr retrieved successfully:', data);				// print succes message to console
      	  if (data.return_value == -1){
      	  	temperatureText.value = "please wait at least 2 seconds between measurements";
      	  }
      	  else{
      	  	ledDisplay(data.return_value);
      	  }
      	}
      };
      
      spark.login({accessToken: accessToken}, function(err, body) {		// login on Particle servers
      	console.log('API call login completed on callback:', body);		// log stuff to console (right click browser "inspect page")
      	openStream();													// call 'openStream()' function
      	getTemp();														// get initial temperature
      });
      
    </script>
  </body>
</html>

Give it a try, and let me know if it worked. If you need any help, I’ll be glad to do so once I’m (somewhat) awake again. Enjoy.

3 Likes

@Moors7 as I think about this, I have a question. @bko used a similar approach for his uptime tutorial which I implemented and one of his warnings is that the code should not be used publicly because it exposes your access code and Core ID to anyone who cares to look.

Do you worry about this with your JS code? As I think about it, wouldn’t any JS-based solution have this risk? I don’t know JS well enough to know if there is an alternative.

Python avoids the above by running the REST queries in the background and so the actual ids are not exposed.

Thank you.

If you decide to hard code your credentials in there, then yes, they are exposed. I’m not exactly sure how much more secure it is when you use Particle’s log-in modal, but it’s a lot better to start with. Basically, you log in to their servers, and you get an accesstoken in return. I’ve used that over here. I then just stored the accesstoken in LocalStorage so I wouldn’t have to enter it every time.

In general, I’m not too worried about it being exposed, as I don’t have any high-risk projects. The worst someone can do is toggle my LED lighting, and that’s hardly a disaster. In more critical applications, you’ll definitely want to look into server-side credentials. But then Node.js, also written in JavaScript, is easy to use :wink:

You could also route your JS requests through a PHP proxy if I’m not mistaken, but I’ve never really tried that. Maybe @bko or @wgbartley have some additional ideas/comments(?)

Garrett @wgbartley has a PHP proxy posted here in the forum that works if you have your own web server.

Another way to go is to write a javascript/html web page that asks the user to log in to their Particle account to get a token via the Particle API. That was too much work for the tutorial but it is quite do-able for a real app.