Using Spark.publish() with Simple JSON Data


#1

Let’s say you have looked at the Getting Started with Spark.publish() tutorial, but you need to send data that needs more processing once it gets to its destination on the web. You have a lot of choices for the data format, but one simple choice is to use JSON, the JavaScript Object Notation which let you send property names and values in an easy to read and easy parse to format.

You may have noticed in fact that the data returned by listening to the event stream from a published event is already itself a JSON with fields like “coreid” and “TTL” for time-to-live, and most importantly the “data” field that holds the data published by your Spark core.

You are going to take advantage of the fact that a JSON can contain another JSON to send formatted data from your Spark core to the cloud.

Firmware Side

The firmware we need on the core is only slightly changed from the previous tutorial. All you need to do increase the size of the string buffer to hold more data and build the data string to publish in a bit more complicated way. The JSON format uses double-quote characters around all the names of the properties. For the values, you can use any normal value you would use but strings may need some escape character to not confuse the parser–check json.org for details.

Here is your core firmware. I named it publishjson.ino but you can call it whatever you like.

// publishjson.ino -- Spark Publishing Example 
unsigned long lastTime = 0UL;
char publishString[64];

void setup() {
}

void loop() {
    unsigned long now = millis();
    //Every 15 seconds publish uptime
    if (now-lastTime>15000UL) {
        lastTime = now;
        // now is in milliseconds
        unsigned nowSec = now/1000UL;
        unsigned sec = nowSec%60;
        unsigned min = (nowSec%3600)/60;
        unsigned hours = (nowSec%86400)/3600;

        sprintf(publishString,"{\"Hours\": %u, \"Minutes\": %u, \"Seconds\": %u}",hours,min,sec);
        Spark.publish("Uptime",publishString);
    }
}

Notice that it is almost identical to the previous tutorial with just two changes: First off the publishString size has changed to 64 characters long. The second change is in the sprintf() that creates the publishString. The JSON data is surrounded by curly-braces ( {…} ) and all the names of the data fields have to be enclosed in double-quotes. How do you put double-quotes in a string for the sprintf function? Well in C/C++ you do this by using a backslash before the double-quote character, so when you sprintf something like "{\"Hours\":" what you get out is just {"Hours":.

The Javascript Web Side

Here is HTML file for the web side–it is every similar to the previous tutorial with one big exception we will talk about below. I called this Uptime2.html but you can pick any name.

Safety Note!

Because you put your access token in this file, you should never put it where other people can read it. Do not put it on a public webserver or in a public area like your public folder on Dropbox. This is private stuff just for you!

<!DOCTYPE HTML>
<html>
<body>
    <span id="uptime"></span><br>
    <span id="tstamp"></span>

    <br><br>
    <button onclick="start()">Connect</button>
 
    <script type="text/javascript">
    function start() {

        document.getElementById("uptime").innerHTML = "Waiting for data...";
        var deviceID = "<<your core id here>>";
        var accessToken = "<<your  access token here>>";
        var eventSource = new EventSource("https://api.spark.io/v1/devices/" + deviceID + "/events/?access_token=" + accessToken);

        eventSource.addEventListener('open', function(e) {
            console.log("Opened!"); },false);
         
        eventSource.addEventListener('error', function(e) {
            console.log("Errored!"); },false);
         
        eventSource.addEventListener('Uptime', function(e) {
            var rawData = JSON.parse(e.data);
            var parsedData = JSON.parse(rawData.data);
            var tempSpan = document.getElementById("uptime");
            var tsSpan   = document.getElementById("tstamp");
            tempSpan.innerHTML = "Core: " + rawData.coreid + " JSON Data: " + parsedData.Hours + ", " + parsedData.Minutes + ", " + parsedData.Seconds +
                     " (" + (parsedData.Seconds + parsedData.Minutes*60 + parsedData.Hours*3600) + " secs)";
            tempSpan.style.fontSize = "28px";
            tsSpan.innerHTML = "At timestamp " + rawData.published_at;
            tsSpan.style.fontSize = "9px";
        }, false);
    }
    </script>
</body>
</html>

The web side almost identical to the previous HTML file but you can see that there are now two JSON.parse calls. First you parse the raw data from the event. This has fields like “coreid” and “published_at” that you are using just like in the previous HTML file, but now the “data” field is not just a simple string. You call JSON.parse() on the data field of the rawData to get the three new fields for hours, minutes, and seconds. And since these are numbers now instead of strings, we can easily reassemble the time in seconds from the parts just by multiplying and adding.

OK, you are almost there. Open up Chrome or other recent browser on your PC and type this in the URL box, replacing the “<< >>” parts as to match you system:

For Windows: file:///C:/<<your path>>/Uptime2.html

For Mac: file:///users/<<your mac user name>>/<<your path>>/Uptime2.html

After you click the connect button, you should see something like this:

Go Play Some More!

Now you need to get out there and make your own JSON events. Just make sure your publishString is large enough to hold your data. In the coming days, the Spark team will add Arduino String objects which are dynamically allocated and you won’t have to worry so much about this. There will always be size limits on platforms like Spark and Arduino, just be aware that you cannot publish huge amounts of data.

You can also save your Uptime2.html file to a private folder on a service like Dropbox. Then on your phone, you can use the Dropbox app to open this file as a web page and see how your core is doing. Remember, do not put the Uptime2.html file in a public spot!


Help! How to POST or use Webhook
Particle.publish and JSON escaped data
Reading sensor data to a web app
Unexpected sprintf() result from GET request
Getting Spark Variables with 1 Json Call?
Spark function and a loop
Wireless communication with ID-12LA RFID reader
Select json.result
Integer to string (Spark.publish)
Tutorial for a simple web app
Reading from the cloud - simple
Spark.getVariable vs Spark.subscribe
Peristaltic Doser - Doser set amount over 24hr period
Form for "control LEDs over the 'net" example
Photointerruptor for fermenter not tallying and publishing
Particle.variable(...) limitations and best approach to read multiple variables from different electron boards
Help! How to POST or use Webhook
Tutorial: Getting Started with Spark.publish()
Sprint 7: Particle.publish() released! Let's build a cloud-connected motion detector
Transfer an array using Spark.variable
[Solved] Spark Publish, NodeJS and EventSource
Using the Tinker API
Tutorial: Spark Variable and Function on One Web Page
Unable to create valid JSON in Spark.publish() [SOLVED]
Spark.publish() parameters being passed?
iOS subscribing to published event's by the Spark Core?
#2

This is a really smart idea. I like it!

Great tutorials! I just need some time to follow along and try this out… it’s been a busy weekend.


#3

BKO… This was excellent (ONCE!) THank you very much for an enjoyable evening. However…

It worked flawlessly ONCE; got the time response… but then no more. Can you help??? Is it possible the cloud is “locked”

Details:
Created Spark and HTM code. Loaded Locally via USB.
Recieved time YEA!..ONCE.
Could not get another resonse; tried a lot of things. Went to sleep.
Started again this morning, hoping time heals…still no response.
Reload core… no change.
Opened Crome console. Nothing displayed there… ???
Modified htm to have a bad ID… Console properly shows “Errored”.
Fix to correct ID… nothing shows in Console, not even a “Opened”.
Have I got the cloud “locked” for my core???


#4

Hi @LZHenry

There are some things you can check.

  • First off, your core needs to be running in the breathing cyan state on the LED.

  • Second, try reloading the web page and hitting connect again (just once). If you can bring up the Javascript debug console like you did to see “Errored” there is another tab labelled “Network” that may show if data is flowing–look for a yellow bar in chrome.

  • If you are not using Chrome, try using that browser. I have tried Chrome, Safari, IE and Firefox, but Firefox has problems sometimes.

  • Lots of people are running this example right now–you can tune in to their streams by taking the “devices/” and deviceID out of the URL for the events, just leaving the accessToken.

  • There rate limits now on fast you can send events since everybody sending at full rate will kill the cloud. The original above sends every 15 seconds but I see some folks are sending events every second or two events every 5 seconds. If you made a change like that, try it back the original way.

  • If you can run curl, which is a command line program that fetches web pages, you can check if your problems are in your core or in your browser. Try this:

https://api.spark.io/v1/events/Uptime/?access_token=<<your token here>>

If fact if you are publishing public events, which is the default, and you tell me your core’s device ID, I can see if you core is publishing too.


#5

I had the same problem running IE 11 and no problem whatsoever using Chrome. I only use Chrome now! :smile:


#6

My tests were with IE 9 on Win7.


#7

Tried Crome for all these tests… only the SINGLE response.
Then Tried FireFox… still no response.

I am confused because Crome Console does not even show the connect or error???

I will look for the ID and poste it soon if it is with me here.


#8

Here is the core ID… The code running in the core is exactly like your example.
50ff6a065067545650440487


#9

LZHenry, I suspect you need to look at your .html file. If you are not even getting the connect button then it sounds like there is a problem there.


#10

I “claim” that it has not changed since the first GOOD response… I looked this morning at both files, and will again tonight start debugging… thanks .


#11

Your core is definitely broadcasting and has indeed been set to send every 5 seconds instead of every 15. You have a browser problem from the looks of things. IE has a bunch of active-x controls that also control the HTML5 but like @peekay123 said, I avoid using IE these days.

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 11, \"Seconds\": 52}","ttl":"60","published_at":"2014-03-27T17:40:55.794Z","coreid":"50ff6a065067545650440487"}

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 11, \"Seconds\": 57}","ttl":"60","published_at":"2014-03-27T17:41:00.796Z","coreid":"50ff6a065067545650440487"}

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 12, \"Seconds\": 2}","ttl":"60","published_at":"2014-03-27T17:41:05.799Z","coreid":"50ff6a065067545650440487"}

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 12, \"Seconds\": 7}","ttl":"60","published_at":"2014-03-27T17:41:10.801Z","coreid":"50ff6a065067545650440487"}

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 12, \"Seconds\": 12}","ttl":"60","published_at":"2014-03-27T17:41:15.801Z","coreid":"50ff6a065067545650440487"}

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 12, \"Seconds\": 17}","ttl":"60","published_at":"2014-03-27T17:41:20.806Z","coreid":"50ff6a065067545650440487"}

event: Uptime
data: {"data":"{\"Hours\": 6, \"Minutes\": 12, \"Seconds\": 22}","ttl":"60","published_at":"2014-03-27T17:41:25.807Z","coreid":"50ff6a065067545650440487"}

#12

Yes… 5 seconds… forgot that last night as I was frustrated.

Back to the basics… as PeeKay said, look at the HTM… now see this error…

SCRIPT5009: ‘EventSource’ is undefined
Register1.html, line 15 character 9

Back to afernoon tasks, and off to debug tonight. Sorry for the simple confusion… thanks for the help.


#13

That’s an IE error–I think you will be less frustrated with Chrome.

Good luck tonight!


#14

Still Stumped… need help with the debugging tools.

Still attempting to communicate with my core, and still confused. However, a warning, as I have done too many things at once before getting stable.

  • Loaded Version 3 libraries. (hoping it would help)
  • Fixed the reporting time to be 15 seconds.
  • Now I have breathing Cyan but flashing RED that I need to investigate.

For the GET commands, things are still getting hung. From these tools it appears that the GET is proper and getting out of my house. The io cloud should answer or this should timeout. WHY IS THERE NO TIMEOUT?

From FireFox Browser Console

(Sorry, I do not know how to post screen captures)

POST https://community.spark.io/message-bus/733b2014682744bea415b23cd07986ed/poll [HTTP/1.1 200 OK 66ms]
19:55:26.315
POST https://community.spark.io/message-bus/733b2014682744bea415b23cd07986ed/poll [HTTP/1.1 200 OK 70ms]
19:55:27.403
***GET https:***//api.spark.io/v1/devices/50 xxxx my id/events/
19:55:38.315
POST https://community.spark.io/message-bus/733b2014682744bea415b23cd07986ed/poll [HTTP/1.1 200 OK 66ms]

(AND question #2… what is the spark.io POSTing to me every 15 seconds??? This POST is continual WITHOUT any devices connected)

FROM CROME NETWORK output

?access_token=6xxxmy tokenxxxxx
api.spark.io/v1/devices/50xxx my id/events
GET (pending) Other 13 B 0 B Pending

Thanks for reading… I am off to seek the explanation for the flashing RED lights. I think it is 4-5-1… off I go.


#15

Hi @LZHenry

The POST are to community.spark.io so that is this forum open in the same browser. The GET is the one trying to get the cloud to connect.

The red flashing lights are panic codes, SOS followed by a number of flashes followed by SOS and then a restart. The number of flashes tell you what the panic was but out-of-heap is the most likely (8 flashes).

I think you should try to debug one thing at time. Can you cut and past the exact code above and run it on the core and work on the web part for now? I do not see your core broadcasting events at this time (11pm EDT or GMT-4).

In Chrome, you could also try the view source URL to look at unformatted events. Just put this in the URL box and hit return with your actual token in place of your_access_token.

view-source:https://api.spark.io/v1/events/?access_token=your_access_token

You can add the devices and device ID to this URL to get just your core.


#16

Hi @bko, I slightly changed your code so it will publish my firmware version (only once during the live time of the code) instead of the uptime. First I did it according your previous example and that went ok. Then I changed it to this example and get an error.

Here is the part of the code:

                        char line[64];   

 //			sprintf(line,"%s",versionStr);
    			sprintf(line,"{\"Version\": %s}",versionStr);
    			Serial.print(line);
    			Spark.publish("Version",line);

The print out shows the following:

{"Version": 0.0.6.9}

The relevant part of the HTML code is here:

    var parsedData = JSON.parse(rawData.data);
    var tempSpan = document.getElementById("version");
    var tsSpan   = document.getElementById("tstamp");
    //tempSpan.innerHTML = "Core: " + rawData.coreid + " Version: " + rawData.data;
	tempSpan.innerHTML = "Core: " + rawData.coreid + " Version: " + rawData.parsedData.Version;
    tempSpan.style.fontSize = "28px";

I use Chrome as browser and that returned the following error:

Opened! (index):19
Uncaught SyntaxError: Unexpected token . (index):26
(anonymous function) (index):26
Errored! 

If I look via curl ($ curl https://api.spark.io/v1/events/Version/?access_token=) I get the following result back:

event: Version
data: {"data":"{\"Version\": 0.0.6.9}","ttl":"60","published_at":"2014-03-28T12:36:39.962Z","coreid":"48ff70065067555042571587"}

I have the feeling that the ‘"’ causes the problem but no idea how to solve it. Do you know what could be wrong?

Thanks,
Henk


#17

HI @nika8991

Your problem is the version with all the periods is being interpreted as a number in javascript (since it does not have double quotes around it) and a number like 0.0.6.9 is a syntax error.

If you want to send the version as a string, you need to publish it with quotes around it from the core:

sprintf(line,"{\"Version\": \"%s\"}",versionStr);

This should give you {"Version": "0.0.6.9"}


#18

Hi @bko, thanks for the quick response. I tried it but still get an error although it is an other error.

From Chrome I get:

Opened! (index):19
2
Uncaught TypeError: Cannot read property 'Version' of undefined (index):30
(anonymous function)

And with curl I see:

event: Version
data: {"data":"{\"Version\":\"0.0.6.9\"}","ttl":"60","published_at":"2014-03-28T15:44:37.932Z","coreid":"48ff70065067555042571587"} 

Thanks,
Henk


#19

Can you try changing rawData.parsedData.Version to just parsedData.Version?

tempSpan.innerHTML = "Core: " + rawData.coreid + " Version: " + parsedData.Version;

#20

Yes, that was the reason, now it works :slight_smile:

Core: 48ff70065067555042571587 Version: 0.0.6.9

Thank you,
Henk