Local server in node.js example


#1

The event publish and subscribe feature of the Particle Cloud is easy-to-use and very effective in most cases. But sometimes you need to transmit more data than is allowed for publishing events. Or maybe you need to do some specialized processing. A server on your home or office network implemented in node.js is a useful tool in some cases.

This sample implements:

  • Server discovery. The Photon uses publish to locate the server IP address and port to connect to, so you don’t need to hardcode it in your Photon firmware.
  • A HTTP POST TCP connection from the Photon to the local server, kept open for sending data in real time.
  • Authentication of the HTTP connection using a nonce (number used once).
  • A small web server so you can easily host HTML, Javascript, CSS, etc. for a web-based application.
  • A SSE (server-sent events) server to allow data to be streamed in real time into a web browser.

By using the combination of the HTTP/TCP connection and SSE, you can stream large amounts of data from a Photon in real time, right into a web browser, where it can be handled by Javascript code. Even better, any number of web browsers on your home network can connect at the same time and view the same live data, limited by the capacity of your node.js server, not the Photon.

Also, unlike using webhooks, your local server does not need to be open for incoming connections from the Internet if your Photon is also on your local network. It can stay safely behind your home firewall and you don’t need to worry about things like firewall/router port forwarding or dynamic DNS.

These are the three basic steps to connecting:

  • The Photon publishes a private devicesRequest to the Particle cloud
  • The server subscribes to these requests and responds by calling the devices function on the Photon with the server IP address, server port, and a nonce (number used once)
  • The Photon issues a HTTP POST to the server IP address and port, with an Authorization header containing the nonce. It then keeps the connection open for sending data.

In the examples below we also use another computer web browser to connect to the server. It loads HTML, CSS, and Javascript from that server, and also opens a SSE channel. This channel is hooked to the HTTP POST channel from the Photon, basically allowing the Photon to send data in real time directly to the browser, via the server.

The server is just a computer running node.js. It could be running Windows, Mac OS X or Linux. It could even be something like a Raspberry Pi.

There are two examples here:

  • livegraph, which uses a simple potentiometer to graph values
  • liveimu, which uses an accelerometer (IMU, Inertial Measurement Unit) and prints the location data to a web browser window in a scrolling table

A video of it in action:

This is what the Particle code looks like for server discovery and connection management:

#include "Particle.h"

SYSTEM_THREAD(ENABLED);

int devicesHandler(String data); // forward declaration
void sendData(void);

const unsigned long REQUEST_WAIT_MS = 10000;
const unsigned long RETRY_WAIT_MS = 30000;
const unsigned long SEND_WAIT_MS = 20;


enum State { STATE_REQUEST, STATE_REQUEST_WAIT, STATE_CONNECT, STATE_SEND_DATA, STATE_RETRY_WAIT };
State state = STATE_REQUEST;
unsigned long stateTime = 0;
IPAddress serverAddr;
int serverPort;
char nonce[34];
TCPClient client;

void setup() {
	Serial.begin(9600);
	Particle.function("devices", devicesHandler);
}

void loop() {

	switch(state) {
	case STATE_REQUEST:
		if (Particle.connected()) {
			Serial.println("sending devicesRequest");
			Particle.publish("devicesRequest", WiFi.localIP().toString().c_str(), 10, PRIVATE);
			state = STATE_REQUEST_WAIT;
			stateTime = millis();
		}
		break;

	case STATE_REQUEST_WAIT:
		if (millis() - stateTime >= REQUEST_WAIT_MS) {
			state = STATE_RETRY_WAIT;
			stateTime = millis();
		}
		break;

	case STATE_CONNECT:
		if (client.connect(serverAddr, serverPort)) {
			client.println("POST /devices HTTP/1.0");
			client.printlnf("Authorization: %s", nonce);
			client.printlnf("Content-Length: 99999999");
		    client.println();
		    state = STATE_SEND_DATA;
		}
		else {
			state = STATE_RETRY_WAIT;
			stateTime = millis();
		}
		break;

	case STATE_SEND_DATA:
		// In this state, we send data until we lose the connection to the server for whatever
		// reason. We'll to the server again.
		if (!client.connected()) {
			Serial.println("server disconnected");
			client.stop();
			state = STATE_RETRY_WAIT;
			stateTime = millis();
			break;
		}

		if (millis() - stateTime >= SEND_WAIT_MS) {
			stateTime = millis();

			sendData();
		}
		break;

	case STATE_RETRY_WAIT:
		if (millis() - stateTime >= RETRY_WAIT_MS) {
			state = STATE_REQUEST;
		}
		break;
	}
}

void sendData(void) {
	// Called periodically when connected via TCP to the server to update data.
	// Unlike Particle.publish you can push a very large amount of data through this connection,
	// theoretically up to about 800 Kbytes/sec, but really you should probably shoot for something
	// lower than that, especially with the way connection is being served in the node.js server.

	// In this simple example, we just send the value of A0. It's connected to the center terminal
	// of a potentiometer whose outer terminals are connected to GND and 3V3.
	int value = analogRead(A0);

	// Use printf and manually added a \n here. The server code splits on LF only, and using println/
	// printlnf adds both a CR and LF. It's easier to parse with LF only, and it saves a byte when
	// transmitting.
	client.printf("%d\n", value);
}

// This is the handler for the Particle.function "devices"
// The server makes this function call after this device publishes a devicesRequest event.
// The server responds with an IP address and port of the server, and a nonce (number used once) for authentication.
int devicesHandler(String data) {
	Serial.printlnf("devicesHandler data=%s", data.c_str());
	int addr[4];

	if (sscanf(data, "%u.%u.%u.%u,%u,%32s", &addr[0], &addr[1], &addr[2], &addr[3], &serverPort, nonce) == 6) {
		serverAddr = IPAddress(addr[0], addr[1], addr[2], addr[3]);
		Serial.printlnf("serverAddr=%s serverPort=%u nonce=%s", serverAddr.toString().c_str(), serverPort, nonce);
		state = STATE_CONNECT;
	}
	return 0;
}

The project is here on github: https://github.com/rickkas7/localserver


TCPClient and NodeJS server
Logging Data Every 10ms - Best way to retrieve it over the web?
Photon/node.js live streaming graph web page example
How to send large file from Particle to server?
#2

What a great writeup! Thanks for putting this together, @rickkas7!!!


#3

Hi,

I am new to the Particle IDE/APIs. I am trying to implement this example to learn for another application (streaming ADC data from Photon to a remote database) and am having some trouble after going through your GitHub steps.

I have flashed livegraph.ino to the Photon.

I am using the node.js spark-server. Is this the correct setup? The Photon core connects (‘Core online!’) but then I get a bunch of ‘Coap Errors’.

When I run the livegraph.js, this is the output:

localserver-master Mukatome$ node livegraph.js --login p@g.c h
cloud logging in
failed to log in HTTP error 400 from https://api.particle.io/oauth/token - Method must be POST with application/x-www-form-urlencoded encoding
no access token, not starting listener
found address en0: 192.168.0.34
server address 192.168.0.34

Am I on the right path? Am I missing something?

Does this work for a local server only or can I (eventually) have this connect to an outside server (e.g. Amazon AWS)?

Any help would be greatly appreciated!


#4

You wouldn’t need the Spark Local Cloud server, unless you want to use the Particle.xxxxx() functions in your own private network without internet connection.
It’s nice to know about it tho’, but might be a bit steep for the beginning, since it involves exchanging the keys and alter target addresses on the device.

Just focus on the samples provided by Rick.


#5

Thanks @ScruffR!

I finally got the local spark-server running. Apparently, having the node v0.10 version is very important heh heh. No more ‘Coap Errors’.

I also was able to get @rickkas7 live graphing post working (very cool), but still no love on this localserver example… same ‘no access token’ error.


#6

I’m not sure why that’s happening, but if you look in the settings folder (in the same folder as livegraph.js, liveimu.js, etc.) there should be a file called accessToken. If there isn’t one, create one and copy an access token (like the one on the settings page at build.particle.io) into that file. It must be surrounded by double quotes on one line, like:

"aaaa6d4ec60ebbbbfbdbccc54366ddddd499ffff"

#7

@rickkas7
I get the same

cloud logging in
failed to log in HTTP error 400 from https://api.particle.io/oauth/token - Method must be POST with application/x-www-form-urlencoded encoding
no access token, not starting listener

when trying node livegraph.js --login myemail mypassword

At first I suspected the special characters in my password but hard coding my username and password into the cloud.js line:

	cloud.particle.login({username: argv.login[0], password: argv.login[1]}).then(

yielded the same result. BTW, I don’t use the Spark Local Cloud


#8

Sorry, what I meant is created the accessToken file and don’t use the --login option so it won’t access the token API at all.


#9

@rickkas7
Ah, all is good now!
Thanks, looks promising


#10

@rickkas7
I’m stuck with this:


And I don’t see where in the livegraph.ino code on github where the photon would actually obtain the server address and port?


#11

You might want to make sure you have an up-to-date particle-api-js. From the directory containing livegraph.js:

npm install particle-api-js

Also, check your node and npm versions:

node -v
npm -v

I’m using node 4.4.7 and npm 2.15.8, but it should work on later versions.

Check the event log at console.particle.io, the devices event should have data that’s in the known devices log entry in the node console; an IP address,port,and a long string of hex digits for the nonce.

For some reason the data is not there when the Photon receives the event. The devicesHandler data= should have the IP address, port, and nonce in the data.


#12

Here is a function from one of my devices that I use to “find” a device on my local network since I hate dealing with static IPs for a home network. It could be easily adapted to your example.

Requires http-client lib and sets request.ip used by http-client as the IP to use on success. Some simple on fail logic kicks off the process anytime I am unable to connect to the device for longer than 3 minutes.

bool findServer()
{
  IPAddress localIP = WiFi.localIP();
  uint8_t myLastAddrByte = localIP[3];
  for(uint8_t ipRange=2; ipRange<255; ipRange++)
  {
    if (ipRange != myLastAddrByte)
    {
      localIP[3] = ipRange;
      request.ip = localIP;
      request.port = SERVER_PORT;
      request.path = MYPAGE;
      httpclientobj.get(request, response, headers);
      if(response.status == 200)
      {
       //Possibly add a header or response check here.  Since 
      //my device uses Basic http auth if I got 200 "OK" that 
       //is what I am looking for
       Serial.println("Located the SERVER");
        return discovered = true;
      }
    }
    Particle.process();
  }
  return false;
}

#13

The way it works in this sample is it uses publish and subscribe to request the server IP, port, and a nonce. Since both side have to be logged into the same Particle cloud account, it offers more security. And the nonce authentications the connection, so only authorized clients can connect.


#14

@rickkas7
Thanks for the reply. Following your suggestions, I still get the following, and there’s no nonce shown in the console either:

the photon was programmed on 0.6.1-rc.1(prerelease)


#15

Oops. My mistake. The liveserver doesn’t return the server IP address, port and nonce with a publish, it uses a function. I don’t see anything obviously wrong with the code, lib/devices.js, line 127. It looks like it’s passing the data, and the data is logged correctly a few lines down, but for some reason it’s not getting passed to and logged in the Particle code. Weird.


#16

@rickkas7
So when I quickly did a terminal “particle call photon8 devices…” with the displayed server info and nonce, the graph displayed!


Thanks for the previous hint, but devices.js somehow doesn’t cut it on my machine by itself yet. Any ideas?


#17

I’ll take another look later and see if I can figure out why that’s happening.


#18

@rickkas7
Works beautifully now! Apparently particle-api-js version 6 was the problem because once I uninstalled that and installed version 5.3.1 all is good. Thanks again. I know I’ll have use for your framework. Probably add a write to file option.


#19

There’s a version 6.0.1 released now that fixes the login issue and this issue as well. Sorry for causing some frustration :bow:


#20

Hi all,

I am getting this,

Maybe i have missed something? I cant view any graph but i can view the published value perfectly.

Looking forward to your replies~ =)