Sending data with mesh devices


#1

As it turns out, I misunderstood a few things about how data exchange works on mesh devices, and there are a lot more options for sending data from Xenons directly to the Internet, if you use UDP. In fact, you can communicate directly with IPv4 hosts on the Internet using UDP, directly from the Xenon!

Particle.publish and Particle.subscribe

The standard Particle.publish and Particle.subscribe functions work across the mesh network, and also allow communication with APIs (via webhooks and SSE) and earlier non-mesh devices (Photon, P1, Electron, E Series, and Core) without modification.

All traffic through publish and subscribe goes through the gateway and to the Particle cloud through the Internet (encrypted, of course).

Sample Code:

#include "Particle.h"

SerialLogHandler logHandler;

void subscriptionHandler(const char *event, const char *data);

const char *EVENT_NAME = "test-cloud-pub-sub";
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;
unsigned long lastPublish = 0;


void setup() {
	Particle.subscribe(EVENT_NAME, subscriptionHandler, MY_DEVICES);
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();
		char data[128];
		snprintf(data, sizeof(data), "counter=%d", ++counter);

		Log.info("sent %s", data);
		Particle.publish(EVENT_NAME, data, PRIVATE);
	}
}

void subscriptionHandler(const char *event, const char *data) {
	Log.info("received %s", data);
}

This code subscribes to an the test-cloud-pub-sub event and prints out any events it receives to USB debug serial. It also sends an event every 10 seconds with a counter that increments each time.

Console event log:

Sample USB serial log:

0000010000 [app] INFO: sent counter=1
0000010064 [comm.protocol] INFO: message id 35 complete with code 0.00
0000010067 [comm.protocol] INFO: rcv'd message type=13
0000010081 [app] INFO: received counter=1
0000010082 [comm.protocol] INFO: rcv'd message type=8
0000012565 [app] INFO: received counter=2
0000012567 [comm.protocol] INFO: rcv'd message type=8
0000020000 [app] INFO: sent counter=2
0000020060 [comm.protocol] INFO: message id 36 complete with code 0.00
0000020063 [comm.protocol] INFO: rcv'd message type=13
0000020075 [app] INFO: received counter=2
0000020077 [comm.protocol] INFO: rcv'd message type=8
0000022541 [app] INFO: received counter=3
0000022542 [comm.protocol] INFO: rcv'd message type=8
0000030000 [app] INFO: sent counter=3
0000030063 [comm.protocol] INFO: message id 37 complete with code 0.00
0000030065 [comm.protocol] INFO: rcv'd message type=13
0000030071 [app] INFO: received counter=3
0000030072 [comm.protocol] INFO: rcv'd message type=8

Mesh.publish and Mesh.subscribe

The Mesh.publish and Mesh.subscribe allow communication within your mesh network. This applies only to the network you are joined to; you cannot communicate with other mesh networks even if they are in the same location.

The main advantage is that communication is local: Xenons can continue to communicate with each other, even if the Internet or even if the gateway goes down.

Code:

#include "Particle.h"

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

void subscriptionHandler(const char *event, const char *data);

const char *EVENT_NAME = "test-mesh-pub-sub";
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;
unsigned long lastPublish = 0;


void setup() {
	Mesh.subscribe(EVENT_NAME, subscriptionHandler);
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();
		char data[128];
		snprintf(data, sizeof(data), "counter=%d", ++counter);

		Log.info("sent %s", data);
		Mesh.publish(EVENT_NAME, data);
	}
}

void subscriptionHandler(const char *event, const char *data) {
	Log.info("received %s", data);
}

The code is almost the same except it uses Mesh.publish and Mesh.subscribe.

USB serial debug log:

0000060000 [app] INFO: sent counter=6
0000060760 [app] INFO: received counter=2
0000070000 [app] INFO: sent counter=7
0000070761 [app] INFO: received counter=3
0000080000 [app] INFO: sent counter=8
0000080760 [app] INFO: received counter=4

Since the data does not go to the cloud it won’t show up in the event log in the console.

You’ll also notice that this version contains:

SYSTEM_THREAD(ENABLED);

This allows the user firmware to run even when not cloud connected. Enabling this allows the publish and subscribe code to continue even if you’re disconnected the gateway from the Internet or even turned the gateway off.

Internally, mesh publish and subscribe use UDP multicast.

UDP

The mesh network is optimized for UDP (unreliable, packet-based) data transmission. One reason is that once you start utilizing nodes that sleep and networks with many hops, having the ability to store and forward the packet later becomes necessary. This is better suited for UDP-based protocols than TCP.

udp-send-only

The udp-send-only example sends a UDP packet directly to a server, on your local LAN, or even a host on the Internet!

In this example, we send to a specific IP address. It’s configured here:

// This is the remote host to connect to (IPv4 address)
//                             vv this part stays the same                                         vv | vv server address goes here
const uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 192, 168, 2, 4, 6};

The IP address in this example is 192.168.2.4, and you should change that to the address you want to send to. Make sure you leave the 6 after it, though. That indicates an IPv6 address.

Here’s the full code:

#include "Particle.h"

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

// This is the remote host to connect to (IPv4 address)
//                             vv this part stays the same                                         vv | vv server address goes here
const uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 192, 168, 2, 4, 6};

IPAddress remoteHost((const HAL_IPAddress&)remoteAddr);

uint16_t UDP_PORT = 7123;
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;

UDP udp;
bool udpInitialized = false;

void setup() {
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();

		if (Mesh.ready()) {
			if (!udpInitialized) {
				udpInitialized = true;

				// Must call udp.begin() whenever the network layer comes up
				udp.begin(0);

				Log.info("udp.begin called");
			}

			char data[128];
			snprintf(data, sizeof(data), "counter=%d", ++counter);

			udp.sendPacket((const uint8_t *)data, strlen(data), remoteHost, UDP_PORT);

			Log.info("sent %s to %s port %d", data, remoteHost.toString().c_str(), UDP_PORT);
		}
		else {
			Log.info("mesh network not ready");
			udpInitialized = false;
		}
	}
}

Sample USB serial debug log:

0000010000 [app] INFO: udp.begin called
0000010002 [app] INFO: sent counter=1 to 64:FF9B::C0A8:204 port 7123
0000020001 [app] INFO: sent counter=2 to 64:FF9B::C0A8:204 port 7123

I used the “nc” program (netcat) to listen on UDP port 7123. Here’s the output:

$ nc -ul 7123
counter=1counter=2counter=3

udp-bidirectional

This is pretty much like the previous example, but it does bi-directional UDP.

The gateway acts as a NAT64 router and is able to route packets back to the Xenon on the mesh network through a temporary reply channel.

Even more amazing to me is that this also works from a home network using NAT to the Internet, so you can send and receive UDP packets from a Xenon all the way to the Internet without any firewall configuration!

Note that this is not a server socket, per se, so it will only live for a short period time, and the request must be initiated from the Xenon.

Device code:

#include "Particle.h"

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

// This is the remote host to connect to (IPv4 address). Make sure the last entry is 6 (indicates an IPv6 address)!
//                             vv this part stays the same                                         vv | vv server address goes here
const uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 192, 168, 2, 4, 6};


IPAddress remoteHost((const HAL_IPAddress&)remoteAddr);

uint16_t UDP_PORT = 7123;
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;

UDP udp;
bool udpInitialized = false;
unsigned long lastPublish = 0;

void setup() {
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();

		if (Mesh.ready()) {
			if (!udpInitialized) {
				udpInitialized = true;

				// Must call udp.begin() whenever the network layer comes up
				udp.begin(UDP_PORT);

				Log.info("udp.begin called");
			}

			char data[128];
			snprintf(data, sizeof(data), "counter=%d", ++counter);

			udp.sendPacket((const uint8_t *)data, strlen(data), remoteHost, UDP_PORT);

			Log.info("sent %s to %s port %d", data, remoteHost.toString().c_str(), UDP_PORT);
		}
		else {
			Log.info("mesh network not ready");
			udpInitialized = false;
		}
	}

	if (Mesh.ready()) {
		char buffer[256];
		int size = udp.receivePacket(buffer, sizeof(buffer) - 1);
		if (size > 0) {
			buffer[size] = 0;
			Log.info("received %s", buffer);
			// Particle.publish("test-pkt-rcvd", buffer, PRIVATE);
		}
	}
}

Because we need to send and receive UDP packets, we can’t (easily) just use nc, so there’s a small node.js server:

// Run with:
// node server.js

var dgram = require('dgram');

var server = dgram.createSocket('udp4');

const SERVER_PORT = 7123;


server.on('listening', function () {
    var addr = server.address();
    console.log('server listening on ' + addr.address + ":" + addr.port);
});

server.on('message', function (message, remoteAddr) {
    console.log('received ' + remoteAddr.address + ':' + remoteAddr.port +' - ' + message);
    
    var buf = new Buffer('reply to ' + message);

    server.send(buf, 0, buf.length, remoteAddr.port, remoteAddr.address, function(err, bytes) {
        if (err) throw err;
        console.log('reply successfully sent!');
    });
    
});

server.bind(SERVER_PORT);

The most important thing about the server is that it responds to the address and port the packet came from. Because of NAT, it will be different than the port being listened on, on the Xenon!

USB debug serial log:

0000010000 [app] INFO: udp.begin called
0000010002 [app] INFO: sent counter=1 to 64:FF9B::C0A8:204 port 7123
0000010018 [app] INFO: received reply to counter=1
0000020001 [app] INFO: sent counter=2 to 64:FF9B::C0A8:204 port 7123
0000020018 [app] INFO: received reply to counter=2

node.js server output:

$ node server.js 
server listening on 0.0.0.0:7123
received 192.168.2.178:48804 - counter=1
reply successfully sent!
received 192.168.2.178:48804 - counter=2
reply successfully sent!

UDP and TCP in Mesh
Using UDP to transfer files between mesh gateway and end-nodes
#2

Great Tutorial Thanks!

Is there a way to do DNS lookup on the Xenon to get the IP address of a hostname?


#3

Yes - here’s an example that uses DNS. The server is the same as the last example (udp-bidirectional):

#include "Particle.h"

// Use node.js server code in udp-bidirectional

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

const char *HOSTNAME = "server.example.com"; // <- change this!

uint16_t UDP_PORT = 7123;
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;
IPAddress remoteHost;

UDP udp;
bool udpInitialized = false;
unsigned long lastPublish = 0;

void setup() {
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();

		if (Mesh.ready()) {
			if (!udpInitialized) {
				udpInitialized = true;

				// Must call udp.begin() whenever the network layer comes up
				udp.begin(UDP_PORT);

				Log.info("udp.begin called");

				IPAddress dnsAddr = Mesh.resolve(HOSTNAME);
				if (dnsAddr.version() == 4) {
					uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0, 0, 0, 0, 6};
					for(size_t ii = 0; ii < 4; ii++) {
						remoteAddr[12 + ii] = dnsAddr[ii];
					}
					remoteHost = IPAddress((const HAL_IPAddress&)remoteAddr);
					Log.info("DNS found IPv4 %s -> %s -> %s", HOSTNAME, dnsAddr.toString().c_str(), remoteHost.toString().c_str());
				}
				else {
					remoteHost = dnsAddr;
					Log.info("DNS found IPv6 %s -> %s", HOSTNAME, remoteHost.toString().c_str());
				}
			}

			char data[128];
			snprintf(data, sizeof(data), "counter=%d", ++counter);

			udp.sendPacket((const uint8_t *)data, strlen(data), remoteHost, UDP_PORT);

			Log.info("sent %s to %s port %d", data, remoteHost.toString().c_str(), UDP_PORT);
		}
		else {
			Log.info("mesh network not ready");
			udpInitialized = false;
		}
	}

	if (Mesh.ready()) {
		char buffer[256];
		int size = udp.receivePacket(buffer, sizeof(buffer) - 1);
		if (size > 0) {
			buffer[size] = 0;
			Log.info("received %s", buffer);
			// Particle.publish("test-pkt-rcvd", buffer, PRIVATE);
		}
	}
}


#4

If it would help, here is a link to some code that I wrote to send syslog messages via UDP.

https://go.particle.io/shared_apps/5c151500cfed66cf83001457

It works on a Photon but I haven’t tried it on any of the meshed devices yet.


#5

Here is a little python script for a UDP server to test the send only code sketch sample from above from a mesh Xenon. I ran the python script on my PC and a Raspberry Pi and it worked for me.

import socket

UDP_IP_ADDRESS = "X.X.X.X"   # IP address of computer this script is running on. 
UDP_PORT_NO = 7123

serverSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
serverSock.bind((UDP_IP_ADDRESS, UDP_PORT_NO))


while True:
    data, addr = serverSock.recvfrom(1024)
    print("Message: ", data)

The pythons script output looks like

Message:  b'counter=219'
Message:  b'counter=220'
Message:  b'counter=221'
Message:  b'counter=222'
Message:  b'counter=223'
Message:  b'counter=224'
Message:  b'counter=225'
Message:  b'counter=226'
Message:  b'counter=227'
Message:  b'counter=228'

#6

This works with the bi-directional xenon sketch from above.

import socket


UDP_IP_ADDRESS = "X.X.X.X"   # IP address of computer this script is running on. 
UDP_PORT_NO = 7123

serverSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

serverSock.bind((UDP_IP_ADDRESS, UDP_PORT_NO))

while True:
    data, addr = serverSock.recvfrom(1024)
    print("Message: ", data)

    if data:
        sent = serverSock.sendto(data, addr)

#7

Is it possible to send UDP multicast from the Xenon? I have an Argon/Xenon Mesh network and I was hoping to get the Xenon to run the same UDP multicast code as the Argon, and not have to write separate mesh.pub/sub code to make the Argon send the UDP on behalf of the Xenon.


#8

IIRC Mesh.publish() is using UDP multicast.

If we knew how that code looks, we might be able to advise.


#9

@ScruffR,

https://go.particle.io/shared_apps/5c3c2be8b7b2e54caf00072b


#10

In order to receive mutlicast packets you’d need to call UDP.joinMulticast() with a valid multicast address and send the packets to that address.


#11

@ScruffR, I’ll give that a try.

I see that I did use

UDP.joinMulticast()

for my receiving code, but interestingly this worked on the sending Argon without it.

EDIT: I added that to my Argon code and got it working, but still no joy with the Xenon. It performs all the other functions of my code, and doesn’t give any errors, but I don’t see the multicast on my network.

EDIT: I was able to adapt the UDP code from @rickkas7 above to send multicast, so I should be able to mod my code above to pull it off. I was a bit surprised that a multicast code that worked for the argon didn’t do so with the Xenon, and don’t really understand why, but I’ll just tweak it to work like the example provided above.


#12

If I just want to send a file of data (base64 encoded bitmap - 5.4K bytes encoded) from a mesh gateway device to another mesh network device (end node), would I be better off using a series of mesh.publish(); or UDP? Currently the mesh.publish() data load is maximum 255 bytes? What would be the positives of using UDP and the negatives?


#13

Pro UDP

  • can transfer binary data
  • can transport bigger packets at once
  • is marginally faster due to less overhead

Pro Mesh.publish()

  • is encrypted
  • system hides some complexity

Considerations for both

  • Mesh.publish() is also UDP based so packet delivery is not guaranteed, you need to take care of that in code