Sending data with mesh devices

rickkas7 · December 14, 2018, 6:17pm

As it turns out, I misunderstood a few things about how data exchange works on mesh devices, and there are a lot more options for sending data from Xenons directly to the Internet, if you use UDP. In fact, you can communicate directly with IPv4 hosts on the Internet using UDP, directly from the Xenon!

Particle.publish and Particle.subscribe

The standard Particle.publish and Particle.subscribe functions work across the mesh network, and also allow communication with APIs (via webhooks and SSE) and earlier non-mesh devices (Photon, P1, Electron, E Series, and Core) without modification.

All traffic through publish and subscribe goes through the gateway and to the Particle cloud through the Internet (encrypted, of course).

Sample Code:

#include "Particle.h"

SerialLogHandler logHandler;

void subscriptionHandler(const char *event, const char *data);

const char *EVENT_NAME = "test-cloud-pub-sub";
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;
unsigned long lastPublish = 0;


void setup() {
	Particle.subscribe(EVENT_NAME, subscriptionHandler, MY_DEVICES);
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();
		char data[128];
		snprintf(data, sizeof(data), "counter=%d", ++counter);

		Log.info("sent %s", data);
		Particle.publish(EVENT_NAME, data, PRIVATE);
	}
}

void subscriptionHandler(const char *event, const char *data) {
	Log.info("received %s", data);
}

This code subscribes to an the test-cloud-pub-sub event and prints out any events it receives to USB debug serial. It also sends an event every 10 seconds with a counter that increments each time.

Console event log:

Sample USB serial log:

0000010000 [app] INFO: sent counter=1
0000010064 [comm.protocol] INFO: message id 35 complete with code 0.00
0000010067 [comm.protocol] INFO: rcv'd message type=13
0000010081 [app] INFO: received counter=1
0000010082 [comm.protocol] INFO: rcv'd message type=8
0000012565 [app] INFO: received counter=2
0000012567 [comm.protocol] INFO: rcv'd message type=8
0000020000 [app] INFO: sent counter=2
0000020060 [comm.protocol] INFO: message id 36 complete with code 0.00
0000020063 [comm.protocol] INFO: rcv'd message type=13
0000020075 [app] INFO: received counter=2
0000020077 [comm.protocol] INFO: rcv'd message type=8
0000022541 [app] INFO: received counter=3
0000022542 [comm.protocol] INFO: rcv'd message type=8
0000030000 [app] INFO: sent counter=3
0000030063 [comm.protocol] INFO: message id 37 complete with code 0.00
0000030065 [comm.protocol] INFO: rcv'd message type=13
0000030071 [app] INFO: received counter=3
0000030072 [comm.protocol] INFO: rcv'd message type=8

Mesh.publish and Mesh.subscribe

The Mesh.publish and Mesh.subscribe allow communication within your mesh network. This applies only to the network you are joined to; you cannot communicate with other mesh networks even if they are in the same location.

The main advantage is that communication is local: Xenons can continue to communicate with each other, even if the Internet or even if the gateway goes down.

Code:

#include "Particle.h"

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

void subscriptionHandler(const char *event, const char *data);

const char *EVENT_NAME = "test-mesh-pub-sub";
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;
unsigned long lastPublish = 0;


void setup() {
	Mesh.subscribe(EVENT_NAME, subscriptionHandler);
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();
		char data[128];
		snprintf(data, sizeof(data), "counter=%d", ++counter);

		Log.info("sent %s", data);
		Mesh.publish(EVENT_NAME, data);
	}
}

void subscriptionHandler(const char *event, const char *data) {
	Log.info("received %s", data);
}

The code is almost the same except it uses Mesh.publish and Mesh.subscribe.

USB serial debug log:

0000060000 [app] INFO: sent counter=6
0000060760 [app] INFO: received counter=2
0000070000 [app] INFO: sent counter=7
0000070761 [app] INFO: received counter=3
0000080000 [app] INFO: sent counter=8
0000080760 [app] INFO: received counter=4

Since the data does not go to the cloud it won’t show up in the event log in the console.

You’ll also notice that this version contains:

SYSTEM_THREAD(ENABLED);

This allows the user firmware to run even when not cloud connected. Enabling this allows the publish and subscribe code to continue even if you’re disconnected the gateway from the Internet or even turned the gateway off.

Internally, mesh publish and subscribe use UDP multicast.

UDP

The mesh network is optimized for UDP (unreliable, packet-based) data transmission. One reason is that once you start utilizing nodes that sleep and networks with many hops, having the ability to store and forward the packet later becomes necessary. This is better suited for UDP-based protocols than TCP.

udp-send-only

The udp-send-only example sends a UDP packet directly to a server, on your local LAN, or even a host on the Internet!

In this example, we send to a specific IP address. It’s configured here:

// This is the remote host to connect to (IPv4 address)
//                             vv this part stays the same                                         vv | vv server address goes here
const uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 192, 168, 2, 4, 6};

The IP address in this example is 192.168.2.4, and you should change that to the address you want to send to. Make sure you leave the 6 after it, though. That indicates an IPv6 address.

Here’s the full code:

#include "Particle.h"

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

// This is the remote host to connect to (IPv4 address)
//                             vv this part stays the same                                         vv | vv server address goes here
const uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 192, 168, 2, 4, 6};

IPAddress remoteHost((const HAL_IPAddress&)remoteAddr);

uint16_t UDP_PORT = 7123;
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;

UDP udp;
bool udpInitialized = false;

void setup() {
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();

		if (Mesh.ready()) {
			if (!udpInitialized) {
				udpInitialized = true;

				// Must call udp.begin() whenever the network layer comes up
				udp.begin(0);

				Log.info("udp.begin called");
			}

			char data[128];
			snprintf(data, sizeof(data), "counter=%d", ++counter);

			udp.sendPacket((const uint8_t *)data, strlen(data), remoteHost, UDP_PORT);

			Log.info("sent %s to %s port %d", data, remoteHost.toString().c_str(), UDP_PORT);
		}
		else {
			Log.info("mesh network not ready");
			udpInitialized = false;
		}
	}
}

Sample USB serial debug log:

0000010000 [app] INFO: udp.begin called
0000010002 [app] INFO: sent counter=1 to 64:FF9B::C0A8:204 port 7123
0000020001 [app] INFO: sent counter=2 to 64:FF9B::C0A8:204 port 7123

I used the “nc” program (netcat) to listen on UDP port 7123. Here’s the output:

$ nc -ul 7123
counter=1counter=2counter=3

udp-bidirectional

This is pretty much like the previous example, but it does bi-directional UDP.

The gateway acts as a NAT64 router and is able to route packets back to the Xenon on the mesh network through a temporary reply channel.

Even more amazing to me is that this also works from a home network using NAT to the Internet, so you can send and receive UDP packets from a Xenon all the way to the Internet without any firewall configuration!

Note that this is not a server socket, per se, so it will only live for a short period time, and the request must be initiated from the Xenon.

Device code:

#include "Particle.h"

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

// This is the remote host to connect to (IPv4 address). Make sure the last entry is 6 (indicates an IPv6 address)!
//                             vv this part stays the same                                         vv | vv server address goes here
const uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 192, 168, 2, 4, 6};


IPAddress remoteHost((const HAL_IPAddress&)remoteAddr);

uint16_t UDP_PORT = 7123;
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;

UDP udp;
bool udpInitialized = false;
unsigned long lastPublish = 0;

void setup() {
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();

		if (Mesh.ready()) {
			if (!udpInitialized) {
				udpInitialized = true;

				// Must call udp.begin() whenever the network layer comes up
				udp.begin(UDP_PORT);

				Log.info("udp.begin called");
			}

			char data[128];
			snprintf(data, sizeof(data), "counter=%d", ++counter);

			udp.sendPacket((const uint8_t *)data, strlen(data), remoteHost, UDP_PORT);

			Log.info("sent %s to %s port %d", data, remoteHost.toString().c_str(), UDP_PORT);
		}
		else {
			Log.info("mesh network not ready");
			udpInitialized = false;
		}
	}

	if (Mesh.ready()) {
		char buffer[256];
		int size = udp.receivePacket(buffer, sizeof(buffer) - 1);
		if (size > 0) {
			buffer[size] = 0;
			Log.info("received %s", buffer);
			// Particle.publish("test-pkt-rcvd", buffer, PRIVATE);
		}
	}
}

Because we need to send and receive UDP packets, we can’t (easily) just use nc, so there’s a small node.js server:

// Run with:
// node server.js

var dgram = require('dgram');

var server = dgram.createSocket('udp4');

const SERVER_PORT = 7123;


server.on('listening', function () {
    var addr = server.address();
    console.log('server listening on ' + addr.address + ":" + addr.port);
});

server.on('message', function (message, remoteAddr) {
    console.log('received ' + remoteAddr.address + ':' + remoteAddr.port +' - ' + message);
    
    var buf = new Buffer('reply to ' + message);

    server.send(buf, 0, buf.length, remoteAddr.port, remoteAddr.address, function(err, bytes) {
        if (err) throw err;
        console.log('reply successfully sent!');
    });
    
});

server.bind(SERVER_PORT);

The most important thing about the server is that it responds to the address and port the packet came from. Because of NAT, it will be different than the port being listened on, on the Xenon!

USB debug serial log:

0000010000 [app] INFO: udp.begin called
0000010002 [app] INFO: sent counter=1 to 64:FF9B::C0A8:204 port 7123
0000010018 [app] INFO: received reply to counter=1
0000020001 [app] INFO: sent counter=2 to 64:FF9B::C0A8:204 port 7123
0000020018 [app] INFO: received reply to counter=2

node.js server output:

$ node server.js 
server listening on 0.0.0.0:7123
received 192.168.2.178:48804 - counter=1
reply successfully sent!
received 192.168.2.178:48804 - counter=2
reply successfully sent!

bittailor · December 15, 2018, 9:32am

Great Tutorial Thanks!

Is there a way to do DNS lookup on the Xenon to get the IP address of a hostname?

rickkas7 · December 15, 2018, 10:35am

Yes - here’s an example that uses DNS. The server is the same as the last example (udp-bidirectional):

#include "Particle.h"

// Use node.js server code in udp-bidirectional

SerialLogHandler logHandler;

SYSTEM_THREAD(ENABLED);

const char *HOSTNAME = "server.example.com"; // <- change this!

uint16_t UDP_PORT = 7123;
const unsigned long PUBLISH_INTERVAL_MS = 10000;

int counter = 0;
IPAddress remoteHost;

UDP udp;
bool udpInitialized = false;
unsigned long lastPublish = 0;

void setup() {
}

void loop() {
	if (millis() - lastPublish >= PUBLISH_INTERVAL_MS) {
		lastPublish = millis();

		if (Mesh.ready()) {
			if (!udpInitialized) {
				udpInitialized = true;

				// Must call udp.begin() whenever the network layer comes up
				udp.begin(UDP_PORT);

				Log.info("udp.begin called");

				IPAddress dnsAddr = Mesh.resolve(HOSTNAME);
				if (dnsAddr.version() == 4) {
					uint8_t remoteAddr[33] = {0x00, 0x64, 0xff, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0, 0, 0, 0, 6};
					for(size_t ii = 0; ii < 4; ii++) {
						remoteAddr[12 + ii] = dnsAddr[ii];
					}
					remoteHost = IPAddress((const HAL_IPAddress&)remoteAddr);
					Log.info("DNS found IPv4 %s -> %s -> %s", HOSTNAME, dnsAddr.toString().c_str(), remoteHost.toString().c_str());
				}
				else {
					remoteHost = dnsAddr;
					Log.info("DNS found IPv6 %s -> %s", HOSTNAME, remoteHost.toString().c_str());
				}
			}

			char data[128];
			snprintf(data, sizeof(data), "counter=%d", ++counter);

			udp.sendPacket((const uint8_t *)data, strlen(data), remoteHost, UDP_PORT);

			Log.info("sent %s to %s port %d", data, remoteHost.toString().c_str(), UDP_PORT);
		}
		else {
			Log.info("mesh network not ready");
			udpInitialized = false;
		}
	}

	if (Mesh.ready()) {
		char buffer[256];
		int size = udp.receivePacket(buffer, sizeof(buffer) - 1);
		if (size > 0) {
			buffer[size] = 0;
			Log.info("received %s", buffer);
			// Particle.publish("test-pkt-rcvd", buffer, PRIVATE);
		}
	}
}

syrinxtech · December 15, 2018, 2:53pm

If it would help, here is a link to some code that I wrote to send syslog messages via UDP.

https://go.particle.io/shared_apps/5c151500cfed66cf83001457

It works on a Photon but I haven’t tried it on any of the meshed devices yet.

Deetle · December 15, 2018, 3:46pm

Here is a little python script for a UDP server to test the send only code sketch sample from above from a mesh Xenon. I ran the python script on my PC and a Raspberry Pi and it worked for me.

import socket

UDP_IP_ADDRESS = "X.X.X.X"   # IP address of computer this script is running on. 
UDP_PORT_NO = 7123

serverSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
serverSock.bind((UDP_IP_ADDRESS, UDP_PORT_NO))


while True:
    data, addr = serverSock.recvfrom(1024)
    print("Message: ", data)

The pythons script output looks like

Message:  b'counter=219'
Message:  b'counter=220'
Message:  b'counter=221'
Message:  b'counter=222'
Message:  b'counter=223'
Message:  b'counter=224'
Message:  b'counter=225'
Message:  b'counter=226'
Message:  b'counter=227'
Message:  b'counter=228'

Deetle · December 15, 2018, 11:56pm

This works with the bi-directional xenon sketch from above.

import socket


UDP_IP_ADDRESS = "X.X.X.X"   # IP address of computer this script is running on. 
UDP_PORT_NO = 7123

serverSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

serverSock.bind((UDP_IP_ADDRESS, UDP_PORT_NO))

while True:
    data, addr = serverSock.recvfrom(1024)
    print("Message: ", data)

    if data:
        sent = serverSock.sendto(data, addr)

jimbol · January 14, 2019, 6:01am

Is it possible to send UDP multicast from the Xenon? I have an Argon/Xenon Mesh network and I was hoping to get the Xenon to run the same UDP multicast code as the Argon, and not have to write separate mesh.pub/sub code to make the Argon send the UDP on behalf of the Xenon.

ScruffR · January 14, 2019, 6:16am

IIRC Mesh.publish() is using UDP multicast.

If we knew how that code looks, we might be able to advise.

jimbol · January 14, 2019, 6:31am

@ScruffR,

https://go.particle.io/shared_apps/5c3c2be8b7b2e54caf00072b

ScruffR · January 14, 2019, 5:04pm

In order to receive mutlicast packets you’d need to call UDP.joinMulticast() with a valid multicast address and send the packets to that address.

jimbol · January 14, 2019, 5:14pm

@ScruffR, I’ll give that a try.

I see that I did use

UDP.joinMulticast()

for my receiving code, but interestingly this worked on the sending Argon without it.

EDIT: I added that to my Argon code and got it working, but still no joy with the Xenon. It performs all the other functions of my code, and doesn’t give any errors, but I don’t see the multicast on my network.

EDIT: I was able to adapt the UDP code from @rickkas7 above to send multicast, so I should be able to mod my code above to pull it off. I was a bit surprised that a multicast code that worked for the argon didn’t do so with the Xenon, and don’t really understand why, but I’ll just tweak it to work like the example provided above.

armor · January 27, 2019, 4:15pm

If I just want to send a file of data (base64 encoded bitmap - 5.4K bytes encoded) from a mesh gateway device to another mesh network device (end node), would I be better off using a series of mesh.publish(); or UDP? Currently the mesh.publish() data load is maximum 255 bytes? What would be the positives of using UDP and the negatives?

ScruffR · January 28, 2019, 6:38am

Pro UDP

can transfer binary data
can transport bigger packets at once
is marginally faster due to less overhead

Pro Mesh.publish()

is encrypted
system hides some complexity

Considerations for both

Mesh.publish() is also UDP based so packet delivery is not guaranteed, you need to take care of that in code

rocksetta · October 3, 2019, 5:58am

Can someone explain what the Mesh.publish(“name”,“data”) actually sends as a UDP multicast?

All I can find about UDP is that a point to point message would look like this:


udp send fdde:ad00:beef:0:bb1:ebd6:ad10:f33 1234 hello

Somehow particle must compress the name and data part of it’s message into something like the hello above. Does it use JSON format? Also the code above is point to point. How does it send to everyone on the network.

For TCP I have seen:

ping ff02:1  
ping ff02:2  
ping ff03:1

which seems to get information about each node on the thread.
but I have not seen anything similar for UDP multicasting. Any suggestions?

rickkas7 · October 3, 2019, 9:09am

Mesh.publish uses UDP multicast on the private IPv6 network between the devices on a mesh network.

The UDP payload itself isn’t encrypted, but the private network is not accessible except on nodes on that mesh network and its gateways. Thread mesh encrypts the data using AES, so it’s not necessary to encrypt it again in UDP.

The payload in the UDP multicast packet is a very simple: A version number, the topic, and the data payload.

rocksetta · October 4, 2019, 9:42pm

UDP can send any message that does not have a space, what format is it sent with I would assume JSON, but it could be anything.

Does this make sense using 9000 as the UDP_PORT

udp send ff02::1 9000 {"version":"2","topic":"myEvent","data":3"}

Topic		Replies	Views
Using UDP to transfer files between mesh gateway and end-nodes BLE // NFC xenon	6	1369	January 27, 2019
Xenon - Boron Communications Firmware	12	1496	November 19, 2018
MESH: Confused new guy :) Getting Started boron , xenon	2	519	April 10, 2019
Looking for advice on communicating with local Mesh nodes BLE // NFC	8	960	November 17, 2018
Xenon and httpclient lib Firmware xenon	2	385	March 21, 2019

Sending data with mesh devices

Particle.publish and Particle.subscribe

Mesh.publish and Mesh.subscribe

UDP

udp-send-only

udp-bidirectional

Related topics