Project Share - Low Power LoRA / Particle Gateway

chipmc · December 29, 2022, 7:50pm

First, to tie some threads together, this is a continuation of a LoRA thread by @lsfarm in 2020:

Which @jgskarda followed up as we started working together on LoRA and Particle:

Both of these got a lot of traction and so I wanted to share the progress I have made on this front just as Jeff has so well in his thread. My hope is that this will be useful to others who need to augment the Cellular and WiFi connectivity Particle offers with LoRA either to address coverage issues or to support high density deployments.

I want to thank everyone who has participated in these threads and to call our @jgskarda for his amazing diligence and energy and @rickkas7 who has provided the libraries and support that the solution is built upon.

Compared to @jgskarda’s implementations, my approach is fairly straightforward. Here is what my goal is for this solution:

To use LoRA to extend the reach of the cellular network to cover locations that would otherwise not connect, primarily in remote parks / recreation areas.
Make the solution easy enough to install that park staff can do it and the network should run reliably.
To keep things simple, I limited my solution to 10 nodes and allowed a generous 10 second spacing between transmission windows.

The solution has two parts: a gateway which is located where it can connect to both the cellular network and to the nodes (within about 2000 feet); and, up to 10 nodes which will connect to the gateway and who can relay messages for each other to extend the reach of the network.

Here is how the system operates:

The gateway is installed in a location with cellular service and put into “connection” mode where it is both connected to Particle and is listening for LoRA messages. This way, I can remotely monitor the installation and configure the network using Particle commands. Once set up, the gateway is put into normal - sleepy - operation.
The nodes are powered up and installed at their desired locations. Pressing the “user” button will cause them to transmit to the gateway and, they will display signal strength by how many “flashes” they give on the Boron’s RGB led. Nodes take a two step process when set up - they “join” the network and get a node number assigned and then they simply report data. All node properties such as time, node number, sensor type, park opening hours snd reporting frequency are set using these join and data messages. All messages are initiated by the nodes.
The gateway will build a node ID data file to keep track of nodes who have joined the network and their configuration. Gateways and nodes all sleep until the reporting window when the gateway wakes first and each node wakes in sequence at 10 second intervals. Node data reports are stored using a publish queue and are delivered to Ubidots via a webhook every hour when the gateway connects to Particle via cellular.
I tried to make this as robust as possible: If a collision occurs, the nodes will follow an exponential back-off to deconflict with up to three tries. If the gateway deems the node connections unhealthy it will reset. If a node fails to connect over two connection periods, it will power cycle and then start trying every minute until it catches the gateway.
To make it easier to configure the gateways and nodes, I created a method to parse multiple commands for the gateway and the nodes using a single Particle function. With this approach, I can set up each node and gateway with just a few commands.

// These commands will set the gateway's opening time to 6 am and closing to 9pm.
{"cmd":[{"node":0, "var":"6","fn":"open"},{"node":0, "var":"21","fn":"close"}]}

or

//Will reset the data on node 1 and assign it to sensor type 1 (person counter).  
{"cmd":[{"node":1,"var":"all","fn":"reset"}, {"node":1, "var":"1","fn":"type"}]}

Hardware:

I designed a variant of my standard Boron / Argon carrier board with a LoRA radio on it as shared here:

This solution is working quite well and it allows me to retrofit existing installations by simply switching out the carrier board and using the LoRA sticker antennae.

Software:
I am trying to get away from writing long - monolithic - code and have taken significant steps in that direction with this project. The software is divided into specific files and almost all functions are implemented using Singleton classes following @rickkas7 amazing tutorial and tools:

Using this approach, I have separated out the main file from persistent storage, Particle functions, LoRA Functions and the pins and IO of the device. My hope is that this approach is easier for others to see what I am doing, easier for me to maintain and written in a way that will make it easier to reuse the code.

One quick word on why Particle is such a great platform for this work and why I chose to use Borons for the gateway and also for the nodes. There are two reasons for this:

It is possible that the nodes will need to connect to cellular for updates in the future. Using a Boron with a special “connect to cellular” option, makes this possible though I hope to never use it.
The value of the Particle platform for example:

This community and the help I have received on this forum
A clear and well documented API
Clear and relevant application notes such as the Singleton Class referenced above
The amazing wealth of the Particle Libraries including the following I used in this project: PublishQueuePosixRK, StoageHelperRK, AB1805_RK, MB85RC256V-FRAM-RK, and @jgskarda’s branch of RF9X-RK. @rickkas7 - you are a rockstar!

I will update this thread with progress as these devices start to be deployed in the coming weeks. Please take a look at the software and hardware repositories below and ask questions or make suggestions for improvements. Again, thank you all for your support.

Github repos for the Gateway and Nodes

and for the hardware:

Colleen · January 2, 2023, 4:19am

This is so cool! Thanks for sharing, Chip!

jgskarda · January 2, 2023, 4:35pm

What a great project! Thanks for all the collaboration we’ve had on this @chipmc. I’ve been a bit quite in the forums the last month or so as started building/shipping hardware out to customer sites. Several of the systems are now “online” and working just as planned. Installed/commissioned by a non-tech savvy customer and hasn’t missed a beat in ~2 weeks since install. Also, I have a few nodes making 1 hope back to the hub so the repeater node functionality is also working as planned. I’ll make a quick post back to my original one forum, but as I shared with you, I also successfully bench tested 50 nodes to 1 Particle hub.

I’m excited to see where this continues to develop!

And yes, ditto, @rickkas7 - You are a Rockstar! The project wouldn’t have been possible without your libraries abstracting away a lot of complexity and overall making it possible for folks like Chip and I.

chipmc · January 9, 2023, 11:17pm

@all,

I have been field testing my Particle / LoRA gateway system and ran into an interesting issue. I wonder if I might share the current solution and see if anyone has a better suggestion. the gateway keeps track of the nodes in the network and assigned them each a unique node number. More on this here:

One issue I have run across is that there is the possibility that two nodes could “claim” to have the same node number. I realized that I needed to have some way for a node to “prove” they owned a given node number. This does not happen often but it seemed like a prudent step.

One way to do this would be to ask the node to send its deviceID each time it reported data - this seemed a bit wasteful as the 24 digits of the deviceID do not change and LoRA messages are meant to be short. So, I decided to create a two hex digit “check sum” that was literally the sum of the 24 digits in the deviceID. With this approach, the nodes add the two digit checksum to their data reports instead of the whole deviceID. The checksum can have values from 0 to 360 and the chances of two nodes having the same checksum (1/360) and claiming the same node number (not common) should be fairly small. I feel like it is a good balance of efficiency and resiliency.

Obviously, this approach gets less effective as the number of node goes up but, then again, if the number of nodes goes up, the chance that two will claim the same node number go down. If someone out there is good with probabilities, it could probably be worked out.

Here is the code I use to take the deviceID (a string) and return the checksum (int). I looked for a more slick implementation using type conversions but did not find anything that would work. Suggestions welcome but this code does the job:

int stringCheckSum(String str){
    int result = 0;
    for(unsigned int i = 0; i < str.length(); i++){
      int asciiCode = (int)str[i];

      if (asciiCode >=48 && asciiCode <58) {              // 0-9
        result += asciiCode - 48;
      } 
      else if (asciiCode >=65 && asciiCode < 71) {        // A-F
        result += 10 + asciiCode -65;
      }
      else if (asciiCode >=97 && asciiCode < 103) {       // a - f
        result += 10 + asciiCode -97;
      }
    }
    return result;
}

Anyway, I hope this is helpful and comments / suggestions are welcome as always.

Thanks,

Chip

jgskarda · January 10, 2023, 12:05am

Yeah, I think what you are proposing will work quite well. I too didn’t want to send a full 10 character serial number with each payload so instead, I send a 2 byte integer. When a LoRa node first powers on, the serial number is used only for the first join request transmission (along with a random generic node address assignment 1-9). The Particle LoRa gateway looks up the serial number and then assigns the device a node address (10-255) along with a unique device id (0-65,000). When the node receives the join ack, it updates it’s node number and then uses the device_id for any/all future transmissions. Since the Particle device assigned the node number and device ID to the device, at any point in time, it can simple compare what it received vs the configuration JSON for that device to see if the device is actually in the network or not. I think LoRa radio head library already does a checksum to ensure integrity of a packet received is valid/not corrupted so it seems very unlikely that you’d ever have duplicate node numbers in a system. That is as long as you have robust code assigning them to start. The only scenario I could think of is if one LoRa node that was talking with a particular Particle Gateway “moved” and became in proximity to an altogether different Particle LoRa Gateway and that other network already had that particular node assigned to a different device in that network. For me, that’s the possible scenario that might warrant the checks you are talking about. If all Particle Gateways are truly isolated, it seems very unlikely.

So I guess the only suggestion would be assign the device a specific 2 byte number instead of using a checkSum and you could guarantee it’s unique for that node up to a certain number of devices. That said, what you are doing seems easier so I’d say stick with what you are doing.

ScruffR · January 10, 2023, 8:11am

Have you considered using strtol()?
With this function you can pass in the base (e.g. 16 for HEX) for the string conversion.

After you found the integer value you could build the checksum purely mathematically.

chipmc · January 10, 2023, 2:00pm

@ScruffR,

Thank you, this could make things easier. However, I am struggling with this function for such a big number.

Here is a typical deviceID (not real) - e00fce68f94e5d9875bc13ff which would generate a checksum of 215. If I convert this to decimal, I get 69343750831196768517698098175 a 29 digit number with a checksum of 152 this is a different number but as long as the node and the gateway do the math the same way, that is OK.

But, this does not work as the number is too long - using the approach below, I can only accommodate a hex number of 7 digits before the conversion to decimal fails. Here is my code (with extra comments):

int stringCheckSum(String str){
    unsigned long deviceIDnumber = strtoll(str, NULL, 16);
    Log.info("Converted %s to %li", str.c_str(),deviceIDnumber);
    int result = 0;
    int oldresult = 0;

    for (int i=1; i <= 32; i++) {
      result += deviceIDnumber % 10;
      Log.info("deviceIDNumer is %lu, current addition is %d and result is %d",deviceIDnumber, result-oldresult, result);
      deviceIDnumber = deviceIDnumber / 10;
      oldresult = result;
    }

    return result;
}

It would seem that this would work but the deviceID is too large - even for long long.

Thanks,

Chip

peekay123 · January 10, 2023, 2:23pm

@chipmc, you may want to look at this CRC library on github, which supports a number of CRC sizes:

Treat the deviceID string as an array of uint8_t bytes and use those for the CRC calculation of your choice.

ScruffR · January 10, 2023, 3:11pm

I see, a 12 byte number/ID won’t be easily converted in one go that way.
8 byte would be possible with long long.
Anything longer would need to be split up into 8 byte (or four when going with long) chunks and the decimal alignment would need to be done manually - so not that straight forward.

Hence @peekay123’s suggestion is probably much more elegant

chipmc · January 10, 2023, 3:49pm

@peekay123 ,

Thank you for this link, this could prove helpful especially when I am looking to detect changes / errors in transmission.

Thing is that I am trying to do something much less robust than a CRC. I am posting this approach here as I am not 100% sure that I am thinking about this right.

Let me briefly restate what I am trying to do.

I am using Particle web hooks and Particle deviceIDs to collect data from nodes and send them on to my back-end service. I want to use deviceIDs for consistency, uniqueness and so I could move a node from cellular to LoRA and back without loosing continuity of data on my back-end.

The node sends its 24-digit deviceID once, when it joins the LoRA network
The gateway stores the deviceID and assigns a single byte nodeNumber
The node then uses the nodeNumber for all further communications with the gateway
The gateway looks up the deviceID for the node when it makes up the Particle webhook which it sends over cellular hourly.

The problem is that it is possible a node might come along and falsely claim it has a nodeNumber that was assigned already. I wanted to add a mechanism to detect this (fairly rare) event and flag the node to rejoin the network and get assigned a proper - non conflicting - nodeNumber. This is where I wanted to introduce the “checksum” of two-bytes that could be easily calculated from the deviceID. The node would send this checksum on each data report and the gateway would validate the nodeNumber’s validity using this.

I realize this checksum is too short to be unique like a CRC but, I want to reduce the number of bytes sent. For example using the deviceID above - the CRC is shown below:

So, the question is this: Is the value of a unique 8 digit CRC enough high enough to justify:

Adding another library to the build
Sending 6 more bytes on each data report
The computational cost of a more complex algorithm

This is the question I am wrestling with - any suggestions welcome.

Thanks, Chip

peekay123 · January 10, 2023, 4:27pm

@chipmc, I believe a 16bit CRC is the sweet spot, requiring 4 bytes vs 2 for the 8bit CRC. Given you scenario, it provides the best probability that you will not have any duplicates. As for the computation overhead, you are likely overthinking it and adding an extra library is not a big deal. Remember that the linker will only pull compiled code that is used into the final image.

Possibly another approach would be to treat the deviceID as an array of 6 sets of 4-byte 16-bit values (using some pointer and casting magic). You could then add (or XOR) those 6 values up to give you the 16-bit “checksum”.

chipmc · January 10, 2023, 4:54pm

Thank you @peekay123 ,

I think you are correct. This approach will support a solution that does not require any pre-configuration of the nodes and still gives a high confidence that nodeNumber conflicts will arise.

Chip

system · February 10, 2023, 2:54am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.