How to get GPS data into JSON object w/ Boron

[krixen1]

I am having a seemingly simple issue with writing a json object over LoRA from one Boron to another. Currently I am able to send sensor data from various sensors in an object over to another boron, and periodically, I want to send in GPS data as well. Whenever it attempts to add the GPS field to the json object to send over to the other boron, the message fails to send.

Here is the code I am using:

Writing GPS Data to json object periodically

 // lorawrite of size 1023, data is somewhere around 300-400 bytes
JSONBufferWriter writer(lorawrite, sizeof(lorawrite));
writer.beginObject();
	writer.name("id").value(System.deviceID());
	writer.name("Pitch").value(String(pitch));
	writer.name("Temperature").value(String(temp));
	writer.name("Pressure").value(String(pressure));
	writer.name("Roll").value(String(roll));
	if (millis() - gps_write_timer > 30000)
	{
		Serial.println(gps_data);
		writer.name("GPS").value(String(gps_data));
		gps_write_timer = millis();
	}
	//writer.name("GPS").value(String(gps_data));
writer.endObject();

if (lora_connected && !is_super_node && !cloud_connected)
{

	if (LoRA::instance().sendMessage((uint8_t*)lorawrite, strlen(lorawrite)) == RH_ROUTER_ERROR_NONE)
	{
		Serial.println("sent message to server");
	}
	else
	{
		Serial.println("sendtoWait failed. Are the intermediate mesh servers running?");
	}

}

getting GPS data

char gps_data[512] = ""; //global

// inside loop
if (millis() - gps_read_timer > 1000)
{
	if (GPS_Read(gps_data))
	{
		if (cloud_connected)
		{
			Particle.publish(gps_data);
		}
		Serial.print("GPS Data:");
		Serial.println(gps_data);
	}
	gps_read_timer = millis();
}

bool GPS_Read(char* gps_data)
{
    if (GPSSerial.available())
    {
        String s = GPSSerial.readStringUntil('\n');
        snprintf(gps_data, strlen(s), s);
        return true;
    }
    return false;
}

Anything obvious I am doing wrong? sometimes the GPS data does come back as a lot of strange characters but most of the time the data is sound, I am printing it and able to get a correct location.

[jgskarda]

I've done a decent amount of development on sending/receiving Lora messages on the Particle Platform. @chipmc can offer some advice as well. Here is an in depth lengthy topic covering a lot of considerations:

Long Range IoT Networks - Chapter 2 | Particle + LoRa Better together

This is intended to be a continuation of an existing closed topic: Long Range IoT Network Since, it’s closed, I thought I’d start another topic to continue the collaboration here. The intent of this thread: In the last 2-3 years, I’ve been continuing to explore the LoRa space in order to overcome several challenges in developing a Particle Powered product for a specific use case. I’ll share more details of that product/use case in another thread in the coming days/weeks to supplement this one.…

You are using the RadioHead library correct? What class level of the radiohead library are you using? Reliable Datagram?

If so, for consideration per RadioHead documentation:
Caution: if you are using slow packet rates and long packets with RHReliableDatagram or subclasses you may need to change the RHReliableDatagram timeout for reliable operations. Caution: for some slow rates nad with ReliableDatagrams you may need to increase the reply timeout with manager.setTimeout() to deal with the long transmission times.

I wonder if the GPS data is pushing you over the edge causing the LoRa Radio to timeout.

I'd try:

1. Only send GPS data and see if you can reliably et GPS data send over LoRa without every single time when the extra baggage of sensor data is removed. This will confirm you are obtaining GPS data correctly and sending it correctly.
2. Can you do a serial.Print of the full JSON Package you are sending before it's being sent. Just to make sure it's valid/good data. If you can share here that might help us identify something.
3. General advice, IMHO, LoRa is not intended for full Strings of data like you have. Rather, I'd consider boiling your data down to only what is required and pack it into bytes of data. When I send data over LoRa I use the following format

1 byte: Date Type
2 Bytes: Value

In your case, assign the following bytes to each data type. Instead of sending over the entire string of "Temperature". Simply send over the value 2 followed by 2 bytes containing the value of temperature or maybe temperature X10.

Pitch = 1
Temperature = 2
Pressure = 3
Roll = 4
Long = 5
Lat = 6
etc.

Then your LoRa message is something like this:
buf[0-4] reserved for Device ID and other info.
buf[5] = 1;
buf[6] = highByte(Pitch);
buf[7] = lowByte(Pitch);
buf[8] = 2;
buf[9] = highByte(Temperature);
buf[10] = lowByte(Temperature);
buf[11] = 3;
buf[12] = highByte(Pressure);
buf[13] = lowByte(Pressure);
....

This will VASTLY decrease your message size, allow the message to be sent quickly with less time on air, less power, and avoid any timeouts. You can pack a lot more data using this approach than sending full strings. Just for consideration.

Also, curios, what's your use case for LoRa? Low cellular signal strength in an area? Concentration of sensors in an area? Something else?

[krixen1]

Hello Jgskarda, you helped me out in the past with lora also!

I am working on a system that will need multiple avenues of communication in case cellular is down, perhaps even underground. It will keep hoping nodes until it reaches one with cloud connectivity.

I think your idea of sending bytes makes 100% sense. I started to implement that but I wanted to first try your first couple of suggestions to see if they would work.

To start, I increased the timeout on both receiver and sender to 1.5s and I still could not get the GPS data on the other side with a JSON packet, so there something wrong there.

Next, I tried to send the GPS data by itself and I did have success, the data did get across. This is something I can easily just send separately without issue since you dont want to poll for GPS constantly anyway.

So i can send the data across, just need to split it up. i wil dig in further to figure out why!

I have a more general question if you don't mind asking. in the below code, is the for loop to set all the bytes to '\0' not actually doing what I am thinking? whenever the GPS data sends, which is a smaller packet, I see the end of the previous packet.

for (int i=0; i<sizeof(lorawrite); i++)
{
	lorawrite[i] = '\0';
}

if (millis() - gps_write_timer > 30000)
{
	Serial.println(gps_data);
	gps_write_timer = millis();
	sprintf(lorawrite, "{\"gps\": \"%s\"}", gps_data);
}
else
{
	
	JSONBufferWriter writer(lorawrite, sizeof(lorawrite));
	writer.beginObject();
		writer.name("id").value(System.deviceID());
		writer.name("Pitch").value(String(pitch));
		writer.name("Temperature").value(String(temp));
		writer.name("Pressure").value(String(pressure));
		writer.name("Roll").value(String(roll));
		// if (millis() - gps_write_timer > 30000)
		// {
		// 	Serial.println(gps_data);
		// 	writer.name("GPS").value(String(gps_data));
		// 	gps_write_timer = millis();
		// }
		//writer.name("GPS").value(String(gps_data));
	writer.endObject();
}

I want to either send the full packet, or the small packet. immediately on the lora receive side, I am printing the packet I am getting across and it looks like:

{"gps": "xxxxxxxxx"}itch: xx, roll: yy....}

I thought the for loop to reset the buffer would work but it doesnt seem to be doing anything. anything obvious i am doing wrong?

[ScruffR]

Can we see the actual definition of lorawrite and not a mere abstract description?

If you want to reset the entire array (providing it actually is declared as array) you could use the faster command memset()

  memset(lorawrite, 0, sizeof(lorawrite));

When you want a nested JSON object you may want to create a new object and inject that instead of creating a "JSON string" and insert a that "wrapped" data as string object.

I'd also advise against this

@krixen1

char gps_data[512] = ""; //global
...
bool GPS_Read(char* gps_data)
{
    if (GPSSerial.available())
    {
        String s = GPSSerial.readStringUntil('\n');
        snprintf(gps_data, strlen(s), s);
        return true;
    }
    return false;
}

Having a global and a local variable share the same name is a recipe for confusion and disaster.
Also, although you use snprintf() you are only may fall pray to an illusion of safety as there is no check whether the length of you received string is less or equal than the size of your array.
It might be true for the moment, but future code rework may - for whatever reason - lead you (or someone else) to reduce the array size and that would never be checked again.
The intent of the length parameter for snprintf() is to limit the length of the input string to its own length (which would be a superfluous endeavor) but rather that the input string cannot exceed the length of the target buffer. Hence that length string should be derived from the length of the target, not the source.
You may also run into issues if your string s should happen to contain anything that may be interpreted as format string place holders since you are using that string as format string but don't provide any input data.

Having said all that, I'd suggest something along the line of this

char gps_data[512] = ""; 
...
// to be called as: GPS_Read(gps_data, sizeof(gps_data));
bool GPS_Read(char* buffer, sizt_t length) {
  if (GPSSerial.available()) {
    size_t len = GPSSerial.readBytesUntil('\n', buffer, length-1); // read bytes up to '\n' or length or a timeout occures
    buffer[len] = '\0';                                            // terminate string
    return len;                                                    // probably too little of validity check (see bellow) 
  }
  return false;
}

Additional to this, you'd probably also want to check whether the data received actually is a valid GPS sentence of proper length and correctly terminated.
stream.readBytesUntil() (just as stream.readStringUntil()) may timeout or not find the terminator or return a too long string - all these conditions want to be handled savely.
The same as you may want to discard any leading fragments of previous GPS sentences before picking up a full/valid transmission.

[krixen1]

thank you for the quick response.

I do think I am running into issues using the readBytesUntil because I often get GPs sentences that do not terminate. I will have to rethink that function a little, concat until I get to \n.

going to take your suggestion and implement it now!

Here is the lorawrite functionality that was left out.

// global buffer
char lorawrite[1023];

// radiohead manager
RH_RF95 driver(D3, D2); 
RHMesh manager(driver, CLIENT_ADDRESS);

// instance of LoRA (taken from "Particle + LoRA" thanks to chipmc and jgskarda
LoRA *LoRA::_instance;       
        
LoRA &LoRA::instance() {
    if (!_instance) {
        _instance = new LoRA();
    }
    return *_instance;
}  

// code that writes over LoRA
if (LoRA::instance().sendMessage((uint8_t*)lorawrite, strlen(lorawrite)) == RH_ROUTER_ERROR_NONE)
{
	Serial.println("sent message to server");
}
else
{
	Serial.println("sendtoWait failed");
}

[krixen1]

I decided to move back to some old code I had when using the adafruit ultimate GPS. The code looks like:

bool GPS_loop(char* buffer, size_t length)
{
    char c = GPS.read();

    if (GPS.newNMEAreceived()) 
    {
        char *nmeaData = GPS.lastNMEA();
        char* ss = strstr(nmeaData, "$GPGGA");
        if (ss != NULL)
        {
            size_t len = strlen(nmeaData);
            memset(buffer, 0, length);
            strncpy(buffer, nmeaData, len);
            return true;
        }
    }
    return false;
}

this is called once per loop very quickly. I combined some of your suggestions into this one. I am looking for GPGGA sentences specifically. This should handle a lot of the reading complications.

then on the other side I write it as:

if (GPS_loop(gps_data, sizeof(gps_data)))
{
	if (lora_connected && !is_super_node && !cloud_connected)
	{
		memset(gpslora, 0, sizeof(gpslora));
		sprintf(gpslora, "{\"gps\": \"%s\"}", gps_data);
                Serial.println(gpslora);
		if (LoRA::instance().sendMessage((uint8_t*)gpslora, strlen(gpslora)) == RH_ROUTER_ERROR_NONE)
		{
			Serial.println("gps: sent message to server");
		}
		else
		{
			Serial.println("sendtoWait failed.");
		}

	}
}

However, on the receiver side, I am getting tons of bad characters and horrible formatting. I am using memsets before writing data to keep it clean. I am not sure what I am doing wrong now. Here is a print of the data right before its sent over LoRA w/ Xs representing good gps data

♣9$GPGGA,XXXXXXXXXXXXXXXXXXXX0Kb�r�b�r�R��"9↔M♣♣��"9I5

sendtoWait failed.

[jgskarda]

Not sure why you are getting the unusual characters. I'm guessing it's type conversion from bytes to UTF-8. If you are getting bogus/unusual characters, it's likely something to do with type conversion. I had similar struggles until I went with my pattern referenced earlier. Conceptually, you should be able to send the String over and then serial print I just ran into similar scenarios. If you want more code snippets I can certainly provide that.

In the meantime... I did have a question regarding this:

@krixen1

It will keep hoping nodes until it reaches one with cloud connectivity.

How do you intend on accomplishing this? Is the cloud connected node change? Are there multiple cloud connected nodes at the same time?

Within LoRa (at least radio head libraries referenced above) you can either send a broadcast (I.e. send out a message for anyone to hear and no one re-transmits) or send a LoRa message addressed to a specific node given a 0-255 node address, additional nodes can act as routers/repeaters in between if you either pre-defined it or use RH_Mesh which implements route discovery.

If you broadcast a message, then all nodes will hear the same message. If two neighboring nodes are both cellular connected they will both publish the data to the cloud. You'll then need to de-duplicate that message in your own backend (or just deal with having duplicates in your data).

If other nodes attempt to re-transmit the broadcast message if they are not cloud connected, I'd imagine the broadcast storm wouldn't allow anything to communicate.

From my LoRa experience, if you need repeater functionality, you'll need to define what node address among all LoRa nodes is the cloud connected node. In other words, it's my understanding you can only have 1 cellular connected device among all nodes and that device must always be the same one. Or did you have something else in mind?

[krixen1]

I do have 'super nodes' that are nodes that are cloud connected. So I do attempt to send data to the super nodes and if any hops along the way have cloud connectivity, I will publish it.

I am okay with handling duplicates/trips/etc on my backend. I think at some point I may have to have these units acts as clusters where each cluster has one super node in it.

I am going to look into the unusual characters soon, i suspect you are right about utf-8 being the culprit.