JsonParserGeneratorRK question

chipmc · November 7, 2022, 3:48pm

All, I am getting my head around @rickkas7’s amazing JSON library. I plan to use it in my LoRA / Particle Gateway code to do three things:

Construct the JSON payloads for Particle publish (use snprintf today)
Interpret JSON formatted commands via a Particle function to configure the gateway and the nodes via API calls
Keep track of the node configurations: node number, deviceID, last connected and sensorType

The idea is to store the JSON objects as strings in the persistent storage using the StorageHelper library. The question is how to organize the data on #3 above.

I have worked through the example code provided but I am having a bit of an issue working with JSON objects that are more than a single level deep. For example, I would like to organize my node data like this:

{
   "Node1":
       {
           "nodeNumber":1,
           "deviceID":"particleDeviceID",
           "lastConnect":1667835489,
           "sensorType":1
       },
   "Node2" .....
}

Will this work? I will need to be able to do the following:

Add new nodes as they join the network (up to some limit for the JSON token size)
Iterate through all the nodes to see if a deviceID had already been registered
Make updates and keep the nodeNumbers consistent even when updated values are moved to the end of the JSON token.
Using a given NodeNumber, (but as a variable not a string), update the values associated with that node.

Many of the tools in this library seem to be oriented toward single-layer JSON tokens (getOuter…) so, would it be better to have a separate JSON token for each of the four variables? If so, as I update some of the values, their index will change so how can I keep them associated together.

That said, if there is a better way to do this, I would be open to any suggestions.

Thanks,

Chip

gusgonnet · November 7, 2022, 4:14pm

Hi Chip, I think this can be done with the following code:

    char jsonBuffer[500]; // adjust to your needs please

    memset(jsonBuffer, 0, sizeof(jsonBuffer));
    JSONBufferWriter writer(jsonBuffer, sizeof(jsonBuffer));

    writer.beginObject();
    writer.name("Node1");

    writer.beginObject();
    writer.name("nodeNumber").value((yourNodeNumberVariable);
    writer.name("deviceID").value(deviceId);
    writer.name("lastConnect").value(yourLastConnectVar);
    writer.name("sensorType").value(yourSensorTypeVar);

    writer.endObject();
    writer.endObject();

// repeat for another node

but now that I write the last sentence, I realize that we need a json array to contain the nodes, something like this, but I have not tried it:

    char jsonBuffer[500]; // adjust to your needs please

    memset(jsonBuffer, 0, sizeof(jsonBuffer));
    JSONBufferWriter writer(jsonBuffer, sizeof(jsonBuffer));

    writer.beginArray();

    writer.beginObject();
    writer.name("Node1");
(...)
// add all your nodes here as shown above

    writer.endArray();

Again, untested.

Json does not care about the order, so this can be achieved.

I believe all the other questions have a yes answer.

I would shorten the words inside json, for instance n instead of node, id instead of deviceId and so on.

Best

chipmc · November 7, 2022, 10:30pm

@gusgonnet ,

Thank you for your help on this. So, with some confidence from your note, I have started implementing the structure outlined above (with shorter names) and made some progress. My plan is to put a structure for the JSON array in memory and then use the “addString()” command to take it from persistent memory into a JSON object at startup. Whenever the object is updated, I will save the modified string back to the persistent memory.

I am running into a couple roadblocks and I hope that someone will know how to proceed. When nodes connect to the gateway the 1st time, they will share their deviceIDs. The gateway will then need to check to see if a node number has already been assigned to that deviceID or if a new assignment is needed. In order to figure that out, I was thinking the best way is to iterate over the array and test the deviceID key value for each node object. I can see how to do this for a single - level array using the .getValueByIndex() function but this does not work if the JSON is an array of objects. For this, I believe I need to use the .getValueTokenByIndex() command but there is no example of this command in the documentation:

http://rickkas7.github.io/JsonParserGeneratorRK/class_json_parser.html#a39d613e94d0d6beafe908159f86bc067

Here is where I am now - based on some sample code from @ScruffR in another link:

#include "Particle.h"
#include "JsonParserGeneratorRK.h"

SerialLogHandler LogHandler;

bool runTest();

SYSTEM_MODE(MANUAL);

const char * const nodeData = "{\"n1\":{\"node\":1,\"dID\":\"000000000000000000000000\",\"last\":0000000000,\"type\":0},\"n2\":{\"node\":2,\"dID\":\"000000000000000000000001\",\"last\":0000000001,\"type\":1}}";


JsonParserStatic<512, 40> parser;  // Global parser that supports up to 512 bytes of data and 40 tokens - about 10 devices

void setup() {
  delay(3000);                        // Give the serial port time to connect
  if (runTest()) Log.info("Passed!");
  else Log.info("Failed");
}

void loop() {}

bool runTest() {

  Log.info("Running Part 1 - Load and Parse JSON");
	// Clear the parser state, add the string test2, and parse it
	parser.clear();
	parser.addString(nodeData);
	if (!parser.parse()) {
		Log.info("parsing failed: nodeData");
		return false;
  }

  // This line works - I can extract the values from the JSON array
  Log.info("Data stored for node %d is deviceID: %s, lastConnect %u, sensorType: %d", parser.getReference().key("n1").key("node").valueInt(), parser.getReference().key("n1").key("dID").valueString().c_str(), parser.getReference().key("n1").key("last").valueInt(), parser.getReference().key("n1").key("last").valueInt());

  // Here is where I try to iterate through the array
  // Based on sample code from @Scruffr
  const JsonParserGeneratorRK::jsmntok_t *node;
  if (!parser.getValueTokenByKey(parser.getOuterToken(), "node", node)) {     // extract the "nodes" object from the root object
    Log.info("node not found");  // ********** This is where it fails *************
    return false;
  }
  
  const JsonParserGeneratorRK::jsmntok_t *nodeArray;
  if (!parser.getValueTokenByIndex(node, 1, nodeArray)) {                     // extract the first object in the "columns" object (assuming it's our array)
    Serial.println("node array not found");
    return false;
  }

  for (uint8_t i=0; i < parser.getArraySize(nodeArray); i++) {                       // iterate over the array
    int val;
    parser.getValueByIndex(nodeArray, i, val);                                   // get the data (assuming it's and int)
    Log.info("%d: %d", i, val);                                      
  }

  return true;

}

I will, of course, keep plugging away but, any suggestions?

Thanks,

Chip

gusgonnet · November 8, 2022, 1:12am

Another alternative is to store the object in persistent memory in a struct type, which may be even more efficient. Arrays may complicate a bit the size, though. Then, you need to only write code to convert it to JSON when you want to send it out (or maybe that is not the intention with JSON here?).
In arrays (a struct), it would be also easier to traverse to modify the info…
Please tell me more.
Gustavo.

chipmc · November 8, 2022, 3:01am

@gusgonnet ,

Yes, I could store it as an object but I am trying to abstract the storage technology away and use the StorageHelper to provide access to data stored in persistent memory. I have this working but it is not scalable because you cannot easily iterate over all the nodes a gateway may be servicing. Each gateway installation could have a different number of nodes associated with it. Without the functionality of the “index” which you get with JsonParserGenerator, you end up with arbitrarily long switch:case statements or nested ifs. That is what caused me to want to figure this library out.

Also, without the ability to parse command input from a Particle function, you end up with a bunch of Particle function definitions each to set a specific setting on an arbitrary number of nodes. Again, the ability to parse a JSON payload of unspecified size allows for a single function that can take commands to configure the gateway and the nodes.

This is why I am undertaking the effort to figure out this library. I think I am close but there are some nuances that I am struggling to figure out without more detailed documentation or more example code.

Hope this makes sense and thank you for engaging on this topic.

Chip

gusgonnet · November 8, 2022, 4:12am

okay.

Imagine the json that we want to parse is like this (I added the array brackets ):

chipmc:

{
  "nodes":[
    {
      "Node1":{
        "nodeNumber":1,
        "deviceID":"particleDeviceID",
        "lastConnect":1667835489,
        "sensorType":1
      }
    },
    {
      "Node2":{
        
      }
    }
  ]
}

I used in the past the json parser like this (docs here):

  JSONValue outerObj = JSONValue::parseCopy(string_to_parse);
  JSONObjectIterator iter(outerObj);

  while (iter.next())
  {
    if (iter.name() == "nodes")
    {
      JSONArrayIterator arrayIter(iter.value());
      Log.info("number of nodes: %d", arrayIter.count());

      while (arrayIter.next())
      {
           // arrayIter.name() should be "Node1" I hope
           if (!arrayIter.value().isObject())
          {
               Log.error("oh oh");
          } else {

             JSONObjectIterator objectIter(arrayIter.value());
             while (objectIter.next())
             {
                 if (objectIter.name() == "nodeNumber")
                 {
                  Log.info("nodeNumber: %d", objectIter.value()toInt());
                 }
                 if (objectIter.name() == "deviceID")
                 {
                  Log.info("deviceID: %s", objectIter.value().toString().data()); // << not 100% sure about this .data()
                 }


// and so on

coding that on the fly - apologies…
does this help?

rickkas7 · November 8, 2022, 12:32pm

I would structure it like this:

{
    "nodes":[
         {
             "nodeNumber":1,
             "deviceID":"aaaaaaaaaaaaaaaaaaaaa1",
             "lastConnect":1667835489,
             "sensorType":1
         },
         {
             "nodeNumber":2,
             "deviceID":"aaaaaaaaaaaaaaaaaaaaa2",
             "lastConnect":1667836000,
             "sensorType":2
         }
    ]   
  }

The reason is that if you wanted to add things other than nodes you can add them to the outer structure easily.

Since you already encode the node number within the node object, and there can be zero or more of them, using an array is better than storing them in an object.

Here’s sample code to read this structure:

github.com

rickkas7/JsonParserGeneratorRK/blob/master/test/JsonTest.cpp#L760


      
          
          
		const JsonParserGeneratorRK::jsmntok_t *nodesArrayContainer;
          
          
		bResult = jp.getValueTokenByKey(jp.getOuterObject(), "nodes", nodesArrayContainer);
          		assert(bResult);
          
          
		assert(bResult);
          
          
		const JsonParserGeneratorRK::jsmntok_t *nodeObjectContainer;
          
          
		nodeObjectContainer = jp.getTokenByIndex(nodesArrayContainer, 0);
          		assert(nodeObjectContainer != NULL);
          
          
		int nodeNumber;
          		String deviceID;
          		int lastConnect;
          		int sensorType;
          	
          		bResult = jp.getValueByKey(nodeObjectContainer, "nodeNumber", nodeNumber);
          		assert(bResult);
          		assert(nodeNumber == 1);

chipmc · November 8, 2022, 3:37pm

@rickkas7

Thank you for this! With this help and some of the other examples on the link you provided, I think I am on my way. One quick question.

My plan is to create a String object function using StorageHelper - this will enable me to store updates to this array in persistent memory.

When I run this code, I get an error on the free(data) line:

		JsonParserStatic<1024, 50> jp;
		String s;

		const char * const data = "{\"nodes\":[{\"nodeNumber\":1,\"deviceID\":\"aaaaaaaaaaaaaaaaaaaaa1\",\"lastConnect\":1667835489,\"sensorType\":1},{\"nodeNumber\":2,\"deviceID\":\"aaaaaaaaaaaaaaaaaaaaa2\",\"lastConnect\":1667836000,\"sensorType\":2}]}";

		jp.addString(data);
		free(data);

commenting out free will clear the error but, I am a bit worried about how many places this large string is stored in memory - StorageHelper, data and again for the JSON array.

Does this make sense or, will I end up with memory fragmentation? Also, why does the free(data) give this error:

/Users/chipmc/Documents/Maker/Particle/Utilities/JSON-Parser-Test//src/JSON-Parser-Test.cpp:32:8: error: invalid conversion from 'const void*' to 'void*' [-fpermissive]
   32 |   free(data);
      |        ^~~~
      |        |
      |        const void*

Thanks as always for your amazing help.

Chip

rickkas7 · November 8, 2022, 4:22pm

Since the string data is hardcoded and const, that copy only exists in flash, so you can’t free it, and it doesn’t use any RAM.

chipmc · November 8, 2022, 4:33pm

Thank you @rickkas7!

chipmc · November 8, 2022, 10:52pm

@gusgonnet ,

I wanted to show you what the “finished” product looks like - this is why I undertook the effort to learn how to use this library (with your and Rick’s help of course).

Here is what I have now - notice this code is only good for up to three nodes - more nodes would further bloat the code:

uint8_t LoRA_Functions::findNodeNumber(const char* deviceID) {

	if (strncmp(deviceID,nodeID.get_deviceID_1(),24) == 0) {
		Log.info("deviceID on file - retreiving nodeID number %d", nodeID.get_nodeNumber_1());
		nodeID.set_lastConnection_1(Time.now());
		return 1;
	}
	else if (strncmp(deviceID,nodeID.get_deviceID_2(),24) == 0) {
		Log.info("deviceID on file - retreiving nodeID number %d", nodeID.get_nodeNumber_2());
		nodeID.set_lastConnection_1(Time.now());
		return 2;
	}
	else if (strncmp(deviceID,nodeID.get_deviceID_3(),24) == 0) {		
		Log.info("deviceID on file - retreiving nodeID number %d", nodeID.get_nodeNumber_3());
		nodeID.set_lastConnection_1(Time.now());
		return 3;
	}
	else {
		if (nodeID.get_lastConnection_1() == 0) {
			Log.info("deviceID not on file - saving as nodeID number %d", nodeID.get_nodeNumber_1());
			nodeID.set_nodeNumber_1(1);
			nodeID.set_deviceID_1(deviceID);
			nodeID.set_lastConnection_1(Time.now());
			return 1;
		}
		else if (nodeID.get_lastConnection_2() == 0) {
			Log.info("deviceID not on file - saving as nodeID number %d", nodeID.get_nodeNumber_2());
			nodeID.set_nodeNumber_2(2);
			nodeID.set_deviceID_2(deviceID);
			nodeID.set_lastConnection_2(Time.now());
			return 2;
		}
		else {
			Log.info("deviceID not on file - saving as nodeID number %d", nodeID.get_nodeNumber_3());
			nodeID.set_nodeNumber_3(3);
			nodeID.set_deviceID_3(deviceID);
			nodeID.set_lastConnection_3(Time.now());
			return 3;
		}	
	}
	return 0;
}

and here is what I looks like now - will work no matter how many nodes I have on the gateway (as long as the size of the JSON object is big enough of course.

byte findNodeNumber(const char* deviceID) {
	String nodeDeviceID;
	int nodeNumber;
	int index = 0;

	const JsonParserGeneratorRK::jsmntok_t *nodesArrayContainer;			// Token for the outer array
	jp.getValueTokenByKey(jp.getOuterObject(), "nodes", nodesArrayContainer);
	const JsonParserGeneratorRK::jsmntok_t *nodeObjectContainer;			// Token for the objects in the array (I beleive)

	for (int i=0; i<10; i++) {												// Iterate through the array looking for a match
		nodeObjectContainer = jp.getTokenByIndex(nodesArrayContainer, i);
		if(nodeObjectContainer == NULL) break;								// Ran out of entries - no match found
		jp.getValueByKey(nodeObjectContainer, "deviceID", nodeDeviceID);	// Get the deviceID and compare
		if (nodeDeviceID == deviceID) {
			jp.getValueByKey(nodeObjectContainer, "nodeNumber", nodeNumber);// A match!
			return nodeNumber;
		}
		index++;															// This will be the node number for the next node if no match is found
	}

	JsonModifier mod(jp);

	mod.startAppend(jp.getOuterArray());
		mod.startObject();
		mod.insertKeyValue("nodeNumber", (int)index);
		mod.insertKeyValue("deviceID", "aaaaaaaaaaaaaaaaaaaaa3");
		mod.insertKeyValue("lastConnect", (int)1667836001);
		mod.insertKeyValue("sensorType", (int)1);
		mod.finishObjectOrArray();
	mod.finish();
	return index;
}

system · December 9, 2022, 8:53am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.