Particle Function with JSON Payload greater than 1024 characters?

Firmware
1

Anyone try to pass down a JSON Payload that was greater than 1024 byte limit using a Particle Function? I was looking for some guidance/inspiration on the best way to structure it.

Off the cuff, I was thinking use a Particle Function to append additional elements to an existing JSON object within the Particle Device. I.e. every time it calls the function, it’ll append the items. The concern comes up with what happens during re-tries or if one the function calls doesn’t go through but other ones do the end result won’t be complete.

Should I just create temporary JSON object and have an index counter i.e. 1 of 4, 2 of 4, 3 of 4, 4 of 4. And only accept the re-assembled JSON on the Particle Device by appending objects if and only if we recently received all 4 sub messages?

Anyone have examples I could use as inspiration or any chance there is a good library out there for this?

The end structure is ideally something like this but with up to 50 or even more members of key 22 and 23.

{
	"21": 6,
	"22": {
		"xxxxxxcec0": 1092,
		"xxxxxx9052": 1093,
		"xxxxxxb8f8": 1094,
		"xxxxxx683e": 1095,
		"xxxxxxe57": 1100,
                 ... (50 key/value pairs, maybe more)
		"xxxxxx4adb": 1103
	},
	"23": {
		"1092": [99, 10, 0],
		"1093": [99, 11, 1],
		"1094": [99, 12, 2],
		"1095": [99, 13, 3],
		"1100": [99, 14, 4],
                ... (50 key/value pairs, maybe more)
		"1103": [99, 32, 5]
	}
}
2

Hey Jeff,
another way to tackle this could be to send in chunks of config that fit the 1024 limit.
Example:
Call the function with:

"22": {
"xxxxxxcec0": 1092,
"xxxxxx9052": 1093,
"xxxxxxb8f8": 1094,

and then with:

"22": {
"xxxxxx683e": 1095,
"xxxxxxe57": 1100,

and so on.
How does it look?

1 Like
3

Yeah, that’s basically what I want to do and then append the results into a single larger JSON within the Particle Boron. What I’m trying to deal with or think through is the edge case where I need to chunk it up into say 4 different messages but only 3 out of the 4 are successful. In that case, the resulting JSON isn’t really valid or doesn’t contain all the data that I need it to.

Hypothetically I was thinking something like:

First Particle.Function()
{
	"chunkNumber": 1,
	"numberOfChunks": 2,
	"22": {
		"xxxxxxcec0": 1092,
		"xxxxxx9052": 1093,
		"xxxxxxb8f8": 1094
	},
	"23": {
		"1092": [99, 10, 0],
		"1093": [99, 11, 1],
		"1094": [99, 12, 2]
	}
}

Second Particle.Function() 
{
	"chunkNumber": 2,
	"numberOfChunks": 2,
	"22": {
		"xxxxxxce33": 1095,
		"xxxxxx9044": 1096,
		"xxxxxxb855": 1097
	},
	"23": {
		"1095": [99, 10, 0],
		"1096": [99, 11, 1],
		"1097": [99, 12, 2]
	}
}

Then the function in Particle would assemble the complete JSON once all expected chunks are received. I.e. in this case, if chunkNumber1 and chunkNumber2 are both received then append the result sets into a single JSON:

{
	"22": {
		"xxxxxxcec0": 1092,
		"xxxxxx9052": 1093,
		"xxxxxxb8f8": 1094,
		"xxxxxxcecc": 1092,
		"xxxxxx90bb": 1093,
		"xxxxxxb8dd": 1094
	},
	"23": {
		"1092": [99, 10, 0],
		"1093": [99, 11, 1],
		"1094": [99, 12, 2],
		"1095": [99, 10, 0],
		"1096": [99, 11, 1],
		"1097": [99, 12, 2]
	}
}

It would be expected that the function is called back to back to back consecutively. This would be done in a Python Script making calls to the Particle API to call the function with the JSON data.

Am I thinking about this in a reasonable way? Anyone try something like this before?

On a related question… how does one know if you reached the limit? I assume I can remove the whitespace right? Python - json without whitespaces - Stack Overflow :
json.dumps(separators=(',', ':'))

And then determine the size assuming utf-8 encoding within Python using something like:
len(JSONPayLoad.encode('utf-8'))

Any other useful tips/tricks, things to consider?

4

The function call cloud API doesn’t return until the device handles it, so the sender will know if the part was successfully received or not. The device firmware can use the int return value to alert on internal errors as well.

But you still need a way for the device to know it’s received the last segment. Including a part number and part count in the payload is best. But because of the way JSON works, it will only be able to be parsed after the last segment is received, so you could technically split the data into 1024 byte chunks with no additional part information at all.

5

@rickkas7 interesting... So if I understand what you are saying. I don't have to send it down as JSON objects and then append the JSON together, but rather I could send it down as segments of a continuous string and then concatenate the string together as it arrives and only parse it out upon the last message? That would make it a lot easier I think. Having the return statement in the cloud makes a lot of sense as well to both send in the proper order, stagger and guarantee delivery. Nice!

I wonder if I just dedicate the first byte to the chunk number and the second byte to number of chunks. When Byte 1 = Byte 2 then this is the last message in the payload:

Payload 1:

12{"21": 6,
"22": {
"xxxxxxcec0": 1092,
"xxxxxx9052": 1093,
"xxxxxxb8f8": 1094,
"xxxxxx683e": 1095,
"xxxxxxe57": 1100,
"xxxxxx4a

Payload 2:

22db": 1103
},
"23": {
"1092": [99, 10, 0],
"1093": [99, 11, 1],
"1094": [99, 12, 2],
"1095": [99, 13, 3],
"1100": [99, 14, 4],
"1103": [99, 32, 5]
}
}

Then on the particle device when the function is called, strip off the first two bytes of the message. If the first byte = 1 initialize a buffer (something like char buf[4096]) then append the remaining message to the buffer. If the second byte = first byte, we are done and the full message was received. We can now parse the final JSON. If Parse successful return some integer value back indicating success.

That should then re-assemble the full JSON properly as:

{
	"21": 6,
	"22": {
		"xxxxxxcec0": 1092,
		"xxxxxx9052": 1093,
		"xxxxxxb8f8": 1094,
		"xxxxxx683e": 1095,
		"xxxxxxe57": 1100,
		"xxxxxx4adb": 1103
	},
	"23": {
		"1092": [99, 10, 0],
		"1093": [99, 11, 1],
		"1094": [99, 12, 2],
		"1095": [99, 13, 3],
		"1100": [99, 14, 4],
		"1103": [99, 32, 5]
	}
}

Does that sound reasonable? It seems easier/cleaner than what I first was thinking. What's the easy way to "append" a string or char array to another string/char array? memcpy?

6

Yes, that sounds reasonable and a lot less work than making each segment valid JSON, which isn’t really required.

When you do publish and subscribe you have to worry about out-of-order messages and lost messages because the sender never will know that the recipient received it or not, but with functions the logic is so much easier because you know if it was received and they will always be in order, assuming you send them sequentially and wait for each response, which you should do anyway to avoid overwhelming the device with too many function calls.

1 Like
7

Very good. I’ll give this a try and post the code here when it’s functional. Really appreciate the guidance! This seems so much easier!

8

Alright, I think I got it… this was ALOT easier than I was anticipating. I simply reduced the max size within the Python script so it chunked it up into 40 bytes at a time, re-assembled on the particle device and it worked as expect. I could clean up the error handling a bit but functionality is there.

Thank you so much for the guidance! I would of spent many hours or even days on the full JSON parse and append method. Here’s the code snippets I used. I’m always open to ideas/feedback if there is an even cleaner way.

Python Code in my backend:

#Convert Python Dict to JSON, specify the operators to reduce whitespace. 
strMsg = json.dumps(msg,separators=(',', ':'))

# Now chunk it up if needed. Store each chunk into an array. 
maxPubSize=1024
chunks = [strMsg[i:i+maxPubSize] for i in range(0, len(strMsg), maxPubSize)]
numChunks = len(chunks)
  
#We do not need to chunk it up, call the normal set Config function we use today
if numChunks == 1:
    data = {
        'arg': strMsg,
        'access_token': Config.PARTICLE_TOKEN
    }

    response = requests.post('https://api.particle.io/v1/devices/'+ hub.device_id +'/setConfig', data=data)
    dictFromServer = response.json()
    if "return_value" in dictFromServer:
        returnJSON = {'return_value':dictFromServer['return_value']}
        print(returnJSON)
  
elif numChunks > 1:
    #We need to chunk it up when we send it. First append the chunk number and number of chunks in total. 
    strToSend = []
    for index, chunk in enumerate(chunks):
        strToSend.append(str(index+1)+str(numChunks)+chunk)
        print(strToSend[index])
        
        data = {
        'arg': strToSend[index],
        'access_token': Config.PARTICLE_TOKEN
        }

        response = requests.post('https://api.particle.io/v1/devices/'+ hub.device_id +'/setConfigChunked', data=data)
        dictFromServer = response.json()
        if "return_value" in dictFromServer:
            returnJSON = {'return_value':dictFromServer['return_value']}
            print(returnJSON)

And on the Particle Device side:

 /*******************************************************************************
 * Function Name  : setConfigChunked()
 *******************************************************************************/
int config::setConfigChunked(String configJSON){
  Log.info("Set CloudConfig with Chunked Data: %s", configJSON.c_str());

  String chunkIndexStr = configJSON.substring(0,1);
  String numChunksStr = configJSON.substring(1,2);
  String msgStr = configJSON.substring(2);

  int chunkIndex = chunkIndexStr.toInt();
  int numChunks = numChunksStr.toInt();
  Log.info("This is %i of %i:", chunkIndex, numChunks);
  Log.print(msgStr.c_str());

  //If this is the first message, then we need to set the Temp message to the content of this to start. 
  if(chunkIndex == 1){
    ConfigTemp = msgStr;
  } else {
    //If it's not the first message, then we can concatenate future messages to what we already received. 
    ConfigTemp.concat(msgStr);
  }
  
  // If the chunk index is not equal to the total number of chunks, then we are not done. Return here and wait for another message. 
  if (chunkIndex != numChunks){
    return chunkIndex;
  } 

  // The chunk index is equal to the total number of chunks, this must be the last message. Let's continue to parse the assembled string. 
  else {
    Log.info("The full configuration string recieved is:");
    Log.print(ConfigTemp);
  
    jsonParser.clear();
    jsonParser.addString(ConfigTemp);    

    //Now let's try to parse the message, if we are successful then it's a valid JSON. 
    if(!jsonParser.parse()){
      Log.info("The Message parsing failed. Must be an invalid JSON");
      return -1;
    }
    Log.info("The Message was successfully parsed"); 

    //Finally, save the config to the Flash File System. If it doesn't exist create it, truncate it to length 0, then write the data to the file and close it. We can read it directly from flash in setup. 
    int fd = open("Config1", O_RDWR | O_CREAT | O_TRUNC);
    if (fd != -1) {
        write(fd, ConfigTemp.c_str(), ConfigTemp.length());
        close(fd);
    }

    //Return 0 indicating success, seems weird to return 0 for success but positive integer values correspond to the message index, negative values are errors. 
    return 0; 
  }
2 Likes
9

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.