the raw data is a JSON stream that looks like this:
{“ID”:“VST08”,“P”:-0.01,“C”:11275,“F”:0,“E”:" ",“M”:0 ,“La”:19.447460,“Lo”:-99.203918,“A”:784.000000}
this line received from the UART and processed in the function above.
You would first parse all these number literals into binary values (of the smallest required data type), place them in a struct and then encode that struct.
You will also not need to transfer the units or keys for the data as that is redundant information without value.
e.g.
struct yourValues {
char id[6];
float P;
int16_t C;
uint8_t F;
char E;
...
}
Yes, see a couple of items above. I have at least gotten the 8 bytes to base85 encode. Is it possible to still encode the entire 622 bytes at a time?
@VSTadmin, first, by using a JSON parser, you could get each value in its native format (float, etc). I have no clue what strFile or strDate mean as I can’t see any obvious correlation to the PLC data. Perhaps you can explain.
Regardless, the MOST compressed method for compressing your data is to reduce it to binary format. For example, a float value is stored in a 4-byte format, an integer with values of less than 65536 can be store in a 2-byte format, etc. By creating an array of bytes that contain the concatenated (multi-byte) values of the all the data, you will end up with the smallest “binary packet”. You can call this a “packed binary array”.
You would then feed this packed array to the base85 code to produce a valid ascii equivalent that you can then send via Particle.publish(). HOWEVER, your server would need to first decode the base85 text back to original binary and then extract each variable value, knowing their position in the array.
The easiest way to achieve the packed array is to create a byte-packed structure of all the variables you want to send. Packing tells the compiler to squeeze all the variables on byte boundaries instead of the typical 4-byte boundary common to 32-bit processors. With some pointer casting, you can then easily send the pointer to the entire structure to the base85 converter.
1 Like
Well that depends on the size of the packed structure for ONE record. You would then have to figure out the base85 overhead to tell you how many records you can send at a time.
the individual record is only 8 bytes and I am trying to get 40 records in one burst. which is 320 bytes but 640 ascii characters. of course, 640 is too much.
@VSTadmin, I’m confused how you get one record = 8 bytes from the content of the PLC data. Can you please explain your data mapping (PLC data to 8 bytes)?
char *r = const_cast<char*>(line.c_str());
strGmid = strtok(r,",");
strFile = strtok(NULL,",");
strDate = strtok(NULL,",");
strTime = strtok(NULL,",");
strPress = strtok(NULL,",");
strCycles = strtok(NULL,",");
strFaultCode = strtok(NULL,",");
strFaults = strtok(NULL,",");
strMode = strtok(NULL,",");
flPress = atof(strPress) * 100 ;
intPress = (int)flPress;
char test[8];
test[0] = line_cnt;
test[1] = intPress;
test[3] = atoi(strCycles);
test[5] = atoi(strFaultCode);
test[7] = atoi(strMode);
Some of the data is not necessary. I really only need a sequence number to track the position within the block (as they can get out of order sometimes I have seen), thats 1 byte, 2 bytes for the pressure, 2 bytes for the cycle count, 2 bytes for the faultCode and 1 bytes for the operation mode.
thats 8 bytes and I have that part working and converting to base85 by itself. Im looking at the larger structure now and how to pack more than a single record into base85. I dont know if there is any specific limit other than memory.
@VSTadmin, there's a whole bunch of wrong going on here. First, the strtok() function will return a number cstring pointers with the delimiting character being a comma. For pressure, for example, the return pointer strPress will point to the string “P”:-0.01 so that when you attempt atof(strPress), it will not return the expected value since the number to be converted is not the first character pointed to. Perhaps you meant the strtok delimiter to be ":" instead?. This issue will also apply to other strings your are trying to convert. @ScruffR, can you check me on that?
Then, doing something like this - test[1] = intPress; will not work as you expected, assuming f1Press is defined as type int16_t (which I suspect it is not). Instead, what the compiler sees is "take the signed intPress value and store it as a single byte char at the fourth byte of the char array test". NOT what you probably expected.
Have you actually reverse converted the base85 text to validate the values?
To achieve what you want, you start with a packed structure (hinted to by @ScruffR) to hold each value you want to send, which might look like:
#pragma pack(push, 1) // Byte pack so received data fits exactly
struct sendDataDef {
uint8_t line_cnt;
int16_t press;
uint16_t cycles;
uint16_t fault_code;
uint8_t mode;
} ;
#pragma pack(pop)
Now, to do a whole bunch of these records, you create an array of that structure. For example:
sendDataDef sendData[MAX_RECORDS];
Since you lose about 25% using base85 and leave a little extra room you could put about 57 records in 622 bytes. So the MAX_RECORDS is 57. So now, every time you read a record from the PLC, you put the values the structure array using, for example, sendData[record_counter].press = the calculated pressure value. Once you have reached the 57 record limit, you then send the entire array to base85 using:
char pub_msg[622];
bintob85(pub_msg, (uint8_t*)sendData, sizeof(sendData)*MAX_RECORDS);
But like I said earlier, your server will need to unpack the base85 data and know that the resulting data is an array of structures as defined above.
1 Like
its going to take me a while to digest all of this 
I certainly appreciate all the help I have received today. Concerning the strtok():
char *r = const_cast<char*>(line.c_str());
strGmid = strtok(r,",");
strFile = strtok(NULL,",");
strDate = strtok(NULL,",");
strTime = strtok(NULL,",");
strPress = strtok(NULL,",");
strCycles = strtok(NULL,",");
strFaultCode = strtok(NULL,",");
strFaults = strtok(NULL,",");
strMode = strtok(NULL,",");
the JSON formatted string is an intermediate step. The original string is a comma-delimited string that I do not have exact representation for at the moment. I have been using the strtok() function for 2 years now successfully to transform the CSV string into JSON to send to the particle cloud. The “strPress” variable will derive into a number between “-27.00 and 27.00” pressure. Normally, all I have to do is send that forward to the cloud. Now, I am converted to integer to as a new function to pack this array.
But let me continue to wade through the rest of this explanation and I will continue tomorrow.
Again, thank you so much for your assistance today!
When you receive a valid JSON string it’s probably best to use a JSON parser to extract the individual values.
As @peekay123 already mentioned, with strtok()ing for commas you will not only get the desired value but the key/value pair with all the “decoration”.
For JSON parsing you can either use this
https://docs.particle.io/reference/device-os/firmware/photon/#parsing
or Rick’s JsonParserGeneratorRK library
If your input data is not a JSON string but you know the exact order of fields you could also use sscanf() instead of strtok().
You might have misunderstood.
I do not receive JSON. I receive CSV and create JSON.
All of that has been working satisfactorily for 2 years.
I was just pointing out earlier that I am used to working with strings, but not with binary data.
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.