I am experimenting with packing as much data as possible into a single record before publishing. The literature says a publish() can contain 622 bytes, but I am only seing 418. I am attempting to publish as many 16byte records as I can pack in. I should be able to get 38. I am only writing 23 1/2. What am I doing wrong?
What kind of data are you sending?
Unfortunately it’s not 622 bytes but characters. Not all 256 possible byte values are allowed as pay load (e.g. \0 will terminate the string).
Just ASCII and HEX.
Example:
VST-999161617557100029F0EE701010001029F0EE701010002029F0EE701010003029F0EE701010004029F0EE701010005029F0EE701010006029F0EE701010007029F0EE701010008029F0EE701010009029F0EE70101000A029F0EE70101000B029F0EE70101000C029F0EE70101000D029F0EE70101000E029F0EE70101000F029F0EE701010010029F0EE701010011029F0EE701010012029F0EE701010013029F0EE701010014029F0EE701010015029F0EE701010016029F0EE701010017029F0EE701010018029F0EE7010100
417 characters
I have been messing with it, changing the length of records, but the function that writes a record should do exactly the same each time. There should be 24 records there and its cutting off after 19.
What device OS version are you running?
2.0.1 on an Electron
Just tested that with one of my Electrons on 2.0.1 with this code
const char* payload =
"00045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n"
"10045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n"
"20045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n"
"30045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n"
"40045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n"
"50045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n"
"60045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\n";
void setup() {
}
void loop() {
if (!digitalRead(BTN)) {
while(!digitalRead(BTN));
Particle.publish("howMany", payload);
delay(1000);
}
}
And the result was this
00045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
10045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
20045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
30045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
40045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
50045678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
6004567890123456789012
So I'm pretty certain it's your code and neither the device OS nor Particle cloud cutting off your data.
BTW, if you want maximum data density you may want to opt for Base85 encoded binary data rather than
I would love to use base85 although I heard with base64, it actually increases your datasize 30%.
I am attempting to broadcast a header of 6 bytes, consisting of a 16bit ID and an timestamp and then as many records as I can smash into the remaining block the should only 8 integer bytes per record. These 8 integer bytes gets expanded into 16 ascii bytes, however. The data comes to our electron serially once every 10 seconds from a PLC and I already have to code it from ascii into integer. converting it back to hex was as far as I have gotten so far.
my brain is getting fuzzled going through all the conversion gyrations. I would love to smash it down with base85 if I could wrap my brain around the code involved and if it truly saved space. Any help would be appreciated.
It does increase data size compared to raw binary data, but when you haven't got the option to use raw binary, you have to swallow the extra demand.
But Base64/Base85 will definetly use less space than packing binary values into a string by just converting into HEX representation which will only use 16 characters out of 256 bit combinations to represent your binary numbers.
That's exactly my point: Converting to HEX representation uses double the amount (200%) needed whilch base64 would only use 30% more (130%).
unfortunately, I have not figured out how to convert directly from integer to base85 yet.
For now you could go with Base64 as there already is a library by @rickkas7 - unfortunately there isn’t one for Base85 (yet 
).
im looking at the base64RK example and unfortunately, I cannot make heads nor tails out of it.
I cannot figure out how to correctly call the function just to create a base64encoded version of my string?
That library (as do most) also features usage examples
https://build.particle.io/libs/Base64RK/0.0.1/tab/example/1-tester-Base64.cpp
peekay123 has just pointed me to a Base85 library that should be simple to incorporate in your project.
I’ll have a look into putting it into the Particle Library Repository
1 Like
That would be wonderful. Now I just have to figure out how to use it!
I haven’t tested it yet but it builds and uses the provided example as a basis for a Particle test example.
https://go.particle.io/shared_apps/6058af7c4c3ada0009fd6973
A Base85 library would of course use a class and follow the structure outlined in Rick’s Base64RK library.
1 Like
I got it working!
However, strange is that it is exactly the same size as the original.
Hex:
270F6058C06900029F0EE701010001029F0EE701010002029F0EE701010003029F0EE701010004029F0EE701010005029F0EE701010006029F0EE701010007029F0EE701010008029F0EE701010009029F0EE70101000A029F0EE70101000B029F0EE70101000C029F0EE70101000D029F0EE70101000E029F0EE70101000F029F0EE701010010029F0EE701010011029F0EE701010012029F0EE701010013029F0EE701010014029F0EE701010015029F0EE701010016029F0EE701010017029F0EE701010018029F0EE701010019029F0EE70101001A029F0EE70101001B029F0EE70101001C029F0EE70101001D029F0EE70101001E029F0EE70101001F029F0EE701010020029F0EE701010021029F0EE701010022029F0EE701010023029F0EE701010024029F0EE701010025029F0EE7010100
base85:
GB+?rHZV0fLohZuFfcGOIYux=MK>@pFflMNFflMPIYux=MK>@pFflMNFfuSQIYux=MK>@pFflMNFf%YRIYux=MK>@pFflMNFf=eSIYux=MK>@pFflMNFf}kTIYux=MK>@pFflMNFg7qUIYux=MK>@pFflMNFgGwVIYux=MK>@pFflMNFgP$WIYux=MK>@pFflMNFgY+XIYux=MK>@pFflMNFhMXfIYux=MK>@pFflMNFhVdgIYux=MK>@pFflMNFhejhIYux=MK>@pFflMNFhnpiIYux=MK>@pFflMNFhwvjIYux=MK>@pFflMNFh(#kIYux=MK>@pFflMNF)%PPIYux=MK>@pFflMNF)=VQIYux=MK>@pFflMNF)}bRIYux=MK>@pFflMNF7hSIYux=MK>@pFflMNFGnTIYux=MK>@pFflMNFPtUIYux=MK>@pFflMNFYzVIYux=MK>@pFflMNFh(WIYux=MK>@pFflMNFq<XIYux=MK>@pFflMNF*z_YIYux=MK>@pFflMNF+nggIYux=MK>@pFflMNF+wmhIYux=MK>@pFflMNF+(siIYux=MK>@pFflMNF+?yjIYux=MK>@pFflMNF-0&kIY
Actually, the Hex version is 620 characters and the base85 is 622?
Am I missing something here?
I literally had it translate my entire hex char array into base85.
char base85[2*strlen(line_buffer)];
bintob85(base85, line_buffer, strlen(line_buffer));
Particle.publish("cons_data", line_buffer.c_str(), PRIVATE);
Particle.publish("b85_data", base85, PRIVATE);
@VSTadmin, the code is designed to take in BINARY data, not ascii coded HEX characters! Feeding it ascii HEX characters will not give you the efficiencies since you have two hex chars (aka bytes) for every “raw” binary byte of data.
Im going to need a little help then. How do I feed it binary?
I work with strings all day long. I understand how to put an integer into an array, but I have no idea how to create an array of bytes and pack integers into to it without changing their values?
@VSTadmin, before you convert those header bytes (6) and 8 "integer" bytes, I assume these are raw binary data, are they not? What exactly is the format of the source data you want to send?
the source data is a string of information that comes from a PLC and is parsed out into individual values. I did manage this so far:
line = readLineFromSerial();
if (line != "") {
char *r = const_cast<char*>(line.c_str());
strGmid = strtok(r,",");
strFile = strtok(NULL,",");
strDate = strtok(NULL,",");
strTime = strtok(NULL,",");
strPress = strtok(NULL,",");
strCycles = strtok(NULL,",");
strFaultCode = strtok(NULL,",");
strFaults = strtok(NULL,",");
strMode = strtok(NULL,",");
flPress = atof(strPress) * 100 ;
intPress = (int)flPress;
char test[8];
test[0] = line_cnt;
test[1] = intPress;
test[3] = atoi(strCycles);
test[5] = atoi(strFaultCode);
test[7] = atoi(strMode);
char base85_2[2*strlen(test)];
bintob85(base85_2, test, strlen(test));
Particle.publish("b85_line", base85_2, PRIVATE);
}
It seems to be working for the smaller, 8 byte values.
@VSTadmin, can you provide an example of a few lines of data received from the PLC? Another way to parse is using the scanf() function. However, I would like to understand what the raw text from the PLC looks like.