Hi All.
I am looking for the most efficient way to split a string into an array of strings with a delimiter.
Any suggestions?
Tread carefully…
There’s a whole host of String related info in the docs https://docs.particle.io/reference/firmware/photon/#substring-
depending on what you are doing you might want to use substring, indexof and maybe reserve() - -Particle some more examples of this in use elsewhere in the docs might be nice!.
However be very aware that use of String objects can cause heap fragmentation causing the Photon to run out of memory and stop responding before eventually rebooting.
It might be safer, providing you know the max size of your string to make your string an array of char and extract your substrings in a very tedious manner by looping through it. More work for you, but is probably what all the nice string functions are doing behind the scenes anyway.
1 Like
what does your string look like?
Sure that is the right advice, IMHO, but we are "stuck" with them as some native Particle functions depend on String exclusively (i.e. Particle.Function(), I wonder why that choice was made...). manipulating the String into a C string is however, somewhat trivial.
1 Like
We are indeed, I would be very interested to see what happens to the memory in a Particle device if you call functions A LOT, presumably someone sufficiently technically savvy with JTAG etc could show us what happens. Personally I’m not sufficiently up on the subject to get a complete handle on exactly what you can do with a String safely, is a String as part of Particle.function actually handled as a const and therefore a char[ ]?
1 Like
same here, I've not gone that deep into the source code for function() as they just work. My suspicion is that the String in that each instance of a Particle.function() holds a buffer of a **reserve()**d nature in the heap.
A cloud function is set up to take one argument of the String datatype. This argument length is limited to a max of 63 characters.
1 Like
I’d say the most efficient way would be strtok().
Since you want an array of strings anyway, you can copy the String object into a char array, let strtok() do its job and in the process keep the resulting char* for each substring and a char pointer array.
strtok() will break up the original string by putting a zero-terminator in place of any delimiter and returning the starting address of each found substring (ignoring consecutive delimiters).
Currently Particle is investigating ways to circumvent the heap fragmentation issue around String objects.
The problem arises with long running programs (without reset/deep sleep) and the use of heavily mutating (growing, shrinking, relocating) strings since currently there is no garbage collection/defragmentation implemented in system.
If you need to use String objects, having them declared with a big enough starting size should prevent the need for relocation which would leave a possibly “lost” fragment behind.
4 Likes
Thank you all…
Will take a deep dive into this
Hi, I have tried to implement a more generic String splitter using the strtok, as I find it very important to many of my projects. And I would rather not work with String objects due to memory considerations.
However, I cannot seem to get the atoi() to work!?
const unsigned int BUFFER_SIZE = 64; // Maximum of 63 chars in an argument. +1 to include null terminator
char paramBuf[BUFFER_SIZE]; // Pre-allocate buffer for incoming args
bool success = Particle.function("cloudFun", cloudFun);
void setup() {
// Nothing.
}
void loop() {
// Nothing, just waiting to be commanded by the mighty cloud.
}
/*
Take a String argument and split it into three integers.
The function is intentionally NOT optimized with a for loop as variable types differ in the real code.
*/
int cloudFun (String paramStr) {
// Convert String to char array.
paramStr.toCharArray(paramBuf, BUFFER_SIZE);
Particle.publish("Chararray", paramBuf); // Publish to verify.
// Split string by the ',' delimiter. Expect three int.
char *pch = strtok(paramBuf, ","); // Create string tokenizer.
Particle.publish("tok1", pch); // Verify first token.
int var1 = atoi(pch); // Convert first token to int.
Particle.publish("var1", var1); // Verify conversion [shows as NULL].
pch = strtok (NULL, ",");
int var2 = atoi(pch);
pch = strtok (NULL, ",");
int var3 = atoi(pch);
}
For the code above, i get a cloud function I run with the arguments “12,34,56”, and I get the following:
So here is what I have reasoned: The strtok() works, but I do not understand the atoi()… Any help as to why this “obvious” example does not work?
the best advice is to use Serial to debug your subscribe handler. so try that and verify that you are not corrupting the buffers...
NOTE 2: Particle.publish() and the Particle.subscribe() handler(s) share the same buffer. As such, calling Particle.publish() within a Particle.subscribe() handler will wipe the subscribe buffer! In these cases, copying the subscribe buffer's content to a separate char buffer prior to calling Particle.publish() is recommended.
2 Likes
That is an interesting point! But I copy the input with:
paramStr.toCharArray(paramBuf, BUFFER_SIZE);
I will plug the board to my computer and do Serial debug, just to be sure…
also double check your tokenizing:
char myBuffer[] = "33,44,55";
int value1 = atoi(strtok(myBuffer, ","));
int value2 = atoi(strtok(NULL, ","));
int value3 = atoi(strtok(NULL, ","));
Serial.println(value1);
Serial.println(value2);
Serial.println(value3);
output:
33
44
55
3 Likes
OK, it was totally something with that! How implicit… 1e6 thanks to you!
For others to freely use, here’s the working code:
const unsigned int BUFFER_SIZE = 64; // Maximum of 63 chars in an argument. +1 to include null terminator
char paramBuf[BUFFER_SIZE]; // Pre-allocate buffer for incoming args
bool success = Particle.function("cloudFun", cloudFun);
void setup() {
Serial.begin(9600);
}
void loop() {
static unsigned int counter = 0;
Serial.printlnf("Heartbeat: %d", ++counter);
delay(1000);
}
/*
Take a String argument and split it into three integers.
The function is intentionally NOT optimized with a for loop as variable types differ in the real code.
*/
int cloudFun (String paramStr) {
// Convert String to char array.
paramStr.toCharArray(paramBuf, BUFFER_SIZE);
// Split string by the ',' delimiter. Expect three int.
char *pch = strtok(paramBuf, ","); // Create string tokenizer.
int var1 = atoi(pch); // Convert first token to int.
Serial.printlnf("var1: %d", var1);
pch = strtok (NULL, ",");
int var2 = atoi(pch);
Serial.printlnf("var2: %d", var2);
pch = strtok (NULL, ",");
int var3 = atoi(pch);
Serial.printlnf("var3: %d", var3);
return 1;
}
1 Like
Just to clarify the reason behind the problem you saw.
Although I wonder how this code actually build (I get an error message when I try) the reason is that you are trying to provide an int as payload/data parameter to Particle.publish().
But the payload has to be a const char* or String. Since you provided an int another overload was chosen that does not carry payload.
Maybe this one (but to know exactly, we'd need to know what system version you targeted)
inline particle::Future<bool> publish(const char *eventName, PublishFlags flags1, PublishFlags flags2 = PublishFlags())
{
return publish(eventName, NULL, flags1, flags2);
}
BTW, your Particle.function() handler should always return a sensible value. Your implementation does lack the return statement completely.
2 Likes
I targeted a photon from the online particle IDE.
On the return comment: whoops, it was in my code, but somehow slipped out of the MWE I posted here…
I have edited the above code to include the return statement!
Thanks a bunch everyone.
1 Like
Hello !!
I hope and you can help me
I am working on a sensor network project that counts on 5 nodes, each one measuring temperature, humidity, CO and CO2. each node sends a string with this info (as seen in the figure).
the idea is to separate each varible in order to use Particle.varible () and from there take them to ubidots.
I leave my code
Thank you!
What exactly is your issue?
You can use strtok() to tokenise your incoming string - or even sscanf() may work (as long you don’t try to parse floating point variables directly).
BTW, you can send variables to Ubidots without the need for Particle.variable()
How is the Xbee code you using working out?
I’m wanting to connect some XBee 900 radios in Digimesh mode to test with a Photon but have not done this before. I have successfully used the Xbee 900 radios before with then XTC windows software though.