EEprom get function not getting the right amount of bytes?

The output is a print out after I did an EEPROM.clear(). I am expecting 232 byte length of non printable char for both variables. But as you can see heatSched length is 233 vs coolSched length is 232. So I am puzzle.

Even when I fill both buf with data it still give me 233 byte for heatSched.

C:\WINDOWS\system32>particle serial monitor
Opening serial monitor for com port: "COM3"
System firmware version: 0.6.0
heatSched (from setup) is: ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
heatSched length is 233
coolSched (from setup) is: ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
coolSched length is 232
init
publish success
String Data received 1,04502000200020002000200020002002020002000200020002000200020002003020002000200020002000200020002004020002000200020002000200020002005020002000200020002000200020002006020002000200020002000200020002007020002000200020002000200020002000
data length heat 232
heatSched length 232
heatSched 1,04502000200020002000200020002002020002000200020002000200020002003020002000200020002000200020002004020002000200020002000200020002005020002000200020002000200020002006020002000200020002000200020002007020002000200020002000200020002000
String Data received 1,04502000200020002000200020002002020002000200020002000200020002003020002000200020002000200020002004020002000200020002000200020002005020002000200020002000200020002006020002000200020002000200020002007020002000200020002000200020002002
data length cool 232
coolSched length 232
coolSched 1,04502000200020002000200020002002020002000200020002000200020002003020002000200020002000200020002004020002000200020002000200020002005020002000200020002000200020002006020002000200020002000200020002007020002000200020002000200020002002

Output after a reset:

C:\WINDOWS\system32>particle serial monitor
Opening serial monitor for com port: "COM3"
System firmware version: 0.6.0
heatSched (from setup) is: 1,04502000200020002000200020002002020002000200020002000200020002003020002000200020002000200020002004020002000200020002000200020002005020002000200020002000200020002006020002000200020002000200020002007020002000200020002000200020002000
heatSched length is 233
coolSched (from setup) is: 1,04502000200020002000200020002002020002000200020002000200020002003020002000200020002000200020002004020002000200020002000200020002005020002000200020002000200020002006020002000200020002000200020002007020002000200020002000200020002002
coolSched length is 232

Data is store by the following handler:

void schedHandler(const char *event, const char *data)
{
  Serial.printlnf("String Data received %s",data);
  HCflag = String(data[231]).toInt();
  if (HCflag == 0 || HCflag == 1)
  {
    Serial.printlnf("data length heat %d", String(data).length());
    strcpy(heatSched, data);
    EEPROM.put(0, heatSched);
    Serial.printlnf("heatSched length %d", String(heatSched).length());
    Serial.printlnf("heatSched %s",heatSched);
  }

  if (HCflag == 2 || HCflag == 3)
  {
    Serial.printlnf("data length cool %d", String(data).length());

    strcpy(coolSched, data);
    EEPROM.put(232, coolSched);
    Serial.printlnf("coolSched length %d", String(coolSched).length());
    Serial.printlnf("coolSched %s",coolSched);
  }
}

@ScruffR, do I still need a zero terminated if I use particle.publish() ? The arg is a String.

Hmm… I get 233 for both lengths when I run your code, but if actually count the number of those symbols, I get 232.

That is perfectly normal IMHO.

As I said String looks for a zero-terminator. If your buffer doesn't contain one the String will expand as long as it needs to find that terminating '/0'.

To understand the different results in length you also need to consider the memory padding on 32bit systems and how unused bytes are initialised by the system in flash and RAM.

@Ric, yes, the symbols is correct but why when I print the length its getting an extra byte. @ScruffR, what is perfectly normal? The bigger problem is I notice some of my other variable is changing by itself. When I search on forums, it said maybe do to address corruption so I am trying to locate the source. Maybe I am storing the variable wrong.

In order to see what I mean try this first

char zero = '\0';
char a[8] = { 'a','b','c','d','e','f','g','h'};
char i[8] = { 'i','j','k','l','m','n','o','p'};
char q[8] = { 'q','r','s','t','u','v','w','x'};
char z = 'z';

String A = String(a);
String I = String(i);
String Q = String(q);

Serial.print(A); Serial.printlnf(" %d", A.length());
Serial.print(I); Serial.printlnf(" %d", I.length());
Serial.print(Q); Serial.printlnf(" %d", Q.length());

and then

char zero = '\0';
char a[9] = { 'a','b','c','d','e','f','g','h'};
char i[9] = { 'i','j','k','l','m','n','o','p'};
char q[9] = { 'q','r','s','t','u','v','w','x'};
char z = 'z';

String A = String(a);
String I = String(i);
String Q = String(q);

Serial.print(A); Serial.printlnf(" %d", A.length());
Serial.print(I); Serial.printlnf(" %d", I.length());
Serial.print(Q); Serial.printlnf(" %d", Q.length());

There may even be a difference whether you declare the arrays global or local in a function.

@ScruffR, Char zero and Char z is extra right? I ran both cases with it and without it and got the same result. The only thing I notice from the above example is the extra byte of buf in the second case. I got the result of length of 8 for the three buf in the second case. The first case I got length 24, 18, and 10. Does the result some right? From the example, zero terminator seem to be automatic unless I am doing something wrong with your code. I just copy your code into two different function and compile it and let it run and look at the output.

The results sound somewhat right (for locals the results are less predictable than for globals), although I expected slightly different numbers for case 1 (25, 17, 9 on a clean stack).
But are you not puzzled about these "unexpected" lengths. Have you also looked at the printed strings?
Although we do String(a) we don't get only the contents of a[] in the A.
If you swap the zero and the z line around, you should see slightly different results for case 1 again.
Also if you change z and zero to int and initialise both with =0 your result may change again.

Not exactly. That's a "lucky" result of two factors unrelated to strings.

  1. Particle made sure that the global variables area is zeroed out on startup, but that does not apply to local variables and may not be the behaviour on all platforms.
  2. Memory padding (im combination with 1.) - every variable declared will have a base location on a 4-divisable address. This means that the [8] arrays are placed back to back, but the [9] arrays have a gap of 3 byte which will be inaccessible without violating array boundaries, but if you didn't have point 1. (as with local variables) you might get random characters in your string and depending on the previous call stack, you may even have the string stretch across the other buffers or not - wherever the first zero-terminator will be found.

If you add an extra character to the [9] arrays, you will see that the zero-termination does not come from within the array (since there is no room for that), but from the padding.

Once you understand the things that happen on the bare metal your 232 vs. 233 puzzle can be explained and counter measures need to be applied.

@ScruffR, when I first ran your example I was puzzled and it was interesting to see it on the screen. This could explain why one of my other variable is changing automatically at random values. It maybe due to the array overflow.

To counter this issue, should I add ‘/0’ to the data being send from particle.publish before storing it? Or should I do it for all char array for safety measure by manually adding ‘/0’ to the end of the array?

char arrays are not automatically null-terminated

char myArray[232];  // not null terminated

but string literals are:

char myArray[232] = "";  // null terminated 
1 Like
char myArray[232] = "";  // null terminated

Any string literal (".....") will be zero-terminated by the compiler just as all string functions (when correctly used) will ensure a zero-term string.
But to be precise the above code only puts a '\0' character at myArray[0] but doesn’t touch the rest of the array, so once you put another byte at [0] you’ll be dealing with a potentially unterminated string again.
So it very much depends on the way you fill up your array, but you could do

  memset(myArray, 0, sizeof(myArray));

to fill the whole array with zeros and make sure to only fill it up to [230] to keep the final zero-terminator intact.

1 Like

right, and agreed if you are manipulating the string with the (I’ll call them) standard c-string functions, it knows all about taking care of the terminator!

start mucking around with the array elements and treating it like a common array… well, beyond there be dragons.

2 Likes

Is Particle.subscribe data argument automatically null-terminated? It seem like it is but just want to make sure. I am using strcpy in the firmware and strcpy look for NULL. It seem the strcpy is causing all my problems.

Once it comes in it will be zero-terminated, but if you are strcpy()ing the string into a char[] that array has to be sized [strlen(data)+1] to allow for the zero-term (which is not counted by strlen())

Can you show your code snippet?

It was interesting to dive into the nitty-griiti here of memory with your little test. Why would you expect 25 for A? Because of z, and a zero in the next address location after that z? I assume the RAM set to all zeros on startup (is that what you mean by a clean stack)?

What I see is 24, but the printout of A has 26 characters; the 2 extra characters are always the same (after resets, or power down, then back up). This seems to have something to do with the creation of A. If I print out a right after the array lines, it only prints out the 24 chars from a to x, but if I print it out after A is created, it now prints the same 26 chars that A prints out. So, the RAM in that location must be being used by the creation of A. So when and where does A get its length? Is that a separate stored value that's picked up when A is created?

I looked at the memory locations of all the variables in your example (and my modification of it), and I see several things I didn't expect which probably explains the 24 not 25 that we see. The location of z is right after the location of zero, both being before the location of the arrays. I assume this is some sort of optimization thing so the 3 bytes after zero are not all wasted.

One thing I tried that seems weird is what happens when I insert another array, g, in between a and i. When I look at the memory locations of the arrays (printing out only a,i, and q) they are all still in the same place; I would have thought they would be in the order that they were created, so a,g,i, and q. Even weirder, if I now print out all the array locations, including g this time, then g is where a used to be, and a is moved back 8 (or 12 if the array sizes are 9) bytes. I can't imagine why changing the print statement would change the location of those arrays.

@ScruffR, the below code fix the problem that I am seeing. Its the null terminator that’s causing the problem. Before I am using char coolSched[232]; and char heatSched[232]; since I am sending 232 byte per publish the buf did not have room for the null which strcpy required.

char coolSched[233];
char heatSched[233];

void schedHandler(const char *event, const char *data)
{
  HCflag = String(data[231]).toInt(); // production
  if (HCflag == 0 || HCflag == 1)
  {
    strcpy(heatSched, data);
    EEPROM.put(0, heatSched);
  }

  if (HCflag == 2 || HCflag == 3)
  {
    strcpy(coolSched, data);
    EEPROM.put(233, coolSched);
   
  }
1 Like

@sheng, as (I think) @ScruffR pointed out before, you should use strncpy() to protect against index overrun on the char arrays if data is larger than the array has space for. :wink:

1 Like

Exactly, but only for globals.
And with clean stack I meant that there were no previous function calls that placed other values on the stack that would still be there when subsequent function calls reuse the same memory locations. Since the system will always execute pretty much the same functions prior to your own function the "residues" on the stack (where local automatic variables are stored) will mostly be the same. And the String constructor is also a function that will leave some stuff on the stack which can end up in the string as these bytes will be placed back to back with the still present variables of the calling function.
And A only gets its length after all the costruction, heap allocation and string copying has already happened.

For the location of variables I have no good answer, but the optimizer and linker may have a say in that. It may also look different on a completely fresh project with no precompiled modules.

In addition to the sound advice of @peekay123 (which I usually give but seem to have forgotten this time :blush:) when using some strn...() functions you should check how they behave when an operation doesn't fit.
e.g. strncpy() will still leave you with an unterminated string when the buffer is too short.
http://www.cplusplus.com/reference/cstring/strncpy/?kw=strncpy

This is what I meant with this

So the safe use of strncpy() would be

char buf[32];
memset(buf, 0, sizeof(buf));
strncpy(buf, someLongString, sizeof(buf)-1); // keep final zero intact
1 Like

@peekay123, @ScruffR, I definitely learn my lesson on this one. :smile: Thank you all for helping out.

1 Like