uint8_t my_data[256000]; Compiles?

Trying to track down a bug when i wanted to see how much ram i had left.
I created a variable uint8_t my_data[256]; I then started kicking it up to see when the compiler would fail as it would run out of ram. I changed it from 256, 512,1024,2048… and so on, it kept compiling.
Then i just slapped a bunch of 0’s to make it ridiculously long so it would fail. Yet it does not.

Why ?

How were you compiling? On the webIDE or locally? There have been a few weird instances where the webIDE is not recompiling despite changing the source code, so you might want to cut and paste this to a new project if you are using the webIDE.

Compiling locally I get an error at 6445 bytes:


/Users/bko/SparkCore/gcc-arm-none-eabi-4_8-2013q4/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/bin/ld: core-firmware.elf section `._usrstack' will not fit in region `RAM'
/Users/bko/SparkCore/gcc-arm-none-eabi-4_8-2013q4/bin/../lib/gcc/arm-none-eabi/4.8.3/../../../../arm-none-eabi/bin/ld: region `RAM' overflowed by 4 bytes
collect2: error: ld returned 1 exit status

With the particular master repository I have, which is a day or two behind the current master, I can get this to pass but one more byte fails:

#include "application.h"

uint8_t testarray[6444];

void setup()
{
}


void loop()
{
}

At first locally. Then i jumped to the web IDE and tried it there as well. In both cases it passed.

If you take your uint8_t testarray[6444]; and change it to uint8_t testarray[6444000];
It then fails. which it should.

However if you then move it inside the setup() function it passes.
Something is very wrong here.

Could this come from the fact that automatic variables are not actually reserved at compile time, but only “created” on call of the procedure by pushing them onto the stack?
They only live as long the called function lives and get popped off during function-return-cleanup-action.
This way you’d get a runtime stackoverflow but the compiler would not need to care, since there might never be a problem if the function never got called.

@ScruffR is right. The compiler does not know what will happen to automatic variables.

I do get a warning that I am not using the var when building locally:

../src/application.cpp: In function 'void loop()':
../src/application.cpp:15:11: warning: unused variable 'testarray' [-Wunused-variable]
   uint8_t testarray[7000] = {
           ^

The compiler is also free to remove any dead code you are not actually using.

If you were to run something like this but with much larger numbers on a desktop computer, the OS would catch you when you ran it and give a segmentation fault since you tried to cross out of the memory allocated for your program. On a small microprocessor like :spark:, you get strange behavior and eventually a crash of some kind like the red SOS flashes we have now.

If you want to know how much RAM your program is using at the start of your program, you can get that when compiling locally:

Invoking: ARM GNU Print Size
arm-none-eabi-size --format=berkeley core-firmware.elf
   text	   data	    bss	    dec	    hex	filename
  81396	   2620	  11488	  95504	  17510	core-firmware.elf

The data field represents variables that have initial values, so those values are first stored in flash with the program and then copied to RAM when your program starts. So the amount of flash used is text+data and the amount of RAM used is data+bss.

Using the web IDE I just tried this and it still compiles.

void setup()
{
uint8_t testarray[644400];

testarray[0] = 0;

}

Sure it will compile, since this is not a problem the compiler does check for.

As said above these variables are sort of non-existent before runtime but the compiler only cares about things that exist at compile time - which automatic variables don’t.
It’s just like setting a pointer address outside the allowed range in your code. The compiler does not care but you’d still get an runtime error.
On the other hand if the program logic happens to correct the pointer before it gets used, what business would the compiler have to complain about it before hand?

Try this to check your free RAM in app:

@BDub this is a nice way to figure out mem usage at runtime :wink:

But this seems to be the pivotal point: Runtime mem usage is not anything the compiler could reliably care about.
This will always be the developers duty - e.g. estimate stack and heap demand in different use cases.

I have to disagree with you on one point here. Shure if it was a pointer then i have no problem with it at all.
However an array is a whole other story. With an array i am telling the compiler i want this much reserved for me and it better check for that. Just like bko first reply, When he made the array 6445 bytes it erred out. Now moving that same array inside any function it does not. That is a serious problem.

@seulater having the array declared global or automatic is nothing alike.
So it’s not the same array anymore.

When declaring it global you tell the compiler to reserve some space for that array and provide a pointer to this space for referencing it.
When declaring it automatic you just tell the compiler to produce the machine code to reserve some space on the stack (whose size the compiler does not yet know) at runtime and provide a pointer to this just allocated memory location - wherever it migh happen to have been found. For some data types the automatic var gets pushed onto the stack directly and for others some space will be allocated on the heap and only a pointer gets pushed onto the stack and there we are at the pointers again.

I’m not sure how this is handled on Cortex µCs, but for other platforms an automatic

 char buf[1024]; 

gets translated into something like

  char *buf = malloc(1024 * sizeof(char));

and now you might see that even passing the 1024 as constant number does not make any difference to the compiler, since malloc does not distinguish if the ammount to allocate originally was a variable or a number literal.

Additionally how should the compiler/linker calculate memory usage if you’d happen to call a function (that contains an automatic array of known size) recursively? Since this means to allocate a full set of local variables for each call to this function - but how often will it be called???

On the other hand it actually might happen that at compile time it might seem that there is enough free space, but due to a lot of nested function calls and heap allocations happening before the one and only call to your function the stack/heap had been almost filled and so there will not be enough space after all - what then?

So you might have to accept that this is not actually a serious problem but is by design.

@ScruffR, Thanks for the explanation. Yet it does not make sense to me that it should do this.
Lets say for conversation sake that i am using a processor with 8k of ram.
I create a global array using almost all of the ram.
char buf[8100];
Then i have a function and in there i also create another array.

void My Function(void)
{
char buf2[1024];
unsigned int x;
for(x=0;x<1024;x++)
{
buf2[x] = 0;
}
}

If i compile this all is well. Yet actually it is not. If we were to run this it would be jacked up, as the compiler would not warn of exceeding ram usage. Now here is where i have the problem / confusion.
Lets now take all the files we use for the spark core. There are so many calls everywhere there is no way one could possibly keep count of just how much ram is being used, If i were to add my own functions in application.cpp i would not know just how much i have left for me to use. I would think that is something the compiler would be keeping track of.
It knows i said char buf[8100]; and char buf2[1024]; so then it should say upon compile that i have exceeded ram useage. On the other hand if i said char *buf and char *buf2, then i get it because i have not told it how much i want to reserve.

I know i am missing something but what ? Basically if all that has been said before is true, then how does any programmer know what they have left for them to use when a program grows so large in size that one person cannot keep track of all the ram being used.

I think the assumption here is that the compiler knows how often the function will be called, and whether or not it is recursive (i.e. calls itself directly or indirectly via another function). It also assumes the compiler knows how much memory will be available at the exact time the function is called - none of which can be determined at compile time. Global variables are always created and are in fact created at compile time, so the compiler can determine if the globals take more memory than is available at startup.

But where does that leave us the coder to know if we have enough to use from whats already being used ?
When i think of this i think of compiling a Linux distro. There are hundreds of files. I cannot imagine any human could ever possibly know how much ram is being used with all the functions calls in there.

The thing is, a true operating system like Linux, Windows, etc. have mechanisms built in to avoid this kind of problem. They have virtual memory, which allows the use of disk space to act like (very slow) memory, and they can limit the amount of RAM a user’s program can allocate. In the end, however, if you allocate more memory than you have available, the program (and sometimes the OS) will crash with some kind of run-time out-of-memory error. Since the Spark has no OS, and no disk space to spill memory into, avoiding that fate is up to you.

As far as “How do we know what we have left?”, I have no good idea. Careful coding and testing, easy to say but hard to do, is the only real solution I know of. Maybe someone else will have a better way…

Thanks guys, but everything so far does not hit the heart of what i find most confusing.
take for instance this spark core with all its files.
We are to write our app in application.cpp file. Just for giggles lets say their source used up all available ram but 1 byte. Now when i create a new function and create a variable char x=0; in there the compiler will reserve space for it. Life is good. But now i have no more room at this point. If i then create yet another function and another variable char y=0; it will compile but when it runs it wont run properly because the compiler is not keeping track of all these variables inside functions. So how are we to know where we are really at in the grand scheme of things if the compiler is not.

Welcome to the wonderful world of embedded programming! :wink:

Ah, I'm afraid that's not quite how it works.

In C/C++, variables can be stored in one of three places:

  1. In memory allocated by the compiler. Variables that go here include global variables, static variables inside functions, and static class variables.

  2. On the stack (more on this later). Variables that fall into this category are non-static local variables (ones defined in a function, such as your char y=0;). The max size of the stack is typically fixed and predefined (I have no idea what it is on the Spark).

  3. On the heap. These are variables allocated by new/malloc(). The size of the heap is generally whatever memory that is left over, after taking into account the storage needed to hold the compiled code, variables allocated by the compiler (item #1, above), and the space allocated for the stack (the max size).

Variables on the stack:

As I mentioned, non-static local variables (ones defined in a function) are stored "on the stack". Memory for these is allocated at runtime, from the stack, and not compile time; the compiler cannot allocate memory for this. Now, you might expect that the compiler could allocate memory for char y=0; defined inside a function, but it can't. Because functions can be called recursively, there can be multiple "versions" of char y=0; -- one for each recursive call.

In other words, if you have a function like:

double f(double x)
{
    char y[100];

    if (x <= 1.0)
    {
        return (1.0);
    }
    return (x * f(x - 1.0));
}

And you call it like f(5);, you will, at one point, end up with five copies of char y[100] allocated on the stack. Note how the compiler cannot possibly allocate this space at compile time. Given enough recursive calls, or large local variables (ones inside a function), it's possible to overflow the stack, which is really bad. Embedded systems like the Spark generally have no way of detecting stack overflow (which could lead to tragic results -- it's been hypothesized that the unintended acceleration in some cars is caused by a stack overflow).

(Note for the nitpickers: yes, a good compiler will warn you that char y[100]; is unused, and, yes, a good optimizer will eliminate it completely, as it's not used.)

So, if you're trying to see how much space the compiler can allocate, you need to use a global variable, and not one inside a function. However, this assumes that the Spark tools knows how much memory there is on the Spark; it's possible (and I have no idea) that the Spark tools will happily create a huge program that cannot possibly be loaded and run on the Spark (perhaps someone else here knows).

1 Like

@seulater, the answer to this is: "It should not be necessary to exactly know, since you cannot anyway"
Any programmer should be aware of the risk, that at runtime memory might get low and hence avoid hogging memory in their own local variables and on the other hand take as much precaution as possible to not crash if there is not enough mem left - e.g. by checking return values that might indicate possible failur to allocate memory.

As for the "problem" that there are loads of functions defined all over the firmware - due to the fact that there is actually no mem reserved for their local vars you and the compiler/linker does not need to bother,
If it were as you imagined, that each and every local var would need to be reserved the problem actually might exist - but good job that automatic/local variables are not created that way.

For example:

char *foo1(int n)
{
  char autoArray[4000];
  // do something 
  if (n > 0)
    return autoArray;
  else
    return NULL;
}

char *foo2(int n)
{
  char autoArray[4000];
  // do something else 
  if (n > 0)
    return autoArray;
  else
    return NULL;
}

Should this compile or not?

I see your guys point. It seems to me that there is more importance on having a debugger so you can watch things behind the scenes. What started all of this was the i have built a 2.8" Full color TFT LCD shield with touch screen for the Spark core. I have my fonts and graphic images stored in the onboard user flash to save processor flash space. It will run for a bit then just stop at random points. In tracing it down i wound up adding this inside main() in main.cpp. After doing this it runs just fine. Its because of this i started to think that maybe something like this thread is the cause.

    // this is the main loop in main.cpp
while(1)
{
    setup();
    while(1)
    {
        loop();
    }