So glad you brought this up, since I’ve been wanting to do this for a while! Dynamic memory allocation will be a key part of the firmware for the Photon and hopefully this can be backfitted to the Core also.
Yes, the stack and heap do grow together, so you could look at the heap and stack pointers to see if allocation might fail.
In fact, in src/newlib_stubs.cpp you’ll find the _sbrk function, and it does exactly that - looks at the stack and heap pointers to see if more memory can be allocated. _sbrk is called by the memory manager to allocate more heap,. and it attempts to allocate more heap, or calls the SOS panic when memory cannot be allocated. Changing this to return NULL rather than panic should be all that is required to have malloc gracefully fail.
I hope that helps you get started - let us know how you get on!