Compiled code for Core much larger than for Particle

I have code running well on Particle and wanted to build it for Core.
I was getting following error:
region `APP_FLASH’ overflowed by 2156 bytes

I assumed that the the size is the problem and so for test built a much simpler application which builds ok but what I do not understand is why the binary is much bigger now:
Core binary: 98k
Photon binary: 13k

Why such difference?
Can I do something to make Core binary smaller?

I am using cli with the following commands:
particle compile core
particle compile photon

For the Core you compile both the system firmware and the user firmware.
For the Photon you compile the user firmware only. That’s also why it’s much quicker to flash your Photon.