Now that you have a voltage reference you can try to calibrate the 3.3V. See this post by @bko.
That would minimize supply inaccuracies. To also compensate for noise, the ADC readings would have to be simultaneous.
The nRF52840 also has an internal 0.6V reference, but I don’t know how to enable it. Also, it looks like you would lose some resolution this way. The 0.6V reference gives you a 3.6V range, but the pins are only rated to 3.3V.
If that doesn’t help, it might be worth exploring hardware band-aids.
The reference breakout has no bypassing. I would add a pair of 0.1uF and 10uF 25V ceramic caps.
You could try to solder the caps right on the Boron module, provided they have a good ground connection.
How is the reference wired to the Boron? I would use a thin coax, as short as possible. Ground the coax shield only at the reference module, leave the other end open. Make a good ground connection between the Boron and the reference module.
I agree figuring out ADC cal would be very handy.