2.8V minimum was mentioned in another Thread, here. But as you said, the 3.6V max (Open Circuit Voltage) may be too high to direct drive the Xenon with (2) Energizer L91's, IDK. The Voltage will be closer to 3.3V when loaded, but I've never tried this with a Xenon.
3xAA L91's (5.4 OCV) would work when feeding Vusb, and might be fine for Li+ Pin, maybe someone can confirm.
Considering you plan to operate at -20 degrees C, I'd vote to go w/ the higher starting Voltage (3xAA 5.4 OCV) and use the most efficient Path available, or an external Buck/Boost converter as previously mentioned. Either way, you'll want to test the actual Sleep & Operating Currents @ -20 degrees C to calculate your Power Budget.