Hi jsharpphoto,
Lots of good ideas in this thread, all of which will take you down the modular rabbit hole, never to return. 
I suppose the first place to start is to ask yourself is what kind of music are you making? That decision will guide you towards some modules and away from others.
Most of your decisions will likely come in the form of modules that can be inserted in the XL’s Mix Out / Filter In jack. Things like filters and sound manglers/modifiers.
For filters, there are a lot to choose from. The dotcom Q107 State Variable Filter will give you low/band/high/notch responses (12db/Oct, with all outputs available at once in two spaces). Ditto for the Happy Nerding Fun VCF, a Polivoks-based dual VCF in a single space (but no notch filter). Either would take the XL into new sonic territory. Others? Eric already mentioned the STG Mankato, but take a look at the STG Sea Devils Filter (EMS clone) and the STG Post Lawsuit LPF (ARP clone). Corsynth makes the C101 OTA LPF (Roland filter clone). Of course, I’m leaving out offerings by MosLab, COTK, et all, but the ones I specifically mentioned seem to have reasonable delivery times (MosLab is currently fully booked and won’t be shipping new orders until Sept 2016!).
As for sound manglers/modifiers, the STG Wave Folder is a great one, and the Happy Nerding FM Aid is a cool way to introduce some FM textures to your sound. Another module to consider here is the Corsynth Freq Divider/Multiplier, which can be used both as a sub-oscillator and super-oscillator. Makes a single oscillator sound huge.
Erik also mentioned the STG Mixer. Caveats aside, it’s a great choice (and you’re going to want another mixer anyway). I also agree with his assessment of the dotcom Q127 FFB - it is a bit of a let-down. You might be better served with a MuRF here.
On the modulation side, one LFO module to definitely consider is the new dotcom Q167 LFO++, a combo LFO, VCO, and EG. Although you said you didn’t think you’d need more oscillators, the Q167 will provide unique wave shapes that you won’t get from the XL. And as an LFO, this thing will go down to 0.015Hz (one cycle in 60+ seconds – great for slowly evolving textures). Highly recommended (and no, I don’t work for Synthesizers.com).
Finally, you might want to take a look at my ‘Tips & Tricks’ article about processing the Voyager’s oscillators independently:
http://forum.moogmusic.com/viewtopic.php?f=2&t=23002
You can get a lot of variation out of a Voyager using this simple technique with just the gear that you have… and your wallet will thank you!