It sounds like you’ve got an inverted envelope programmed to affect the pitch. The pitch going ‘down and up’ is the result of the inverted attack and decay segments of the envelope. Try adjusting the Decay parameter to 0 and increase Sustain to 255. This will make the pitch dive down and stay down.
You will still have an issue with the envelope Release parameter bringing the pitch back up when you release the key. You’ll have to experiment with this control to get the sound where you want it.
Another thought: why not program the Pitch Wheel for maximum bend (+/- 2 Octaves +5th) and do the dive bomb manually?
one way to acheive this affect is to have the following item and with some tweeking it will work.
separate cv envelope generator.
patch the keyboard gate to the gate in on the cv envelope and its main out to the pitch cv in. set the attack to its shortest setting and the rest to your needs.
I tried using the filter envelope as the modulator. By having a very long decay and setting the sustain to 0 I was able to have a very long dive which didn’t come back up. The Attack level will also dictate if the pitch will rise and then go down or just start to dive. You’ll have to tweak that as well.