I have a vintage 1974 Minimoog (with oscillator board 3 installed) that has an issue when the decay is on. The notes will gradually fall in pitch when the decay is turned on; when the decay is short, the notes gradually fall maybe a half step and as you turn the decay up the notes fall gradually lower and lower.
Yes, your keyboards sampled voltage is slowly leaking away.
The pitch is falling away all the time… you just don’t hear it all when you set the decay short.
You have a fault. Could be a few things… faulty component or dirty (conductive) circuit board etc… take it to a tech.