Hi there & welcome!
I reproduced your scenario with my Minibrute 2s and have the same jumps in Clocked exactly as described.
I often sync the other way around - external stuff from NI Machine or Reaper and I can see similar jumps in my external sequencers bpm-display, ‚unstable clock!‘-warnings, as well. I find it quite common and most of the time I just deal with it as I find the bpm-displays concerning, but the timing not really strange from a listening point of view.
I don‘t think your computer is a weak link spec-wise. Keep in mind that midi-hardware-syncing in both directions was successfully done on Ataris 30 years ago.
You seem to be on a PC? - I read about macs that they went away from giving midi, especially over usb, a high priority, which is why I am using the midi-port on my firewire-soundcard sometimes for syncing. (To use a dedicated port with nothing else disturbing the clock might be a good idea in general.)
Although this ain‘t perfect either I have smaller jumps on my sequencer‘s display using the firewire-port. Maybe this is just black magic, I haven‘t really tested and compared it.
I guess my point is: unfortunately in neither direction I experience a trustworthy stable clocking over midi ever and nevertheless I still do it all the time, and it kinda works for me. And also: I am not sure, this is a VCV-problem, only. - Not sure a version 2 will solve it completely.
Did you try other DAWs as slaves having a stable read and feel?
I have no experience doing this, but I wanted to mention it: there are ways of syncing with audio instead of midi, maybe you want to look into that.
Let us know, if you came up with a solution that works for you. Also, if you run into other problems, of course.