Under System Preferences > Keyboard > Shortcuts tab > Mission Control, untick the default Show Desktop shortcut, which is F11.
The F11 keyboard shortcut in Rack then works as intended – or it does on my Mac running VCV Rack v1.1.6 on macOS Mojave 10.14.6.