Modular Synthesis in Virtual Reality

Rack v2’s headless mode would seem to be a prime step toward developing an alternate “3D head” for the software, then allowing interaction via VR (Oculus, Vive, Index). VCV already seems well prepared for such an application, given the degree to which it matches real (3D) hardware in its design language.

Besides the clear need for a beefy graphics card, perhaps the biggest hangup I can potentially see would be in allowing for “dual cursors” so that left and right hand can both interact with module parameters simultaneously.

Another would be in allowing for the possibility of alternate 3D parameter models to be loaded in this mode, so that all the knobs and sliders aren’t just 2D textures.

Programming the 3D cable physics might also be non-trivial.

Has any thought been given to this? Any official plans? I haven’t found any relevant discussion via forum search.

5 Likes

I’m drooling for this as well. Unreal Engine already has the cable physics built in. I think that would be the best place to start… I will keep pushing my other half to give it a shot. :wink:

Correct me if I’m wrong but headless mode will not be about providing an interface to plug in a different rendering/ui system, and more about using Rack easily in setups without displays, where you’ll be able to control things mainly through midi-in/out modules (think standalone hardware synth running vcv patches).

So a VR interface will not be more possible than it is now, if you create some VR app that sends midi/osc with manually mapped out knobs and such, which might be pretty cool, but not the same thing as Rack in VR.

The main issues I see with the latter is the necessary stable api to control things like cabling, module and patch management etc. and all the work needed to port modules to 3D.

Nevertheless I’d love to see either of these became a reality.

There is a ‘‘DJ simulator’’ game in VR, i don’t remember the name, but i wonder if it sends midi

Wouldn’t it be uninspiring though to control modular with DJ gear interface/scenario?

What I’d need is a rack on the ISS, under the ocean, flying through wormholes, noodling inside a cell etc. :stuck_out_tongue:

1 Like

There is also No Man’s sky to make music in VR :slight_smile: one of the lastest updates includes a Daw machine computer you can build in chains on your planet

1 Like

Personally, owning a first-gen rift and dabbling in UE4 development, I don’t see much appeal in this.

The VR controllers are not very precise. They’re good enough to swing a sword around or shoot a gun at close range and enact other kinds of videogame brutality, but definitely not for delicate control over knobs and mini jacks. Realistically porting over gizmos that are smaller than your hand and require fine motor control isn’t gonna work.

And if you’re gonna mess with scale for convenience, why not re-think it as a VR-first UI, instead of porting over real-world metaphors?

Also, for those who consider it a concern, how do you make an exciting VR performance? Modular performing already has this problem that the artist is some dork turning their back to you while twiddling knobs, at best you can film them from the side with a wide angle lens. VR conceals the performer’s face!

3 Likes

Not VR per se, but addressing the controller responsiveness for performing issue, have you checked out the MiMu gloves? https://mimugloves.com/

  • Paul “Uncle Chrome” Artola Ellicott City, Maryland

They’re one order of magnitude over what I’d budget for that kinda stuff!

I bet you could put on a more exciting performance than those big name artists made of money with just a couple of beat up first-gen wiimotes

One order of magnitute???

£1,250 per glove, and you need as many of them as you have hands! Boutique tech is in an entirely different price range than repurposed enthusiast consumer tech.

The VR controllers are not very precise.

While I don’t own any VR controller, their precision seems to be on-par with midi hardware (128 discrete values) in terms of rotation for example. Have you checked out Soundstage I linked before? It is definitely a VR-first design but rotating knobs, sliders and making connections look fine to me from the vids. Of course there’s no reason to stop there, there might very well be other ways of control in such a VR program. Also specific hardware for just this, like a rotary encoder stuck to a VR controller that uses something like the SAIL module to adjust whatever you are pointing at. This could be done with a mouse too for 2D Rack. :wink:

why not re-think it as a VR-first UI, instead of porting over real-world metaphors?

This argument works against Rack in general too :stuck_out_tongue: I guess the appeal of the VR idea ( and Rack itself in a similar sense ) is that there are many great modules already working, so it makes sense to port instead of reinventing (reimplementing in this case) the wheel.

how do you make an exciting VR performance?

First is good music of course then visuals and any additional theatrics. For example deadmau5 conceals his face and still manages to pull off an exciting show. This isn’t a concern that’s specific to VR and has been around since people started using laptops to perform. If nothing else, just project whatever the performer sees on their screen/headset. Idk why this isn’t a thing (apart from algorave), maybe cause people don’t really care? Or cause most daw’s don’t look that interesting. The problem beneath it all is that twisting knobs and pushing buttons isn’t considered to be an act of virtuosity, the creation of electronic music is more of a cognitive effort than a physical one, and this just doesn’t translate well to stage without some bells and whistles.

there are some cheap alternatives like the leap motion

I guess I should have been following this more closely but isn’t it a question of whether you can do VR eurorack (VeeArr-o-rack?) for less money than an actual eurorack setup?

I guess this is mostly just daydreaming here :smiley:

That being said, an actual eurorack setup will always be more expensive than something finite like a VR setup :stuck_out_tongue:

1 Like

I disagree that current controllers lack the needed precision for a great experience. I’m currently working on getting this very idea off the ground, and my experiments so far are promising.

Some experiences from the game half-life alyx boost my confidence. What they accomplished in that game in terms of user control is fascinating. I was surprised with how much fine control you could have when drawing/writing with sharpies on this one window in the game. It was quite entertaining.

I’m more worried about licensing issues more than anything right now. There are some test modules we modeled in the look of the free modules that come with vcv rack. I’m not sure we would even be able to use those.

I’m hoping to have the application in some sort of usable alpha state here very soon. If anyone would like to playtest, definitely send me a message.

If you have concerns about licensing it is best to talk to the developers, almost all of us are on this forum.

I’m curious though, are you making a VR thing from the ground up or are you planning on building on top of the Rack/modules codebase(s)?

Its being built up as it’s own separate thing. It’s actually meant to be a virtual midi controller/inspiring environment simulator.

The original idea came from a desire to play piano in inspiring settings and also frustrations with not having enough knobs and sliders on my midi controllers to fully map omnisphere, so there is a plan to have more custom/originally designed virtual midi’s for all the different synth vsts out there.

1 Like

Cool!

Are you tracking/rendering the piano keys and hands in any way or just play blind?