Yeah, me too. I’ve been working towards that experience for close to 50 years now. I’ve mentioned in this forum that I built my first modular synthesizer from kits in 1973. By 1980 I was interfacing an EEG to my Apple II+ computer and experimenting with computer music controlled by thought. I started working with VR hardware somewhere around 30 years ago, but had to build my own headsets and stereo video cameras doing wireless telepresence. I’m now on my ~7th generation of commercial VR hardware. 24 years ago I developed my own game engine so I could play with VR. 15 years ago I created a worldwide 3D metaverse system for my employer as well as a Python scriptable API for their 3D modelling and animation software. I included MIDI control in that API. I could control character animation via a MIDI controller.
Perhaps my effort that has come closest to VR/music immersive experience was 4 years ago with the Oculus Rift 3rd gen. I developed a mixed reality (VR, AR, synthetic reality) system coded from scratch with the idea of doing interactive music with either my Meander program, Ableton Live or Max/MSP . For that, I had a stereoscopic 3D video camera mounted on a robotics pan and tilt turret. I had the Oculus Rift VR headset and controllers interfaced such that the head and hands positions and movements could be sent to Max/MSP via OSC and from there to an Arduino which controlled the camera turret. The stereo camera signal went back to my program which them rendered their view in the Oculus Rift headset so I saw whatever I looked at by moving my head and controlling the cameras. Then I augmented that reality with synthetic reality which was my 3D avatar that was controlled by my head and hand movements and appeared in the VR scene. Another virtual pair of my arms could be seen reaching into the scene. I called this the “Cyberpunken Virtual Trio”. My goal was to turn this into a music performance experience, but alas, my Oculus Rift was deprecated and I did not want to redevelop it all, so I moved on to the HP VR system and then the HTC Vive Pro, but I did string theory physics rather than music.
Here is an image of that Oculus Rift experience. It will warp your mind
In this picture, I had to take off the VR headset as I couldn’t see how to do a screen capture from within VR.
Anyway, it is feasible to do an immersive Rack and VR/AR/MR experience, but it is exceedingly complex, at least for me. I doubt that I will do such but I won’t rule it out. But, I would be tempted to use GP-GPU massively parallel computing as I have done on almost all of my projects since ~2009. That would not be very portable. Of course, VR hardware is not portable!
By the way, I discovered VCV Rack in 2019 and started porting Meander to it so that I might have a future path for esoteric human-computer-music interface.
Thanks for such a detailed reply. You history and work are truly inspiring.
I do not even consider myself anything close to an amateur. I am back in grad school for an MFA in Studio Art, where sound and music are great but an only by-product of some of my artistic experiments.
I do 3D scan plants and have a plan to create a virtual landscape with them, with a soundtrack made with bio-synthesis.
I got the bio-synthesis part, now I need to see if I can figure out the 3d to VR/Game landscape.
One of my courses is max/msp so I might use it to be part of the environment.
Anyhow, I am not sure if you know this project but it is quite inspiring
Very interesting. Many, many years ago I had a “project” involving mycelia.
One of my more successful collaborative research projects in my university research lab was with an art professor who did sculptures in various media. I and my team helped her do 3D scanning of her sculptures and then bring them into the Unity game engine. She built a virtual 3D gallery. I then taught her how to set up an early Oculus Rift and gaming laptop and view her artwork in stereoscopic 3D. She stopped hauling her physical sculptures to conferences and shows but rather took her gaming laptop and the Oculus Rift. Sadly, she passed away a couple of years ago.
All told, we had about as many collaborative research projects involving the arts as we did involving the sciences and engineering. I was also the director for the Digital Media Studio and the 3D Animation Lab for faculty and student use. So, digital media and 3D visualization is a topic close to my heart.
It could be a great this multiply instance addon in Meander. It’s changing everything I’m a little worried about my potato pc. Can it handle more than two instances? But it will only turn out when the library is updated. Your last piece on SoundCloud are great. It show a potential of this module. Cheers Ken
Thanks. Meander is lite on the CPU but heavy on the GPU. A single instance of Meander should have about the same performance as the singleton version of Meander. So, all of your existing patches that use Meander should continue to work the same. It will be interesting to see what kind of performance you get with multiple instances.
I just did tests on my system with from 0 to 12 instances of Meander and nothing else in the patch. For these tests, I have VCV Rack set to 60Hz frame rate and 4 threads.With 12 instances, each is consuming 1.4% of the CPU.
With 1 instance of Meander, my frame rate is 60 fps with avg CPU of 3.2% and max of 6.1%.
With 2 instances of Meander, my frame rate is 34.2 fps with avg CPU of 4.7% and max of 7.0%.
…
With 12 instances of Meander, my frame rate is 37.0 fps with avg CPU of 9.5% and max of 12.7%.
Performance is pretty much the same independent of view zoom.
With 1 thread:
With 12 instances of Meander, my frame rate is 55 fps with avg CPU of 13% and max of 28%.
With 12 threads:
With 12 instances of Meander, my frame rate is 43 fps with avg CPU of 82059% and max of 82059%.
I’m on a Dell Alienware Aurora R7 gaming tower with an NVIDIA GeForce GTX 1080 GPU.
That’s a good information. I will testing on my AMD FX-8320E 8GB RAM & Radeon RX-470. It is not optimal equipment for creating music, but it is sufficient for me.
Here is a new song “Pulling out all the stops” using my multi-instanced Meander module that should be in the library any day now. In this, 3 coordinated instances of Meander are set to Dorian Db with a tonal harmonic progression (I-II-III-IV-V-Vi-VII). There are 9 instances of Organ Three being played. In addition, the parts fade in and out via a phased automation control of the MindMeld MIXMASTER tracks volume.
Just a quick update on Meander v2.0.15 non-singleton module version. I submitted to the library review and build queue on March 18. Yesterday Andrew added the comment to my issue topic:
“The panel of Meander appears to not load on Linux. Is the filename correct?”
Everything looks correct to me with the filenames, but I did have a typo in my loadImage() calls that was innocuous under Windows but perhaps not under Linux. I made the correction but also committed a couple of fixes for minor issues that I had identified after I had submitted to the queue.
I’m not sure what the status is on this, nor which panel image was failing to load. I suspect it was in the browser case code, which is where my typo was located. I asked for clarification in the issue thread, but I am not sure how often those are checked.
I’ll post more when I know more. With luck, my fix will fix the Linux problem. If it doesn’t, I may need to ask if there are any volunteers who could build Meander for Linux and take a look at the symptoms. It probably does not make sense for anyone to do that until I know when and where the panel image was failing to load. Note, this is probably related to my static .PNG panel image load and rendering in the browser.
Yeah, that was my first thought, but names and case match exactly. Thanks though. Does Linux have any problems with dashes in the name? Here is that actual line where I load the PNG:
Here is the line with the typo where I was doing a multiple non-integral type assignment accidentally. Does Linux behave differently with multiple assignments?:
Thanks. It doesn’t make too much sense to dig into this deeper until I get some feedback from VCV as to what exactly was failing. It might not be the loadImage() at all.
Does anyone here know what the proper protocol is for communicating with VCV when there is a problem that crops up during execution of a module after it is built for the library but before it appears in the library. Do I just comment in my open VCV Github issue thread? I’ve done that but there has been no response in two days, so I’m not sure I am following protocol. My open issue still shows to be in the “Build update” stage of the process. Thanks.
Don’t know if this is relevant, but if you install the VCV toolchain then you can make builds exactly like the library builds. And of course you can make builds for other platforms. I run it in an Ubuntu 20 image in VMWare on my win11 machine. It’s a minor pain to setup, but after that it’s great to have.
I think it’s even easier to use one of the docker images, but that’s something I didn’t know as well, so I went with the local Linux install.
How do you test MacOS builds? I have set up a MacOS VM on VirtualBox before, but it was not exactly legal. I have Debian installed on my old ASUS gaming laptop and have had Ubuntu installed before on multiple machines. I just don’t have the interest in messing with anything other than Windows these days. I ended up on Windows several years ago as I was doing 3D game engine development and DirectX was about the only game in town in the beginning (1998).
I’m hoping someone at VCV will give me more details in answer to my question of what it means that the Meander panel image does not load on Linux. If I do not get feedback, it may be time to retire again. If I do get feedback, who knows whether I can figure out what’s up. Maybe it will be something simple and stupid.
