Controlling Vcv rack with a webcam

Anyone has seen this story from Andrew Huang’s instagram where he controls notes (i guess ?) by moving his fingers in front of the camera ? I’ve got no idea how he does that :no_mouth:. Is there a way to link your webcam to vcv, and affect a parameter with it ?

1 Like

I didn’t saw it but anyway… You can make patch in vvvv or MAX, and then send anything you want via MIDI / OSC protocols — it’s relatively simple task :slight_smile:

1 Like

Something like this?


very cool ! I will check both these things, thanks !

Also possible via Usine Hollyhock :wink:

1 Like

It’s also worth taking a look at this thread: DSW Vision 2 'Flow alpha' (currently macOS only)

The plugin was based on the openCV library (where CV stands for Computer Vision in this case) and even if it had no further development - afaik - after the Mac alpha-release for VCVRack 0.6, it’s a nice example of possible implementations.

1 Like

I want to keep this thread alive! The Webcam MIDI link above has no files and no clear place to download so would somebody let me know if theres a followup there? Also as far as making a vvvv patch to translate webcam signal to midi is there anywhere for somebody with no knowledge of the program to find a similar patch to modify to my needs? or just worth learning from scratch? Thx!

Wekinator is another option for converting webcam data to OSC:

1 Like

Hello, i have made a little script for controll the knobs with my webcam laptop :slight_smile:

have fun :slight_smile:


Very cool, do you have any plans on release it by chance?)


thank you for the comment, There is a great chance that I include my facial recognition scripts, facial emotion, position tracking, body movement … in the same software package yes :slight_smile:

thank you for the comment :slight_smile:

1 Like

Fascinating !

More tech. details would be very interesting :sunglasses:

Month ago I saw a demo on Youtube about this kind of controlling (from the Site Unfortunately I could not go to test & play with it - as I promised. Drastic healthy problems made me to shift my life priorities. But this is another story and off topic.

hello, thank you for your comments, software use :: custom python3 script

(Python3 + OpenCV —> OSC protocol (network) → VCVRack)

OpenCV : Capture Video from Webcam Track/Detect Hands (dlib library, look around landmarks hand model, face, body…) Identify ‘Thumb + Index finger’ Distance Measurement Sending to the local network the last value length with OSC protocol

VCVRACK : module : cvOSCcv from trowaSoft (accept OSC message “length” and map it to buttons, knobs, fader…)

Last Step : Have fun :slight_smile: Music : The Horse Valse

1 Like

Thank You for the infos to dive into this fascinating matter :sunglasses:

Btw. ‘The Last Step’ is a direct hit :smiley: Since I can remember, music in all possible facets has been a welcome companion in my life.

For real fun you should dive into the Kinect universe. There‘s a lot of software for movement recognition on multiple axis.

Pure Data works nicely with VCV Rack.

1 Like