This sounds like an amazing project!
If you are interested in connecting cables directly by MIDI messages without a matrix mixer you can take a look at this thread. I did a working prototype which is currently not available publicly, but it will be at some point. It is not very user-friendly but it works quite well. Let me know if you are interested.
Also, @mudjakub is working on something like that, with standard patch cables.