TapPatch proto (fast patching from keyboard or MIDI pads): feature requests requested!

[gc3 (gc)]

I’ve prototyped a new interface module. This thread is a combined announcement, planning thread, and request for early community engagement around features. Here are plans, details, background, caveats, and thanks, and here is the feature roadmap.

Update here as of 6/2: the alpha is in good shape and running about a week behind target.

Update [here] as of 9/13: the alpha completely blew through its target and is now on hold pending the immanent release of Rack 2, but I remain extremely committed to the project. Best guess for a full release is Q1 2022, but anyone who’s interested should be able to try it well before that.

Introduction

TapPatch brings fast and fluid MIDI and key-driven cable patching to Rack. It works with your rack in its current state, with no setup required.

Input and output ports visible onscreen are selected by tapping pads or pressing keys to select divisions of a grid, while positional, directional, and color cues appear simultaneously in the Rack window, on the module’s pads, and on feedback-capable MIDI controllers.

When both types of port have been selected, cables can be added (with optional color choice) or, if a cable already exists between the ports, it can be deleted.

MANY more features are under development or mapped out (to give you some idea, there will be three more pads, named Module, View, and Control) but even these basics will take you pretty far towards patching by tap.

Basic Operation

When TapPatch is freshly added, all pads are dark except for [OUT] and [INP], which are colored white. This indicates that neither an output nor an input port is currently selected.

Adding a cable has three steps: select one port, select the other one, and pick a color. That’s it! To select ports, we enter…

PORT SELECTION MODE

To start selecting an output port for the virtual volts in your new cable to flow from, tap the [OUT] pad. This will put TapPatch into Port Selection Mode, targeted to output ports. (The suggested mapping for the [OUT] pad, by the way, is the upper right hand pad of a 4x4 MIDI controller, or the / key on a numeric keypad; you can also click on it if you like.)

In Port Selection Mode, TapPatch will locate all output ports on your screen and place them into 9 equally sized rectangles, each of which corresponds to a pad on the 3x3 grid. (You can see the squares as thin black lines, which can be customized or turned off). It will then put a marker over each output port indicating which rectangle that port lands in (eight isosceles triangles, pointing outwards, for the outer pads, and a diamond for the inner pad). Each rectangle’s markers have a consistent (but of course customizable) color, and these colors also appear on the 3x3 grid for reference. (The suggested mapping for the 3x3 pad is the upper left corner of a 4x4 controller, or the number keys on a keypad.)

This is a lot of visual information, especially when zoomed out in a large patch, but the combination of positional, directional, and color indication–splitting up by squares, showing markers that point in the direction of the square, and consistently coloring the markers–is designed to let you select ports more and more quickly as you get used to the interface.

Back to the nine rectangles. Recall that, together, they cover all of the output ports on screen. Each of them will contain either no ports (in which case the corresponding pad stays dark and doesn’t do anything), one port (in which case pressing the corresponding pad will immediately select that port), or more than one port. When there is more than one port in a rectangle, pressing the corresponding pad will repeat the selection process for all output ports in that rectangle. This process repeats until either a port is selected or you cancel the process by tapping the [CLR] or [DEL] pads.

Picking an input port works the same way (and can, of course, be done first). The markers for input ports have the tip of the triangle turned back in, to differentiate them from output ports even when zoomed out. Suggested mapping for the [INP] pad is below the [OUT] pad on a 4x4 controller, or * on a keypad.

Only one more detail (for now) about Port Selection Mode: notice that when pressing a pad will result in selecting a port, both the (single) marker associated with that pad and the representation of the pad on the module will both have a white border. On feedback-capable MIDI controllers with sufficient color support, the color of the pad will be brighter than normal.

There are lots of features, refinements, and customizations planned for Port Selection Mode, but they all build on this basic interaction: drilling down among the onscreen ports until one is selected.

PATCH MODE

Once you leave Port Selection Mode, having either selected a port or cancelled selection, TapPatch goes back into Patch Mode (which is actually where it started). With both ports selected and empty, you will see nine (customizable) colors on the 3x3 pad; pressing any of these will create a new cable of that color between the ports, and reset the [OUT] and [INP] selection to white.

It’s time to discuss the colors of the [OUT] and [INP] pads. As mentioned above, white means void: no port is selected. Green means that a port is selected which has no cables currently attached. For output ports, blue means that one cable is currently attached, and purple means that more than one cable is currently attached. For input ports, orange means that a single cable (the limit for input ports) is currently attached.

These colors are different, though, if there is already a cable between two selected ports. In this case, the [INP] pad shows red, and the [OUT] pad shows red if that cable is the only connected cable, or magenta if there are also other cables connected.

Notice that TapPatch drops square markers of the same color over the selected ports in the Rack window.

These colors give you a quick indicator of what you can do with your selected ports in Patch Mode. The rules are simple: if Input is green, and Output is green, blue, or purple, tapping on the grid will add a new cable between the ports. If Input is orange, and Output is green, blue, or purple, tapping on the grid will “bump” the existing cable and create a new one (something not yet implemented in the default mouse interface).

Finally, if Input is red and Output is red or magenta, tapping [DEL] will remove the existing cable between the ports.

There are lots of other features planned for Patch Mode, but the basic idea is to give you as much (indeed, more) control over Rack cables from your keyboard or controller as you have with your mouse.

[gc3 (gc)]

Planned Development and Release Schedule

TapPatch will be a free, open-source plugin with a community development process. Because it depends on behind-the-scenes features of Rack which may change in Rack V2, it will not be submitted to the Library while we are still on Rack V1. Once TapPatch enters its open beta period, I will make builds available on GitHub (in the manner of @stoermelder’s Pack Tau) which can easily be downloaded and installed.

The code is not quite ready for release, but the community development process begins today! I’m already recruiting a group of around 5 to 10 alpha testers (if you’re interested, let me know soon) who will start giving feedback and suggestions on the nitty-gritty details of the plugin. I’m hoping to release code and builds to the alpha testers by the end of May (there’s more I want to add first, and the code needs a serious refactor before it’s ready for anyone’s eyes).

What I’m hoping for now, from anyone interested in MIDI or keyboard patching, is blue-sky thinking about features. There’s a big list of features below, but if there’s anything you’d like to see, now’s the best time to mention it, as the interface is going to get pretty full in the next three weeks. Now that the basic features (like markers and port search) are in place, it turns out that, for most features, getting them integrated into the interface is harder than actually coding the functionality. And since TapPatch is, by design, limited to 16 pads, there’s not unlimited space for features, although this is unlikely to be my last Rack control module…

After the alpha testing period (~two weeks?), my hope is to pull in another group of people for a short closed beta (probably a week), to make sure the plugin works and is stable on a larger range of systems and to do some additional refinements. Once that’s done, I’ll open up the code and builds, along with a feature roadmap based on alpha and beta feedback plus any requests and discussions in this thread. So if you’re not in the closed alpha or beta, should be able to try this out by mid-to-late June.

Further Details

As the drab grey panel, huge blank spaces, poor alignment, use of Calibri, and all-caps word PROTOTYPE suggest, this is a prototype. (It’s so prototypical that it doesn’t have rack screws yet!)

Right now, it’s being driven by a mash-up of @stoermelder STROKE and VCV MIDI-GATE coming in over that polyphonic cable, although I plan to integrate keyboard and MIDI support by the beta (so that it can be used by itself out-of-the-box). Here’s what’s happening offscreen in the demos above:

I may keep the external gate input to allow things like merging multiple MIDI control signals or allowing multiple keyboard shortcuts for the same pad. (By the way, TapPatch ignores its own gate input when selecting ports, so you don’t have to keep not choosing it!)

The full interface will consist of sixteen pads (of which thirteen are implemented in the prototype). These can be simultaneously activated by external MIDI controllers (4x4 pads recommended) or by ordinary keypresses (with number pads offering a very convenient mapping; plenty of suitable USB number pads are available for less than $20, if your primary keyboard doesn’t have one or you want to place it near other gear). On-screen clicks are also supported as a convenience, but are never required (except for module configuration options which wouldn’t normally be done in the flow of patching).

As mentioned in the feature list below, smaller interface configurations (with a 2x2 grid splitting the screen along the two main diagonals) are also planned. The 2x2 interface would be suitable for non-numpad keyboard use as well as 2x8 and (with some concessions) 2x4 MIDI pad controllers.

External devices capable of displaying colors via normal MIDI messages can show real-time feedback, generally following the on-screen pads, with some differences (for example, the white rings around the on-screen pads will be expressed on external devices as differences in brightness). (External devices which require Sysex messages for feedback will be supported once Rack V2 is released; sending Sysex out from Rack V1 requires a custom build).

And by the way, if you were wondering, it’s already pretty performant (graphically and computationally it’s doing very little compared to what Rack has do on every cycle anyway):

Caveats

While I am making every effort to ensure that TapPatch is as stable, seamless and problem-free as possible, its use of Rack features such as module introspection, cable manipulation and out-of-module labeling means that it’s hard to guarantee the same level of effortless interoperability that we are used to with Rack modules. So, for the foreseeable future, there will be, to quote @stoermelder’s T7 module (which obviously inspired this one and whose code I relied upon), “NO SUPPORT FOR VCV RACK WHEN USING THIS MODULE.” Your use of TapPatch will be supported as much as possible, but no one not involved in the project should be bothered about problems you have when using it, least of all @Vortico, who already has plenty to do.

Background/Philosophy

By this point, Rack has a suite of excellent options for mouse-free parameter control, both built in (MIDI-MAP) and via the Library (MIDI-CAT, for starters). Mouse-free patching, however, has been largely unexplored, despite some interest.

The leading (and, as far as I know, only) option for mouse-free patching is @stoermelder’s groundbreaking T7 suite, part of his experimental PackTau plugin. Jakub Mudrak’s intriguing Midilar controllers couple T7 with an innovative hardware setup, as seen here.

The T7/Midilar approach, however, relies on advance configuration; ports intended to be patched using the system are pre-designated, and cables are added to or removed from them when messages come in. This is very suitable for fixed racks and performance contexts, but not (at least in its current state) for fluid exploration of an ever-changing set of modules.

Hence TapPatch, which accepts the primacy of the screen and focuses on eliminating click-and-drag mouse movements. With hardware modular you need to see your rack to patch it, but you don’t need to drag your finger across it; you pick one port with one motion, and another port with another, and the cable falls into place. This is the basic interaction pattern which TapPatch seeks to emulate in software.

Thanks…

…to everyone (especially if you read this far) for your interest, and thanks in advance for your ideas! Special thanks to @stoermelder (whose top-quality, innovative open-source code, particularly for GLUE and T7, was crucial to this development) and to @Vortico, not just for Rack but for the decision to make Rack’s plugin interface so incredibly open. That something like TapPatch (or the many other interface hacks available) can be added by a plugin, not a fork, is a huge testament to the quality of Rack’s software architecture and to the decisions made during its development.

[gc3 (gc)]

Feature Roadmap

Immediate implementation: barring unexpected technical or interface design difficulties, these will all be finished before the closed alpha begins.

PATCH MODE FEATURES

• Single-port cable delete: With a single port selected, delete cable(s) from that port
• Cable follow to select: With a single port selected, follow attached cable(s) (looping through them for outputs with multiple cables) and select the linked port
• Multi-wiring: Add a cable without clearing output selection (for quickly wiring multiple inputs from one output)
• Cable color randomization: option for the center pad or for all pads
• Quick-add option: string cable with preselected, last-used, or randomized color as soon as ports are selected
• Change existing cable color by activating color pads under the circumstances that activate the [DEL] pad

PORT SELECTION MODE FEATURES

• Port Select Both: begin port selection with ports of both types available
• Recall last-used port (input or output)
• Back out of grid to previous selection level instead of having to start selection over
• Zoom in to grid when the sub-rectangles are selected, snapping back to original viewport on selection or cancellation
• Indicate port status on select markers (connected, unconnected, etc.); probably only when sufficiently zoomed in, unless anyone has a very good design idea
• Simple select option: when choosing between two or three ports with no ambiguity, allow more than one pad to select the port (so if one port is above another, allow all three top pads to select the upper port and all three bottom pads to select the lower port)
• Spill select option: when a chosen grid area has <= 9 ports, map all of them to the pads, saving clicks at the expense of interface consistency (since nine ports in a line would be mapped to all nine directions)
• 2x2 grid (for 2x4 or 2x8 “bar” MIDI controllers or non-keypad keyboard use); more clicks, but fewer selection options, and better for colorblind users

Pre-Beta Implementation: these are intended for inclusion before the plugin is released in open beta.

MODULE BUTTON FEATURES

• Select module via grid search, then limit selection to ports on that module
• In and outlist of modules (editable from the pads) to limit both port and module selection
• Module-level operations (like randomize, disconnect cables, etc.) from the pads.

VIEW BUTTON FEATURES

• Navigate Rack from the pads, so that the mouse isn’t even needed for moving the viewport (several existing modules do this, and I’ll be looking to them for features)
• View shortcuts, like zoom-in-on-selected port, follow-cable, etc.

GENERAL FEATURES

• MIDI pad color feedback for supported controllers
• MIDI in and out on-module (per Iverson)
• STROKE-based keystroke interception on-module
• Actual visual design, especially of pads and panel
• Interface helper: optional text display of interface next steps (for learning plugin)
• "Playing nice" with Rack in general (menu configuration, serialization, interoperability with mouse-based cable selection, optimizations, etc.)

Long-Term: these are not intended for inclusion before the open beta, but will definitely be explored at some point (and might make it into the beta, especially if there’s interested).

• Mouse-select port if you’re already using the mouse and want to cache a port for later use
• Port snapshot: option to show a picture of the selected port on the module, if I can wrangle the API
• Save quick recall ports (probably 9 input slots and 9 output slots)
• Expanders: several ideas here, including an option to defer a sequence of cable adds/deletes and perform them simultaneously (especially good for performance, and possibly on external triggers)
• The enigmatic Control button

[Curlymorphic (Dave French)]

Looking interesting, cant wait to test this. Someone has to ask, so it may as well be me, any builds or links to source yet?

[gc3 (gc)]

Thanks! Coming in “details” (edit: now added here) but the short answer is builds for testers within 2 weeks, open source beta not too long after

[stoermelder (Ben)]

I really like this “divide and conquer” approach of selecting and patching ports! I’m curious how many will actually use it for working with keys…?
Still, great work and I’m happy my modules could contribute some ideas

[gc3 (gc)]

Thanks! And it wasn’t just ideas but bulletproof code snippets that your modules contributed

Yeah, I’ll be interested to see if that many people will want pad or key patching enough to bring it in. I can say (even from the prototype) that it feels very comfortable to not have to leave my MIDI controller(s) and go over to the mouse, but my stuff is more input-driven and less generative. Also, it plays well with the mouse (big design requirement), so you can just have it sit there if you want it and do most of your patching the way you’re used to.

That said, I have some nifty features planned just on the patching side–and in a sense the patch side of it is really just a warmup/foundation for what I have planned for the Control pad

[dwat3r (Oliver Hechtl)]

This looks awesome!

I really like keyboard-driven workflow as a developer, and that’s why I like tiling wm-s like i3, dwm and xmonad, and Renoise as a daw as well. If you want, sign me up on the alpha testing, I’m happy to provide you feedback (and also eager to try this out )

[Squinky]

If you feel that way about the keyboard, you might want to check out my Sequencer. It is quite keyboard driven. You can even set your own keyboard mapping by editing a json file. Here’s the keyboard reference: https://github.com/squinkylabs/SquinkyVCV/blob/main/docs/sq2.md#Keyboard-reference

[gc3 (gc)]

Many thanks, @dwat3r (and welcome to the forum!) Really appreciate the kind words, and your experience with keyboard-driven workflows in general will be very handy in the alpha!

(also, ++ to @Squinky Seq++, it’s awesome)

[LarsBjerregaard]

VIM for Rack, yay! Seriouslly, this looks very interesting. For me the biggest damper on Rack patching pleasure is mouse usage, so if this can be made to work smoothly it could likely be my primary interface.

[gc3 (gc)]

Thanks, @LarsBjerregaard ! Now I’m thinking about what emacs for Rack (or ed for Rack) would look like

VIM is definitely the right comparison because of the modes (happily this is quite a bit simpler to learn!)

I don’t know if you’re interested in joining/have time to join the alpha (or later, beta) but even if not I’d love any thoughts you have on useful features, what smoothness would look like, your controller setup, etc.

[LarsBjerregaard]

Sure, sign me up! I can’t promise I’ll deliver anything on deadlines but I’ll be happy to try it out and give feedback. My first thoughts are ambitious. I can use my keyboard for patch-cables, but the other large part of mouse usage is placing modules and moving them around, and moving knobs/controls on modules. Doing that by keyboard as well would be awesome!

There was an old Firefox extension I used to love dearly. You would hit Escape and every control/anchor on the page would get a little number next to it. Then you would hit the number and the cursor or focus would jump there. Fantastic and effective keyboard usage for webpages. Wondering if something similar could work in Rack.

[dwat3r (Oliver Hechtl)]

You’re talking about Vimperator That was an awesome plugin I’ve also used http://vimperator.org/

The suckless guys actually created a browser named Surf, which has this keyboard-driven control as default. https://surf.suckless.org/

I agree, module arrangement would be the last thing which would need a mouse after this plugin.

[gc3 (gc)]

@LarsBjerregaard - excellent! no worry about deadlines, anything you can think of sounds great.

@dwat3r - thanks for the link, that looks neat!

Both (& everybody else interested) - I’ve started with the pads model b/c it unifies keyboard and MIDI controller use (and flexibly scales down to 2x8 and 2x4 configurations) but there’s no technical reason why a keyboard-requiring approach that used letter/number “paths” instead of directional “paths” couldn’t be implemented; in fact it would actually be a pretty quick thing to test out (or bake in) as an alternate selection mode. I would be interested to know what’s a better experience as the directional selection feedback may move more quickly from screen=>eyes=>brain=>fingers even if the paths are longer. But that’s a perfect candidate for alpha/beta test feedback and if you want to think through what a full-keyboard system design would look like I can definitely work up a prototype for it!

Yes, patching is not the end goal here by any means.

Here are Rack’s interface layers (I think this is a complete list, please add if I’m missing anything!) and what I think can be done with them from the pad and key perspective (in roughly priority order, although some of the module-level stuff could bump up if the interface design becomes clear because I’ll have implemented module select in the patching context):

1. Patching–pretty much covered by feature list, proto is already highly usable
2. Standard controls (basically anything that can be mapped with existing tools)–LOTS of possibilities here, more details soon
3. Module-level operations (e.g. disable, disconnect, randomize, delete): already have keyboard shortcuts which module select could pass through; for pads, module select plus modal pad commands would work fine, though the positional mapping will be somewhat arbitrary
4. Module movement post-creation: tricky interface design but doable with module select and move keys (probably something like “place a shadow”=>“move to shadow”)
5. Module insertion: this is tricky insofar as it has a different positional concept (mouse in empty space) and would need a custom menu to be workable with pads, but @stoermelder MB has shown that it is completely doable; just need the design.
6. Nonstandard controls and right-click module menus: probably nothing standardized enough to work with (I think of these like DIP switches anyway, so if going to the mouse is like unscrewing something from the Rack, that’s fine)
7. Top-bar menus: probably nothing (and nothing needed)
8. edit: I should really list view here, which supports the others; some existing keyboard shortcuts, ots of possibilities with pads (including integrating with other modes)

This is a V1 list, of course; V2 will add some other things (more hover feedback, etc.) which raise interesting possibilities of their own.

[LarsBjerregaard]

@dwat3r

You’re talking about Vimperator That was an awesome plugin I’ve also used http://vimperator.org/

Yeah! Thanks for the link, and via that I found https://vimium.github.io/ which I definately need to try out on my Chrome browser.

[gc3 (gc)]

Hi all! Here’s a quick update, since it’s past the end of May

I’ve been able to progress on this pretty steadily despite a scarcity of time. The technical debt from the prototype is mostly paid down, so the module has a sensible structure, proper command/event system, etc., and the alpha is close to ready (I’m hoping to start it sometime next week). Some of the planned alpha features have been bumped (either to a second alpha or to the beta), but the core features have been expanded and refined in a way I’m pretty happy with. I’ll update the public feature list once the first alpha is out, in case anyone’s interested in following along.

Right now I have a decent group of people interested in the first alpha, but I could add a few more, so let me know if you want to get involved at this point. This is also the best time for big feature requests, as the interface is still finding its shape, so post here or message me if you have any ideas, no matter how outlandish!

Existing alpha testers, thanks again, and I’ll be in touch shortly by message.

[rsmus7 (Stephan)]

I would like to jump in for testing the module

[gc3 (gc)]

@rsmus7, fantastic! Many thanks!

I’ll send around a 3-minute survey (regarding setup/interests/etc.) in a few days and follow up soon after with builds and/or code.

[dwat3r (Oliver Hechtl)]

I have one idea in mind for total control over Rack with the module: knob / fader / button setting. It could work like you tap a different key, then enter the parameter adjust mode, and after select the knob / fader / button the same way you select a port, then for knobs / faders you’ll have eventually 2 keys, for buttons only one. With this, you can patch and configure a fixed rack just with this module.