Tip: Rack SDK with Catch2 via vcpkg and CMake with CI/CD

[StoneyDSP (Nathan)]

Hey all.

I haven’t been around here much, so apologies for any lack of etiquette in my posting.

Recently, I made a start on learning the Rack API as means to upskill on C++ audio DSP testing/debugging, and such forth.

To facilitate this, I’ve come up with some workflows for building, testing, and deploying Rack plugins, CI/CD style, which I feel are working well enough to share, in the case that it may help others who are trying to figure out similar things.

In my dep directory, I created a VCV Rack SDK CMake project file which accepts/requires a -DRACK_DIR="/path/to/unzipped/Rack-SDK", pointing at some unzipped copy of the Rack SDK on your local filesystem - and parses all of the contents into relocatable CMake targets which other CMake projects can include as a package dependency, and then link with.

I’m using vcpkg’s “filesystem registry” functionality to tell vcpkg that there is a CMake package called rack where that CMakeLists.txt file is. My plugin project depends on this rack package, so vcpkg fetches and parses it for me, and then I just link to rack::lib or rack::sdk.

Additionally, I’m using vcpkg to pull in Catch2 - and potentially, future dependencies, if I should choose to incur any.

The “normal” Makefile workflow is still functioning as expected for regular build/install commands. The vcpkg/CMake/Catch2 sits atop or alongside it, as the “developer” workflow, for myself and contributors, etc.

NOTE: I have not published any packages to vcpkg’s registry. The way this works is just by using local files for the rack package. I’m aware of protocols about unofficial packages in the vcpkg registry, but don’t currently plan to push this package anyway.

The current implementation is a little bulky, and rough around the edges; I’m likely to change some naming conventions, possibly improve the rack targets in various ways, and I may move the VCVRack faux-project files into it’s own repo… but, it works - at least, as far as I can currently tell. It would also be easy to extend this for different Rack SDK version numbers (but that is beyond my own personal need; PR’s welcome )

I hope I’ve managed to explain the mechanism clearly, but I’m sure the code, workflow files, and pertaining docs will be helpful otherwise. I’m happy to provide further insights also.

I have learned a whole tonne of things from lurking here and other audio programming forums over the years (definitely couldn’t have gotten the CMake package off the ground without this very forum), so I just thought I’d offer something back. If there is any interest - and it doesn’t step on any licensing issues - I’d consider parting this implementation out in some way such that it could be more easily shared and used by others. Elsewise, feel welcome to peruse the repo. If you know CMake and vcpkg, it’s fairly straightforward.

Thanks for reading, looking forward to sharing some cool modules with you all in the near future!

[Steve_Russell]

I’ve been following your repo closely for a little while now - I tend to repo search for new VCV stuff and came across yours. Very interesting; I myself am wanting to learn Cmake more as a lot of C++ projects I like use it and want to get it going for some of mine in the future.

Good stuff and looking forward to the modules!

[StoneyDSP (Nathan)]

Thanks for your interest, I appreciate it.

As yourself, I got interested in CMake due it’s adoption by other projects and libraries; since Rack itself does not have a CMake API, I wouldn’t really argue that this approach offers benefits over others that I’ve seen here and elsewhere… but, I think it will scale well over time, and will fit my own needs in various ways that might perhaps be more clear to others once I get further along using it with my own modules.

Since it’s working ok for my current needs, I’ll now be focusing on module development for the most part. But I am curious about fleshing it out (adding more SDK version numbers being an obvious idea) and improving on things I’ve possibly misunderstood (correct vcpkg triplet for MacOS? Should I target arm64, or x64?).

So with that said, I already opened an issue where I’m logging some thoughts on upkeep and future development. I welcome any interest, feedback, contributions, questions, etc from the Rack community.

Hope to be seeing you around!

EDIT: for anyone interested in learning more about CMake, I recommend going through “Mastering CMake” from the official webpage, as well as the Importing and Exporting guide, and the official docs, of course.

[baconpaul (Paul)]

Although I don’t use vcpkg, I’ve been cmake only in rack for my projects for a few years now, following the excellent work from @qno to make the rack api build able with cmake (GitHub - qno/vcv-plugin-cmake-example: Example repository for developing a VCV Rack plugin with CMake.)

This means Clion gets me full debug builds, debugger access, and more. And more importantly I can directly integrate all my cmake dependencies. It’s great!

So basically “I agree it’s super awesome” once you get it set up and if you are looking in this space you may want to look at surge, airwindows, baconmusic etc… which are fully cmake based and work in the library the docker container etc

[StoneyDSP (Nathan)]

Hi @baconpaul , thanks for stopping by - I’m a big fan, quite aware of your excellent work

Being also a huge airwindows fan and advocate, I’ve been aware of your contributions to the build system Chris is using nowadays to target all those platforms, not to mention Consolidated and much else… and that is some inspiring work indeed; salute!

CLion is something I’ve not really used; My two preferred CMake extensions for VSCode, along with the testing and debugging workflows and all that stuff, seem to cover everything. But that highlights one cool thing with CMake - the IDE integrations out there tend to be quite robust, no matter the IDE.

I had not seen this CMake template before, but it clearly works well.

About a year ago, I wrote a CMake API for cmake-js - it allows you to write NodeJs binary addons with a pretty simple function signature in CMake. Actually, I “borrowed” much of the design from vcpkg itself. It took a while to get it just right… but it worked out pretty well I think.

My Rack plugin’s CMakeLists.txt is already getting wildly repetitive, and I’ve only got three modules in there (plus the plugin, and the Catch2 tests). So, I’m feeling a strong urge to just wrap all of the Rack plugin requirements into a couple of CMake functions - just like the cmake-js one did with the Node Addon API - and to write my CMakeLists more functionally…

The CMake package would export the targets (dep, core/sdk - name undecided, and lib), along with a CMake API (some helper functions) that would look a bit like this:

function(vcvrack_add_plugin __slug __sources __version)
    cmake_parse_arguments(...)
    # This lib name is hard-coded; CMake only permits one of these per-project! ;)
    add_library(plugin SHARED)
    add_library(${__slug}::plugin ALIAS plugin)
    # validation: plugin 'VERSION_MAJOR' >= 2...
    set_target_properties(plugin PROPERTIES VERSION ${__version})
    target_sources(plugin PRIVATE ${__sources})
    target_compile_options(plugin PRIVATE "-fPIC")
    target_link_options(plugin PRIVATE "-static-libstdc++")
    target_link_libraries(${__slug}::plugin PUBLIC 
        rack::dep 
        rack::sdk 
        rack::lib
    )
    # ...and so forth!
endfunction()

function(vcvrack_add_module __name __slug __sources __version)
    cmake_parse_arguments(...)
    # This one accepts a name, but is joined to 'plugin' by the alias namespace (the 'SLUG' param)
    add_library(${__name} OBJECT)
    add_library(${__slug}::${__name} ALIAS ${__name})
    set_target_properties(plugin PROPERTIES VERSION ${__version})
    target_sources(${__name} PRIVATE ${__sources})
    target_compile_options(${__name} PRIVATE "-fPIC")
    # Since we can be sure that *all* modules must link with a target which has the name 'plugin' 
    # (since we hard-coded the name), the user just passes in the desired namespace/'SLUG' to link with :)
    target_link_libraries(${__slug}::plugin PUBLIC ${__slug}::${__name}) 
    # ...and so forth!
endfunction()

A consumer project’s ideal CMakeLists.txt might then look like this:

cmake_minimum_required(...)

project(StoneyVCV VERSION 2.1.107)

# The 'helpers' component would be the functions...
find_package(Rack 
    VERSION 2.5.2 
    COMPONENTS dep sdk lib helpers
)

# This would create a redistributable shared library called 'plugin.{so,dylib,dll.a}' with lots of platform-dependent defs...
vcvrack_add_plugin(
    SLUG StoneyVCV
    SOURCES "src/plugin.cpp"
    HEADERS "include/plugin.hpp"
    VERSION 2.0.31
)

# This would create an object library; 'plugin' links to it via 'SLUG'...
vcvrack_add_module(
    NAME HP1
    SLUG StoneyVCV
    SOURCES "src/HP1.cpp"
    HEADERS "include/HP1.hpp"
    VERSION 1.0.2
)

# Create as many modules as you wish with it...
vcvrack_add_module(
    NAME LFO
    SLUG StoneyVCV
    SOURCES "src/LFO.cpp"
    HEADERS "include/LFO.hpp"
    VERSION 1.0.1
)

vcvrack_add_module(
    NAME OSC
    SLUG StoneyVCV
    SOURCES "src/OSC.cpp"
    HEADERS "include/OSC.hpp"
    VERSION 1.1.4
)

# Since these functions would create normal CMake targets, the normal CMake functions would still work too..
target_compile_definitions(OSC
    PRIVATE
    "DEBUG"
    "OSC_SPECIFIC_VARIABLE=foo"
)

# Other deps could then be added customarily, on a per-module basis...
find_package(StoneyDSP
    VERSION 1.0.0
    COMPONENTS Core SIMD
)

target_link_libraries(LFO
    PRIVATE
    StoneyDSP::Core
    StoneyDSP::SIMD
)

That’s pure pseudo-code of course, but it’s the general idea.

I should add that I haven’t tried runtime-testing my CMake-built plugin on all platforms yet; I’m definitely interested in tips about runtime testing in CI/CD runs, where there is no display…

Thanks for indulging my thoughts; and, for all your contributions to the scene!

[baconpaul (Paul)]

Right so that looks interesting. The thing I would do early-rather-than-late is build the docker toolchain (or use the GitHub actions toolchain build) to see if you are using deps which are not in toolchain. For instance, I think the toolchain doesn’t have vcpkg. Building in the toolchain is a pre-req of getting in the library and it is a constrained environment (basically a custom ubuntu image with enumerated deps).

One of the things we had to do for instance was chat with cschol to get cmake to the level we wanted.

CLion is great, but vscode is also super popular. The thing I didn’t want to do was to rewrite the surge build system in make just for rack. That would have been a non-starter, which would be a bummer, since surge in rack is pretty popular.

Happy to offer thoughts if it helps and also happy for you to copy any of the code we have in surge-land or in the other projects.

[StoneyDSP (Nathan)]

Thanks!

I’m not sure I understand why I would need the Docker image - based toolchain, for anything in particular, though? My CI/CD workflows are already green, doing make install and uploading the *.vcvplugin, just using the deps listed in the “setting up your environment”… it’s just regular GitHub runner installations I’m using (plus an MSYS action for Windows) to get the deps.

My VCVRack/ CMake / vcpkg - thing fetches the Rack SDK from the Rack website, arranges them into targets, and all the rack dev files and lib all end up in your binary dir, where vcpkg links them in when it controls CMake for you… It’s all working since before I started the thread; I’m just refining it and throwing more ideas around now.

Perhaps there’s something I missed about why I would need to use the toolchain instead of the SDK, but the modules appear to be working just fine here at home. Not sure… I’ll probably find out sooner rather than later, though!

Thanks for stopping by

[baconpaul (Paul)]

You need the docker toolchain so you can see if the library build will work. Which is how you get in the library. Basically

[StoneyDSP (Nathan)]

Maybe you’re talking about automated runtime tests here? It’s definitely something I was really hoping to find our more about from around here

[baconpaul (Paul)]

No im saying something simpler

When the team at vcv rack builds your plugin so it can be available at library.vcvrack.com they use a constrained os image and cross compile windows and Mac for Linux. If your system doesn’t work in that environment you can’t share your plugin in the library. If you do things like assume your host is and target os are the same you will break. The os may not have vcokg. Etc

If you never plan to distribute your code in the library it’s ok but that limits the utility of your system

[StoneyDSP (Nathan)]

Ah, gotcha! Thanks!

I’d have to ship them pre-built libs, per-target, which is a not-great solution. Guess I understand now why so much module code is inlined into one compilation unit…

There’s a nice script for bootstrapping a vcpkg installation from the command line in their docs. It also works as a git submodule. I’ve got my CMake/vcpkg stuff working with Makefile locally and plan on adding a make dep to automate it all, best I can… but overall, yes, I do see the benefit in checking against the Docker image per your description - thanks a lot for informing me. I’ll give it some thought.

First, I’ll also need to focus on some modules to publish

[baconpaul (Paul)]

I ship prebuilt nightlies - it works great. But be careful the libc version in Ubuntu latest on GitHub is very new compared to most users - for my rack builds I actually use the docker image in GitHub (and for our vst properties use a custom docker image for Linux there too).

But yes all useful paths lead to the library hence my thought to test early either in a GitHub toolchain build (check out the surge or airwin actions) or by building the toolchain yourself

[StoneyDSP (Nathan)]

Thank you, the advice is incredibly helpful; I’ll make it a priority to properly investigate the deployment process and requirements for the user library, and will think carefully about compliance with it, between now and when I’m ready to cross that line.

If I get around it all with this current build system, I’ll share some notes about it here.

[StoneyDSP (Nathan)]

Update:

I have completed what appears to be a fair-enough working implementation of the subject matter of this topic: building plugins against the Rack SDK with CMake, and using vcpkg to manage the Rack SDK as a dependency of my plugin.

In it’s most minimalized form, it allows one to replace the Makefile build system with the below CMakeLists.txt file content:

cmake_minimum_required(VERSION 3.14...3.31 FATAL_ERROR)

project(MyPlugin)

find_package(rack-sdk 2.5.2 REQUIRED COMPONENTS dep core lib CONFIG)

vcvrack_add_plugin(
    SLUG "MySlug"
    BRAND "MyBrand"
    HEADERS "include/plugin.hpp"
    SOURCES "src/plugin.cpp"
)

set(MYPLUGIN_MODULES)
list(APPEND MYPLUGIN_MODULES
    LFO
    VCO
    VCF
    VCA
)
foreach(MODULE IN LISTS MYPLUGIN_MODULES)

    vcvrack_add_module(${MODULE}
        SLUG "MySlug"
        BRAND "MyBrand"
        SOURCES "src/${MODULE}.cpp"
    )

endforeach()

The project is configured with vcpkg via CMAKE_TOOLCHAIN_FILE, which looks up and finds the Rack SDK, and those helper functions, and allows the find_package() to pull them in:

cmake -S ${PWD} -B ${PWD}/build -DCMAKE_TOOLCHAIN_FILE="${VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"

Build the project like normal CMake:

cmake --build ${PWD}/build

There are numerous other functions in the package which I’ll document at some point. The two main ones, vcvrack_add_plugin() and vcvrack_add_module(), both have some cool extended functionalities, like being able to turn your CMake-powered plugin into a shareable CMake package; I’ll also document all of this properly a bit later. Overall, it isn’t very complex, isn’t too highly opinionated* about what you try to do with it, and strives to just wrap the repetitive stuff and the platform-dependent stuff into a tidy function, with a little bit of input validation.

vcvrack_add_plugin() creates a target named plugin, so can only be used once per project. The name of this target cannot currently be changed… but you can name your source/header files anything you want, of course

Disclaimer: I have manually installed the build output by scraping the shell commands from the end of make install and adjusting them to fit the CMake build output. On my home machines, everything seems to be running.

However, I want to put it through it’s paces now by scaling up a bunch of modules, in the course of which I’ll possibly encounter quirks, bugs, unintended behaviours… or, better ideas than what I’ve got already.

At that point, I will probably move this Rack SDK CMake stuff into it’s own repository under my org, so I can manage it separately (tests…) and version it and stuff.

I moved a few files around in the project tree, so my apologies if any links in this thread are now broken. I’ll go over the posts later this evening and edit my posts accordingly.