Unofficial binaries are risky - should we keep posting them?

I have been thinking more lately about how we are all collectively creating a risky situation on this forum by posting binary plugins. Lest anyone interpret my words as sanctimonious, I want to make it clear I include myself in this assessment. Just a few days ago, by request, I posted a pre-release version of my own plugin. And it wasn’t the first time.

Every time I post a link to download vcvplugin binary files, I have an uneasy feeling that I’m helping normalize poor security practices. I know the binaries I post are legitimate. I trust my fellow developers on this forum by default, and I have no reason to believe any specific download I have seen is malicious.

However, it only takes one bad actor among a thousand well-intentioned developers to create a nightmare for everyone.

Recently I even saw someone posting binaries for a plugin that is closed-source. This is taking the risk to another level, and in my opinion, this is definitely a bad idea.

In my mind, starting with the safest ways to obtain a plugin binary, and moving down to riskier levels:

  1. Installing an open-source plugin from the VCV Rack Library. This should be the gold standard because it means VCV Rack maintainers have built the same source code (verified by commit hash) as is publicly visible, and they have reviewed the code and did not spot anything suspicious in it. It also means anyone who can read the code can take a peek also.
  2. Installing a closed-source plugin from the VCV Rack Library. This is a close second to #1. The only slightly additional risk is that a very sneaky exploit slipped by the VCV Rack review process, although I wouldn’t personally lose any sleep about this.
  3. Downloading a binary artifact from GitHub (or other git host), from an open-source project, that was created using its automated build process (e.g. GitHub Actions). At least you have some confidence that the binary was produced from the same source code everyone can see. To be paranoid you would have to review both the source code and the build scripts. On GitHub, this approach requires having a GitHub account, which is free, but many VCV Rack users don’t have one and may not want to create one. Even if you are not a programmer, and you know you can’t verify the code yourself, there is an audit trail that could lead to punishment of anyone with bad intent.
  4. All other binaries. There is no way to know for sure what the code does. It could be stealing data from you or installing malware that keeps running after you close VCV Rack. The bad behavior could be subtle, or it could have a time-delay to make it difficult to correlate with the download. One day something terrible happens on your computer, and you don’t know it was because of that VCV Rack plugin you downloaded several months ago.

I propose we should not install any binaries that fall under #4. I for one have decided I will no longer post binaries like that. Even though I know I am a well-intentioned developer, doing so

  • normalizes bad security practices by setting a bad example.
  • puts myself at risk of accusations if unrelated malware harms a user after someone downloads one of my binaries.

There is still a need for running pre-release plugins; we developers benefit from beta testers. Part of my motivation for this post was to open the floor for ideas. How can we plugin developers better facilitate safe computing? Perhaps #3 in my list is a good compromise, or perhaps there is a better idea? I’m curious to hear what others think.

8 Likes

Good points made here.

I run binaries in category 3 all the time, and we distribute them to thousands of users with the Surge team, albeit with the additional security step of code signing on MacOS. These are considered pretty safe in general but it’s true that someone could absolutely build a malicious binary, expecting (reasonably) that no one will actually read the code straight away, giving time to net at least a few unsuspecting targets.

But I’m gonna keep considering #3 to be good enough unless shown a good reason to do otherwise. It seems like such a common practice and I haven’t heard of any cases (though I imagine some must have occured). So I think it’s fine.

The aforementioned apple signing can work for running pre-releases as well btw! Which would make #3 about as safe as anything can be I guess. That’s what you get if you download the Surge modules nightly. The downside is the dev needs to pay Apple a hundred bucks (and give them lots of data on your person). But if you’re willing to do that you’re good.

Totally agree category 4.

3 Likes

I don’t disagree. Just want to verify that what I do/have done is category 4.

I do a git checkout of the branch that has my test stuff. I do a git pull into a folder in my ubuntu. I run the toolchain to build for all four platforms. I post the resulting 4 plugins in a github release.

Since it’s impossible for anyone to know/verify that is what I did, I assume this would fall into “category 4”.

Unrelated: do you think that the “average” VCV user can distinguish between these categories? Esp 3 vs. 4?

2 Likes

I’m also likely to fall into category 4 if/when I get to releasing the things I’m working on, if only because I have non-standard build toolchain. I’m not sure there’s any way around that. At the moment I’m leaning towards open source anyway, but should I decide to make some things commercial at any point, I’m not sure I’d be able to avoid 4.

Realistically, people run binaries that they haven’t meaningfully verified all the time. I’m not saying that’s a good thing, but I think it’s a realistic thing. I think better protection is likely to be provided by making sure that everything complies with minimal OS-level sandboxing and protections, and train people not to escalate privileges for things which shouldn’t need them (VCV Rack probably being one of those things). I’m not sure of a good answer, and it’s great for people to care!

1 Like

Good question. You know “the code is right there and I can verify that this binary was built from it (granted that I trust Github)” seems obvious to me, but the difference between it and “I downloaded a .zip from a google drive link” may not be as obvious to everyone.

Responding to both of these:

I agree in both cases. If we can move category 4 (posting binaries on the forum) closer to category 3 (binaries built from a source code that can be audited by anyone), that would be a big win. And the less effort and fewer manual steps, the better.

So far I haven’t figured out a solution that feels satisfactory. The closest I have is to only post binaries that come from GitHub Actions, so that people with GitHub accounts can access both copies (here on the forum and directly from GitHub). Someone you trust with a GitHub account can vouch for the posted binaries on the forum by downloading both and doing a binary comparison, or verify matching sha256 checksums.

But that’s pretty clunky. Ideally, there would be someplace the GitHub Actions script could write to, with nothing other than that script having permissions to write to, that could be downloaded by anyone, even if they don’t have a GitHub account. I don’t think anything like that exists, but I would really like to be wrong.

In summary, the best idea I have requires both

  • “trusted” developer(s) on the forum
  • who take manual steps each time to verify another developer’s post

I am going to investigate what might be a slight improvement. It would be to enhance the GitHub Actions yml script most of us use (thank you, @qno!) to calculate the sha256 checkums and include the resulting text file as another build artifact. Same problems, but at least the manual verification steps could be a tiny bit simpler.

So even though this is not satisfactory, I’m posting it in case it sparks a better idea from someone else.

(Edit: I thought of another vulnerability: bait-and-switch. Even if someone verifies a posted binary, the hypothetical bad guy can change the binaries after they are vouched. Sigh.)

Oh you can download a binary from github without having an account. Wouldn’t that be enough?

That is the core of the problem: people without GitHub accounts have told me they cannot see my artifact pages. For verification, here is the build artifact page for my most recent Sapphire release, version 2.5.7:

If you are logged into a GitHub account, you can scroll to the bottom of that page and see 4 zip files you can download, each one having a vcvplugin file inside it. I am told by people who don’t have GitHub accounts that you cannot see or download the files given that link.

You need to tag releases. About releases - GitHub Docs

In all the Surge team repos what we do is for an official release we tag current head as that release, and then never change that tag after. Then there’s a nightly tag which moves to current head each commit. So the “releases” page of the repo has the latest pre-release/nightly build + a history of all real releases. Non-github users can see and download those I’m 99.99% sure.

See this for example: Tags · surge-synthesizer/surge · GitHub

OK, now this is very promising. I can see that the Surge XT releases include the built binaries, which is wonderful.

I have been creating git tags manually for each version of Sapphire, e.g. v2.5.7 for the most recent Elastika CPU optimizations that are now live in the VCV Rack Library. But I didn’t create a GitHub release based on the tag until just now. I just made a 2.5.7 release based on that tag.

But mine does not include the binary artifacts like yours does. How did you tell the release to include the artifacts along with the source code?

Paul can answer that better than me. Though he’s currently on vacation for another couple days so I don’t wanna tag him. :wink: Check back Tuesday!

1 Like

Sounds great, thanks @andreya.ek.frisk !

In the meantime, I asked Microsoft Copilot how to do this and this is an example yml file it responded with:

name: Release Binaries

on:
  push:
    tags:
      - 'v*'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout the repository
      uses: actions/checkout@v2

    - name: Build binaries
      run: make build

    - name: Upload Release Asset
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }}
        asset_path: ./bin/your-binary-file
        asset_name: your-binary-file
        asset_content_type: application/octet-stream

I don’t have time tonight to experiment with this, but I will take a look over the next couple of days. I’m posting it here for anyone else who might also be interested.

I don’t understand. It’s all posted on github, how can anyone but a repo owner pull a “bait and switch”? Oh, I guess you must mean if the developer is malicious?

I meant if someone posted a link to say Google Drive that is supposed to match what is on GitHub, for people who don’t have a GitHub account. Someone else could verify they are the same binaries, but then you could change the Google Drive links later.

Hopefully creating a GitHub release is the answer to preventing bad behavior like this, while still allowing anyone to download the binary.

1 Like

One way to try to ensure a file is what it’s supposed to be is using SHA-1 sums (or higher), this for example, would be the one for the current Windows Mutants Nightly:

8C5B3992E7751E74B5662E4D6F0A0F018A6BE2E1

I’m pretty sure they can be generated using GitHub actions.

IIRC SHA-1 has been compromised. Still, better than MD5, which was compromised much longer ago.

SHA-256 (or higher), then.

2 Likes

OK, I think for Sapphire, I now have a working “category 3” pre-release build process. At least, I do if people without GitHub accounts can download the files.

Do I have a volunteer to take a look at my new development release page? Just let me know if you can see and download one of the vcvplugin files without being logged into GitHub.

And yes, sha256 is my preferred secure hash algorithm these days. I included checksum files in that page too, one for each of the 4 supported platforms.

Thanks to @andreya.ek.frisk for pointing me to Surge XT as a reference. I based my yml script on the theirs, for any other developers who are interested.

1 Like

I just did, you can, indeed, look at the page and download files :slight_smile:

1 Like