Unit Testing

Development
1

I’m curious if any of you uses unit tests in their plugin development workflow, and if so, which testing framework/library would you recommend for VCV or C++ in general ?

2

I use Google Test. I find it quite awkward compared to the unit test frameworks I use in other languages, but I don’t know of a less awkward one for C++.

1 Like
3

What do you find awkward about it ?

4

Let me throw in here Catch and Hippomocks for Mockups.

1 Like
5

I’ve started using catch for the start of some unit tests in surge and find it useful also. Seconded!

1 Like
6

I had a rapid look at it and it looks both simple & powerful indeed. Thanks for your advice guys !

7

I used https://github.com/JerrySievert/arptest previously, but haven’t updated it for 1.0 yet - it would be fairly trivial to do if there were interest.

see https://github.com/JerrySievert/arptest/blob/master/example/tests/Mult.cpp#L18-L58 for an example test of a clock multiplier.

1 Like
8

Here’s a small set of tests for the surge synth

On that branch the tests fail (something for 2020 - sigh) but the azure pipeline for surge builds and runs the test as part of pr validation

1 Like
9

Squinky Labs plugins are all designed test first. I think 2/3 our code-base is unit tests. I don’t use any framework, just build a command line c program and assert that the results are correct. The google stuff is pretty good, I use that at work.

Here’s the source for our tests: https://github.com/squinkylabs/SquinkyVCV/tree/master/test

I find it almost impossible to do DSP programming without unit tests. Very small errors can be difficult to find by ear. Filters are often at the wrong frequency, bleps aren’t added at the right offset, etc…

10

Prefer to let the creativity flow so don’t unit test per-say but ultimately I am the unit test!

1 Like
11

Interesting, I wasn’t aware there has been experimentations in that domain. I wonder if it would even be possible to “mock” the rack engine and build integration test with simulated user input and so on. Seems like a massive task though.

12

CI workflow is a must for collaborative development. Has been using travis for a long time and it’s really a life changer when maintaining a project.

Interesting stuff ! Is the Surge::headless() part of the original surge project or something you developed to simulate integration with VCV ?

13

Yes, testing complex code is a must ofc. I rarely do unit-test first development though, because my class hierarchy is often subject to change in the first half of the project, which mean having to rewrite tests as well. So most of the time I start with a few high level integration tests and start unit testing when I’m 99% sure I have the final design.

If you don’t use a framework, do you use mocking or just build your own stubs ?

14

I’m a big fan of test-driven development (TDD). At long last I’ve found a pattern that not only allows me to test most of my code, but also allows me to test-drive new functionality. The pattern is to organize each module into several components:

  • The “engine” performs the module’s main computation.
  • The “controls adapter” acts as the interface between the engine and the Rack controls (inputs, params, outputs, and lights). The adapter’s main job is to insulate the engine from the details of where the values come from and go to. If the engine needs a duration, it simply asks the adapter for a duration. The adapter might read several inputs and params to compute the value (e.g. a duration knob, CV input, attenuverter knob, and duration range switch). The engine knows nothing about Rack controls. It simply asks the adapter for a duration and the adapter gives it a duration. Similar for outputs.
  • The “module” connects the controls adapter to the Rack controls, connects the engine to the adapter, and then simply forwards each “process()” call to the engine. That’s it. I don’t test the module itself, but there isn’t much left for it to do, given that all of the computation happens in the engine and the controls adapter.
  • The “panel” displays the control widgets. I don’t test this, either.
  • Additional components as needed to keep the engine simple. If these components send information to and from Rack, they do so via the controls adapter.

With the code organized in this way, I can readily test-drive new engines and controls adapters:

  • For a controls adapter, the test setup creates vectors of real Rack controls and passes them to the adapter. Then it’s straightforward to verify that each adapter function sends information to or from the appropriate controls.
  • For an engine, the test setup mocks the controls adapter and passes the mock adapter to the engine’s constructor. Then each test mocks the adapter’s expectations and responses, and executes the relevant engine function.
  • For other components, the tests mock the controls adapter if needed.

If that sounds like a lot of work, it’s because it’s a lot of work. But it lets me test-drive my code, and gets my core functionality under test. I find it worth the effort.

2 Likes
15

Funny, I’m working on a bigger project since last month, and after a first refactoring I ended up using a very similar architecture to the one you describe, though my motivation wasn’t testing in the first place; the goal was easily merge controls from several sources of inputs not just the VCV params/interface. But I guess after years of TDD we end up designing with testing in mind from the ground up.

1 Like
16

I added surge headless to find a bug with filter initialization in April 2019. (Ironically the test for that didn’t become a unit test). Once I added it I realized I had 90% of what I needed to build for rack so rack came after. Then this fall I turned it into more structured unit tests as I started thinking of a bigger restructuring in 2020

17

hah - I do a crazy ad-hoc thing. I write my modules so that they are a very-very thin wrapper around my class called Composite. Most of the interesting stuff is at the composite level or lower, and that stuff it not allowed to call any rack code at all.

It means I can’t test UI code at all, but that’s usually not a problem. Although there are times I would like to be able to test UI-related code, but I can’t at the moment. I’ve been considering linking to rack code directly from my test code, but have not pursued that.

18

Ah, I hadn’t read your reply until now. Sounds like you do a more organized (architected?) version of what I do.

19

I believe it is the obligation of every dev, no matter what they are working on, to deliver code of adequate quality. In my long programming life unit tests + TDD is the only way I have found that works for me.

I guess in VCV land there are devs who must do traditional “black box” testing of their stuff to get the level of quality they are going for. That’s fine of course. There are well-known trade-offs here.

And, of course, there are always a few module around where the quality couldn’t really be called “adequate”.

1 Like
20

“Architected” is a very generous characterization :wink: I moved code around more or less randomly for a year, and eventually this pattern emerged.