Help needed: VCVrack high CPU usage on Linux

Hi,
Running Vcv Rack and the multi core experimental FreeRack on Windows for a while now on my HP all in one pc without problems. Now I installed both on my Mint Linux system which is quite fast 32gb ram 6 core fast cpu. Nvidia 660 graphics. Running Rack or FreeRack shows me that even mults take up about 40 to 50 ms. Had to switch to FreeRack adding additional cpus for even simple patches. Don’t think it’s gpu because resizing and zooming in Rack is much faster than on my hp windows pc and running only in a small window does not influence the high cpu consumption. Does anyone has an idea what is causing this?
Regards Dieter

To give a little bit more information. It looks like that EVERY module takes about 40mS base load without doing much, even mults.

in linux its all about proper jack, governor and realtime prio settings - i’m using vcvrack with surprisingly good results even on small arm boards with proper tuning (https://github.com/hexdump0815/sonaremin). here are some settings to give you a start:

  • use jack and allow realtime prio for it
  • use the “threadirq” kernel option if it works for you (on some systems jack was strangly hanging with it on other systems it works flawlessly)
  • if it works, then: systemctl stop irqbalance
  • some kernel settings (maybe google for the details):
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 524288 > /proc/sys/fs/inotify/max_user_watches
echo 1 > /sys/module/processor/parameters/ignore_ppc
# get the max_possible_frequency from cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
for i in /sys/devices/system/cpu/cpu?/cpufreq/scaling_max_freq ; do
  echo max_possible_frequency > $i
done
for i in /sys/devices/system/cpu/cpu?/cpufreq/scaling_governor ; do
  echo performance > $i
done
chmod a+r /dev/rtc0
chmod a+r /dev/hpetsysctl -w kernel.sched_rt_runtime_us=-1
  • maybe start vcvrack with realtime prio as well: chrt 20 Rack
  • use two threads if one is not enough (more make no real sense)
  • use low screen refresh rates (for me it works well with 15hz in vcvrack)
  • lower sample rates are better than higher (32khz is better than 44.1khz is better than 48khz etc.)

i think this should give you enough things to try out … let me know if it works out for you

best wishes - hexdump

Hi hexdump,
thx for your information.
Dumped my mlinux mint and installed ubuntu studio 18…04 instead.
Got the same problems.
Allready did everything you suggested except the:
chmod a+r /dev/hpetsysctl -w kernel.sched_rt_runtime_us=-1
because there is no /dev/hpetsysctl on my system.
Still have the cpu problems .
Switching on cpu meters shows about 5% !!! on each module even if no no audio device is selected.
With my much slower notebook running centos 7 everything works fine.
Also already tried also and jack.
Sound is working fine.
Also changing frame rate from 70 to 30 didn’t change anything.
top is showing about 150% CPU for Rack with a really simple patch.
Any other idea whats going on here ?

Thx in advance
dieter

What modules are you using? Its happening to every single module across the board?

Yes it is every module.
Tested with just Impromptu Clocked, a Vult Knock, a VCA-1 and Audio.
top shows two lines for Rack:
2170 dieter 20 0 958616 133744 73104 S 205,0 0,4 33:51.86 Rack
2897 dieter 20 0 983900 181348 151104 S 55,8 0,6 0:19.47 Rack
sums up to 260% CPU !!! with threads only set to 1 !!!
Rack CPU Meters:
Clocked 4.8%, Knock 4.9%, VCA 4.5%, Audio 67.1%

Update: there was a process for rack still running which I killed.
So the correct measures for the trivial patch above:
1 Thread:
top shows 77 % CPU
and cpu meters in rack: Clocked 6.3%, Knock 6.6%, VCA 5.6%, Audio 57.3%
6 Threads:
top shows 220 % CPU
and cpu meters in rack: Clocked 8.8%, Knock 9.2%, VCA 7.7%, Audio 84.4%

Hi Dieter,

FWIW, it doesn’t here either (and I have decent/good performances): the difference between Rack in a minimum-sized window and 1080p full screen is barely noticeable.

Hi mixer,
thx for your suggestions.
Yes I’m running the proprietary Nvidia driver which works fine and fast.
Did everything mentioned in the survival kit.
Even built Rack successfully on my system.
Running from ‘make run’ gives me a default patch with AUDIO-8 and NOTES.
Both showing 4.5% CPU with no audio connected :frowning:

Are you getting a lot of xruns? I’ve had some strange behaviour lately with what I think are kernel crashes when using Rack and even without high cpu usage I get quite a lot of audible xruns.

The crash issue is puzzling, not sure if this is just something in my system that’s gone wrong or Rack that’s causing it.

Hi TroubleMind,
No, even with low blocksize of 128 I do not get xruns.
Sound is working fine. Every module just takes huge amount of CPU.
Just a Clocked, Knock, VCA and AUDIO above 50% CPU.

Maybe of intrerest ist that when adding a module like NOTES it appears with low 0% CPU usage and then ramps up to 4.5% CPU usage (about 1 us) in about 5 seconds.

Just adding 5 NOTES ramps up CPU from way over 100% (using linux top command to monitor)

Hmm, I’m not sure then. Are you using a clean /home with the new Studio install or did you reuse your existing configs etc?

Didn’t reuse any existing config files.
Just installed ubuntu studio 18.04 from scratch and Rack.
For some reason it looks like any module will create a CPU load of 1us after about 5 seconds.
This doesn’t change when setting engine threads to 6 but CPU load is going to 150%.

Switching from CPU performance mode to energy saving mode it gets even worse.
No change for single thread but using 6 engine threads module cpu usage ramps up to about 7% and Rack is consuming about 225% CPU.

Just used gdb ./Rack (own build) and run,
occasionally stopped with ctrl-C then continuing.
Most the time the ngdb stacktrace shows:

0x00007ffff56c3bf9 in __GI___poll (fds=0x7fffffffd810, nfds=1, timeout=1000) at …/sysdeps/unix/sysv/linux/poll.c:29

Suspicious might be, that the timeout of 1000 matches with the about 1.0 us

The backtrace in gdb look mostly like this:

Thread 20 (Thread 0x7fffd9a14700 (LWP 4927)):
#0 0x00007ffff7ffab62 in clock_gettime ()
#1 0x00007ffff56dfea6 in __GI___clock_gettime (clock_id=0, tp=0x7fffd9a139f0) at …/sysdeps/unix/clock_gettime.c:115
#2 0x00007ffff600b30e in std::chrono::_V2::system_clock::now() () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x0000555555791d25 in rack::engine::Engine_stepModules(rack::engine::Engine*, int) (that=that@entry=0x555555f16660, threadId=0) at src/engine/Engine.cpp:254
#4 0x0000555555794624 in rack::engine::Engine_step (that=0x555555f16660) at src/engine/Engine.cpp:335
#5 0x0000555555794624 in rack::engine::Engine_run (that=0x555555f16660) at src/engine/Engine.cpp:432
#6 0x0000555555794624 in rack::engine::Engine::<lambda()>::operator() (__closure=) at src/engine/Engine.cpp:461
#7 0x0000555555794624 in std::__invoke_impl<void, rack::engine::Engine::start()::<lambda()> > (__f=…) at /usr/include/c++/7/bits/invoke.h:60
#8 0x0000555555794624 in std::__invoke<rack::engine::Engine::start()::<lambda()> > (__fn=…) at /usr/include/c++/7/bits/invoke.h:95
#9 0x0000555555794624 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::_M_invoke<0> (this=) at /usr/include/c++/7/thread:234
#10 0x0000555555794624 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::operator() (this=) at /usr/include/c++/7/thread:243
#11 0x0000555555794624 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > > >::_M_run(void) (this=) at /usr/include/c++/7/thread:186
#12 0x00007ffff601366f in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff7bbd6db in start_thread (arg=0x7fffd9a14700) at pthread_create.c:463
#14 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7fffd3bfe700 (LWP 4926)):
#0 0x00007ffff56ca839 in syscall () at …/sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007ffff6ec6dc7 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#2 0x00007ffff6e9f8b5 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#3 0x00007ffff6e9e8af in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#4 0x00007ffff6e9e048 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#5 0x00007ffff6ec49d6 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#6 0x00007ffff7bbd6db in start_thread (arg=0x7fffd3bfe700) at pthread_create.c:463
#7 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7fffd80f5700 (LWP 4925)):
#0 0x00007ffff7bc7384 in __libc_read (fd=33, buf=0x7fffd80f49c0, nbytes=4) at …/sysdeps/unix/sysv/linux/read.c:27
#1 0x00007ffff6ec60ee in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#2 0x00007ffff6ecba15 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#3 0x00007ffff6ec49d6 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#4 0x00007ffff7bbd6db in start_thread (arg=0x7fffd80f5700) at pthread_create.c:463
#5 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7fffd82e2700 (LWP 4924)):
#0 0x00007ffff7bc39f3 in futex_wait_cancelable (private=, expected=0, futex_word=0x55555651a2ec) at …/sysdeps/unix/sysv/linux/futex-internal.h:88
#1 0x00007ffff7bc39f3 in __pthread_cond_wait_common (abstime=0x0, mutex=0x55555651a290, cond=0x55555651a2c0) at pthread_cond_wait.c:502
#2 0x00007ffff7bc39f3 in __pthread_cond_wait (cond=0x55555651a2c0, mutex=0x55555651a290) at pthread_cond_wait.c:655
#3 0x00007ffff6ec584c in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#4 0x00007ffff6eb5875 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#5 0x00007ffff6ec49d6 in () at /usr/lib/x86_64-linux-gnu/libjack.so.0
#6 0x00007ffff7bbd6db in start_thread (arg=0x7fffd82e2700) at pthread_create.c:463
#7 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fffec8d7700 (LWP 4908)):
#0 0x00007ffff7bc7c60 in __GI___nanosleep (requested_time=requested_time@entry=0x7fffec8d6af0, remaining=remaining@entry=0x7fffec8d6af0) at …/sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000555555720d80 in std::this_thread::sleep_for<double, std::ratio<1l, 1l> >(std::chrono::duration<double, std::ratio<1l, 1l> > const&) (__rtime=…) at /usr/include/c++/7/thread:373
#2 0x0000555555720d80 in rack::serverConnect () at src/bridge.cpp:401
#3 0x0000555555720d80 in rack::serverRun() () at src/bridge.cpp:414
#4 0x00007ffff601366f in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff7bbd6db in start_thread (arg=0x7fffec8d7700) at pthread_create.c:463
#6 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7fb2ac0 (LWP 4904)):
#0 0x00007ffff56c3bf9 in __GI___poll (fds=0x7fffffffd810, nfds=1, timeout=1000) at …/sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007fffd897a315 in () at /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
#2 0x00007fffc32fb9c4 in () at /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.390.116
#3 0x00007fffc321c97e in () at /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.390.116
#4 0x00007fffd8970db1 in () at /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
#5 0x0000555555741692 in rack::Window::run() (this=0x555555f11810) at src/window.cpp:388
#6 0x00005555556b7645 in main(int, char**) (argc=, argv=) at src/main.cpp:186

When running just single threaded with no audio device, neraly every ‘thread applay all where’ in gdb looks like this:

(gdb) thread apply all where

Thread 20 (Thread 0x7fffd9a14700 (LWP 5196)):
#0 0x00007ffff7ffab62 in clock_gettime ()
#1 0x00007ffff56dfea6 in __GI___clock_gettime (clock_id=0, tp=0x7fffd9a139f0) at …/sysdeps/unix/clock_gettime.c:115
#2 0x00007ffff600b30e in std::chrono::_V2::system_clock::now() () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x0000555555791d25 in rack::engine::Engine_stepModules(rack::engine::Engine*, int) (that=that@entry=0x555555f166e0, threadId=0) at src/engine/Engine.cpp:254
#4 0x0000555555794624 in rack::engine::Engine_step (that=0x555555f166e0) at src/engine/Engine.cpp:335
#5 0x0000555555794624 in rack::engine::Engine_run (that=0x555555f166e0) at src/engine/Engine.cpp:432
#6 0x0000555555794624 in rack::engine::Engine::<lambda()>::operator() (__closure=) at src/engine/Engine.cpp:461
#7 0x0000555555794624 in std::__invoke_impl<void, rack::engine::Engine::start()::<lambda()> > (__f=…) at /usr/include/c++/7/bits/invoke.h:60
#8 0x0000555555794624 in std::__invoke<rack::engine::Engine::start()::<lambda()> > (__fn=…) at /usr/include/c++/7/bits/invoke.h:95
#9 0x0000555555794624 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::_M_invoke<0> (this=) at /usr/include/c++/7/thread:234
#10 0x0000555555794624 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::operator() (this=) at /usr/include/c++/7/thread:243
#11 0x0000555555794624 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > > >::_M_run(void) (this=) at /usr/include/c++/7/thread:186
#12 0x00007ffff601366f in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff7bbd6db in start_thread (arg=0x7fffd9a14700) at pthread_create.c:463
#14 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fffec8d7700 (LWP 5178)):
#0 0x00007ffff7bc7c60 in __GI___nanosleep (requested_time=requested_time@entry=0x7fffec8d6af0, remaining=remaining@entry=0x7fffec8d6af0) at …/sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000555555720d80 in std::this_thread::sleep_for<double, std::ratio<1l, 1l> >(std::chrono::duration<double, std::ratio<1l, 1l> > const&) (__rtime=…) at /usr/include/c++/7/thread:373
#2 0x0000555555720d80 in rack::serverConnect () at src/bridge.cpp:401
#3 0x0000555555720d80 in rack::serverRun() () at src/bridge.cpp:414
#4 0x00007ffff601366f in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff7bbd6db in start_thread (arg=0x7fffec8d7700) at pthread_create.c:463
#6 0x00007ffff56d088f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7fb2ac0 (LWP 5174)):
#0 0x00007ffff56c3bf9 in __GI___poll (fds=0x7fffffffd810, nfds=1, timeout=1000) at …/sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007fffd897a315 in () at /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
#2 0x00007fffc32fb9c4 in () at /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.390.116
#3 0x00007fffc321c97e in () at /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.390.116
#4 0x00007fffd8970db1 in () at /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
#5 0x0000555555741692 in rack::Window::run() (this=0x555555f11820) at src/window.cpp:388
#6 0x00005555556b7645 in main(int, char**) (argc=, argv=) at src/main.cpp:186

Looking at the backtraces, it looks like that the clock_gettime() call is maybe very slow for any reason.
Just guessing.
Any further help would be welcome.
Anyone else running VCVRack on ubuntu studio 18.04 ?

Maybe I just ‘solved’ the problem.
On my system clock_gettime() is incredible slow.
When switching off cpu meter the cpu load load drops dramatically.
So I can use Rack but the cpu meters are useless on my system because it looks like the messurement of time with now() which is calling clock_gettime() will messure the messurement and not the module performance :frowning:
Any tips for getting this fixed in my ubuntu studio setup are welcome.

After reading a bit on the clock_gettime() problem I think the way cpu statistics are evaluated in Rack is not feasible. Better counting clock ticks which is much faster.
CPU meters mainly measuring its own overhead are crap.

No, but I’m running Ubuntu LTS 18.04, with additional kxstudio repositories and currently the 4.15.0-54-lowlatency kernel. Let me know if you need any further details about my system. You seem to know about your problem much more than I do.

Thx for your support. Same kernel on my ubuntu studio.
uname -a gives:
Linux Abraxas 4.15.0-54-lowlatency #58-Ubuntu SMP PREEMPT Mon Jun 24 11:45:10 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux