Short answer: larger frames/period setting (256/512/1024) and more Periods/buffer.(2/3) - is where i would start. And make sure to run a “PREEMPT” kernel - or compile an “RT” kernel. The CPU Governor should be set to Performance.
There’s much to tweak on linux
https://wiki.linuxaudio.org/wiki/system_configuration
and the forum here is a good place to seek information and ask.
Perhaps start out with a fresh install of “Ubuntu Studio”. It has “Ubuntustudiocontrols” to make audio setup easier.
Some prefer Arch linux (Manjaro) - They usually have high quality wiki pages on configuration. Professional audio - ArchWiki