Pretty sure this used to be the fix for me:
https://forums.linuxmint.com/viewtopic.php?t=314918
( scroll down to comment about default.pa and tsched=0 )
But I just checked my default.pa and it is stock values, so I am not sure anymore
When experimenting, setting cpu governor to performance also helped.
You will not believe this. Solution was adding a line in some config file.