Speeding up extracting of large movie files
Is there a way to speed up extraction of 40+gb 4k videos? This is by far the most time consuming part of the process and I was wondering if there was any hardware acceleration or even multi cpu way of speeding this up.
My cpu isn’t the newest, it’s an i7 5820k, overclocked just a bit, but I have a hard time believing that a newer cpu on its own will have more than marginal gains on this front, but I could be way off here.
edit: I just realized my processor came out in 2014. So I'm sure in 10 years we've made some inroads, I thought I bought this guy in 2018.
The unpack process is usually single threaded, and a stock 5820k is slower than a ryzen 1600 in that case, so you're probably just running out of CPU performance. My server runs a 6700k and it's pretty slow to unpack even with nvme arrays
Excluding everything that is more difficult to deal with (like getting an SSD/more RAM/more swap on Linux), have you tried using different extraction apps? Peazip is good: https://peazip.github.io/
Edit: Here are some benchmarks for fastest extraction, Peazip is very good https://peazip.github.io/peazip-compression-benchmark.html#7-zip_vs_winrar
I have not, I will give it a go, thanks!
Your drive speed might be the limiting factor here, not your processor.
Interesting, I'm using a synology NAS in order to store lots of data, it should be relatively speedy but probably not compared to an onboard SSD. My SSD only has about ~100gb of free space so even using it as a temporary download center may not be right, looks like I'm looking into a terabyte nvme. Thanks for the suggestion!
Are you sure network isn’t a bottleneck? Even with a gigabit network, copying 40gb worth of data back and forth can take a while.
Yeah network speed for gigabit is max 125MB/s. Or closer the 115MB/s including network overhead.
Even with 2.5G networking I'm only hitting 250MB/s for my NAS. I've never seen anything close to 300MB/s
To be clear, I'm not saying that I'm sure your bottleneck is the SSD. But it could be either that or the processor.
If you're using the *arr apps to download, consider using unpackerr because it'll handle extraction and cleanup in the background.
I am, this is cool, thank you!
tf are you getting 4k torrents that are ziped and why??
It’s usenet, everything is zipped into hundreds of RAR files
What type of speed is your hard drive?
It's hard to say since it's all RAIDed out but they're not SSD's, I just didn't have the space on the drive. But looks like I'm in the market for a new cpu, motherboard and NVMe drive now. Thanks for your input!
Are you unpacking them directly on your NAS?
Because chances are, that if you use a Synology and map the drive or use the GUI, that the data keeps getting transferred between your PC and the NAS. This is a very slow process, especially if you connect by WiFi.
Some players can play rar files directly. Also look into rar2fs. I just try to avoid rars if possible