Background

OS: Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-211-generic x86_64)

I want use tar to compressing many large files.

Appearance

The compressed file content is close to 1TB, I use the tar -czvf target.tar.gz target/ to compress the target files, it was very slow.

Reason

The tar command uses a single thread.

Solution

Compression using the multi-threaded tool pigz

Use pigz

Install pigz

1

apt -y install pigz

Compression with pigz

1

tar --use-compress-program=pigz -cvf target.tar.gz target/

Result

Using the top command, we can see that the cpu usage is significantly higher, and the tar execution compression process is singificantly accelerated.