Compress Large Files Faster | pigz | Parallel gzip compression | Tech Arkit

       Pigz  compresses using threads to make use of multiple processors and cores. The input is broken up into 128 KB chunks with each compressed in  parallel. The  individual  check  value for each chunk is also calculated in parallel. The compressed data is written in order to the output, and a combined  check value is calculated from the individual check values.

       The  compressed  data format generated is in the gzip, zlib, or single-entry zip format using the deflate compression method.  The  compression  produces partial  raw deflate streams which are concatenated by a single write thread and wrapped with the appropriate header and trailer, where the trailer  contains the combined check value.



Example: tar -cf - DirectoryPath | pigz -9 -p 32 > TarFile.tar.gz

tar -I "pigz --best -r" -cf TarFile.tar.gz DirectoryPath

               pigz -9 -p 4 -kr DirectoryPath

          Explain: -9 Faster Compression -p = Processors you want use -k = Keep original files -r = Recursive

De-compress Faster: tar -I pigz -xf Archive.tar.gz -C /opt/



Install using TAR file

wget https://zlib.net/pigz/pigz-2.4.tar.gz

tar -xzvf pigz-2.4.tar.gz

cd pigz-2.4

make

make install

cp pigz unpigz /usr/bin/