How to get the maximum compression?

[demhademha]

How would one get the maximum directory compression regardless of time taken
Regards

[pete4abw]

Obviously, using -z for ZPAQ would yield the best compression. And -p1 would compress the largest possible block size. Many factors, however, will impact your results. System ram, Processor cores, Free ram, whether you're using nvram or a hard disk, and of course the size of the file being compressed. Best time results are when multi-threading is used. Best compression is by using 1 thread. YMMV. Here is some output for you to chew on.

The sample file in this case is 1,434,736,640 bytes.
System Ram: 16GB
Processor: Intel I7 1065G7 4 cores, 8 threads running around 3Ghz
Kernel: 5.16.11

If the directory is already tar'ed, then:
lrzip-next -p1 -zL9 file.tar
If the directory is NOT tar'ed, then:
tar -cf - directory | lrzip-next -p1 -zL9 -o directory.tar.lrz

Here is an example:

$ lrzip-next -vvzp1 -L9 -o linux-5.x.tar.p1zL9.tar.lrz linux-5.x.tar
linux-5.x.tar - Compression Ratio: 12.919. Average Compression Speed:  0.748MB/s.
Total time: 00:30:28.86

and

$ tar -cf - linux-5.x | lrzip-next -vvzp1 -L9 -o linux-5.x.tar.p1zL9.tar.lrz
Compression Ratio: 12.919. Average Compression Speed:  0.754MB/s.
Total time: 00:30:13.71

A very slight difference in compression size between tar and not. However, this run using lzma shows much faster performance, but poorer compression.

$ lrzip-next -vvL9 -p1 -o linux-5.x.tar.p1L9.lrz linux-5.x.tar
linux-5.x.tar - Compression Ratio: 9.771. Average Compression Speed:  2.698MB/s.
Total time: 00:08:26.71

and Standard multi-threaded lzma

$ lrzip-next -vvL9 -o linux-5.x.tar.L9.lrz linux-5.x.tar
linux-5.x.tar - Compression Ratio: 9.733. Average Compression Speed:  7.558MB/s.
Total time: 00:03:00.31

Much lower time and very slight degradation in performance.

lrz size	file name	time	desceription
146,835,899	linux-5.x.tar.p1L9.lrz	08:26	# lzma p1L9
111,059,904	linux-5.x.tar.p1zL9.lrz	30:28	# zpaq p1L9
111,059,904	linux-5.x.tar.p1L9ft.lrz	30:13	# zpaq p1L9 through tar
147,404,759	linux-5.x.tar.L9.lrz	03:00	# normal lzma with default threads and memory allocation

The p1 will create the largest possible block to compress instead of splitting a chunk into multiple blocks. Depending on your system memory, you may also try overriding the ZPAQ block size with --zpaqbs=11. With 16GB the largest block size possible is 9. See libzpaq.h for details on block size computation.

Be warned though, it takes a long time with -L9. And the benefit gained with ZPAQ for files, while greater than lzma, is not worth it for smaller files and directories.

Benchmarking is an ongoing problem because what's good for one is not good for all. There are so many ways to slice up how "good" a compression method is. You must find your own "best" way.

[demhademha]

Is there a link to the Linux tarball used for this test

[pete4abw]

kernel.org

[demhademha]

If I may know, how does the unlimited flag affect the compression

[pete4abw]

I've never used it or tested it. IMHO it's totally unnecessary. There's some documentation on it in the original README. With today's modern systems and large reservoirs of RAM, it's not useful. With files larger than available RAM, -U may offer some benefit but at a huge cost of time. -- Peter Hyman Sent from mobile. Please excuse brevity