feat: ultra mode, an optimal-parse LZ4 compressor - #226
Conversation
|
I think this is very similar to #216
Can you move this into a seperate PR? |
Yes, HC is but not ultra. Ultra uses a different algo to find even more matches (at the cost of compression time/memory). However, decompression is MUCH faster, 20-40% faster
Will do boss! |
A pure-Rust, optimal-parse block compressor behind the off-by-default `ultra`
feature. Output is a standard LZ4 block, decodable by any LZ4 implementation.
Engines are selected with `CompressionMode` (shared by the block and frame APIs):
- `Fast` — the existing greedy/lazy compressor (unchanged default).
- `Ultra` — a suffix array (SA-IS) + LCP-interval match finder feeding a backward
optimal-parse DP. Best ratio, decode-optimised stream.
- `Hc` — a hash-chain match finder with chain-swapping + a price-based optimal
parser; output matches `lz4hc -12`. Fast, low memory.
Block API: `compress_with_mode`, `compress_prepend_size_with_mode`,
`compress_into_with_mode` (positional dictionary, mirroring the `compress*`
family). Frame API: `FrameEncoder::set_compression_mode`.
no_std + alloc; the ultra module is `forbid(unsafe_code)` under `safe-encode`.
CI builds/tests the feature across the matrix. Covered by round-trip,
ratio-parity, and byte-identical-to-`lz4hc -12` tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4acd00e to
e5e10d9
Compare
2 participants
A pure-Rust, optimal-parse block compressor behind the off-by-default
ultrafeature, plus a crate-wide decoder speedup. Output is a standard LZ4 block, decodable by any LZ4 implementation. Ported from https://github.com/emmanuel-marty/lz4ultraEngines are selected with
CompressionMode(shared by the block and frame APIs):Fast, the existing greedy/lazy compressor (unchanged default).Ultra, a suffix array (SA-IS) + LCP-interval match finder feeding a backward optimal-parse DP. Best ratio, decode-optimised stream.HC, a hash-chain match finder with chain-swapping + a price-based optimal parser; output matcheslz4hc -12. Fast, low memory.Block API:
compress_with_mode,compress_prepend_size_with_mode,compress_into_with_mode(positional dictionary, mirroring thecompress*family). Frame API:FrameEncoder::set_compression_mode.Decoder: overlapping (period < length) copies now grow by doubling the valid prefix with memmove/memcpy instead of byte-by-byte, O(log n) copies — ~10-20x on run/periodic data, text unchanged. Validated under miri and against the C decoder.
no_std + alloc; the ultra module is
forbid(unsafe_code)undersafe-encode. CI builds/tests the feature across the matrix and adds a miri job for the unsafe overlap copy. Covered by round-trip, ratio-parity, and byte-identical-to-lz4hc -12tests.