Zstd: Update bundled library to version 1.5.7

[alex-moch]

This pull request updates the bundled Zstandard (zstd) library to version 1.5.7 and adds the necessary OpenZFS integration and build system changes required to make zstd 1.5.7 compile and function correctly.

---

Motivation and context

This pull request is motivated by build failures observed when building with GCC using -Og, as reported in:

• Can't compile openzfs with "-Og" - inline failed within the ZSTD module #14755
• Gentoo's Bugzilla – Bug 967762 - sys-fs/zfs-2.4.0

The version of Zstandard currently shipped with OpenZFS ignores -DZSTD_NO_INLINE due to how inlining is enforced around xxhash.c in the upstream sources.

Fixing this issue for the currently bundled Zstandard version would require carrying local patches to upstream code.

Zstandard 1.5.7, among other changes, removes the standalone xxhash.c source and adjusts inlining behavior such that builds with -Og -g -DZSTD_NO_INLINE succeed. As a first step, this pull request updates the bundled Zstandard version in OpenZFS to 1.5.7 and performs the required integration changes. This is considered the cleanest approach for the master branch.

Updating the Zstandard version in released OpenZFS stable branches is likely not an option. The approach taken here is therefore to fix master first and then evaluate whether a targeted workaround is feasible for OpenZFS 2.4 and possibly earlier releases.

Description

The latest upstream Zstandard 1.5.7 release has been imported into module/zstd. Where required, corresponding changes to the OpenZFS build system and integration code have been applied.

This pull request intentionally deviates from the “one commit per pull request” model:

• The first commit is a direct import of the Zstandard 1.5.7 sources without functional modifications.
• The second commit contains the necessary OpenZFS integration and build system adjustments.

This separation is intended to simplify review and improve traceability of upstream versus OpenZFS-specific changes.

A third commit will add SPDX license identifiers to newly imported files to match existing OpenZFS conventions.

The upstream Zstandard sources are imported without reformatting or style adjustments, which intentionally deviates from OpenZFS coding style guidelines to keep the vendor update easy to track and review.

How has this been tested?

• OpenZFS CI Chain
• OpenZFS Alpine Linux 3.23 Runner
• Build testing on Gentoo ~arm64 with musl libc
  • From the source tree and through Portage
• Build testing on Gentoo ~amd64 with --enable-linux-builtin

Todos before marking this ready for review

• Add SPDX license identifiers to newly imported files.
• Remove temporary allocator bypass and implement proper fix.
• Test against the new Alpine Linux CI runner.
• Perform additional compatibility checks with older ZFS releases.
• Check kernel builds with --enable-linux-builtin.

---

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[alex-moch]

Memory is now unpoisoned when handing buffers out to Zstd and poisoned again when buffers are returned to the pool. Testing on the new Alpine runner looks good. I also restored SPDX license identifiers and fixed minor style issues in zstd-in.c.

Compatibility with older ZFS releases and --enable-linux-builtin still need to be verified.

[alex-moch]

I successfully build-tested --enable-linux-builtin on a custom kernel built with LLVM/Clang using Full LTO. I also created a new pool with -O compression=zstd, wrote a random file, verified its hash, and successfully read it back on 2.4.0 without issues.

In the latest push, tracing is disabled for kernel builds, as it caused issues on AArch64.

This is now ready for review.

[robn]

I don't wanna be that guy, but...

The traditional wisdom is that we can't upgrade the compression unless it outputs exactly the same compressed bytes for a given input as the previous version. This is because we calculate checksums over the compressed data, so to change the compression output would mean that the same data post-compression would not produce the same checksum. The more specific claim is something like: ZFS assumes that Compress(Decompress(Compress(X))) == Compress(X). More info: https://www.youtube.com/watch?v=_5JeyCYV5nE

So the question here is does the newer zstd version guarantee the output is the same as our current version? The broader question, I suppose, is to what extent ZFS still requires that this is still the case. I don't think I know that off the top of my head (obviously I understand that it'll produce a different checksum, and will look different after encryption, but its not quite jumping out at me what the knock-on effects of that are).

[robn]

There's also sometimes performance changes between different versions and I know in the past newer versions have been slower for whatever reason. I don't personally care as much about that as correctness (well, within reason) but it still is worth being aware of.

[alex-moch]

After posting this pull request, I realized there was prior discussion about compressor updates.

This topic was discussed in the January 2022 Leadership Meeting. Some relevant quotes from that discussion are:

1. all three problematic cases are very minor. We can just replace-upgrade decompressor and compressor.

2. LZ4 (Rich) and ZSTD are two concrete upgrades that are planned

3. For ZSTD, which changes more frequently, we might not want to maintain multiple compressor versions.

With that in mind, I’d like to understand what the recommended path forward is for upgrading Zstd. As far as I can tell, the vendored Zstd version is still at 1.4.5 from May 2020.

[gmelikov]

@robn I've tackled those questions in #9416 , especially with encryption. That PR changed compression threshold logic, and was merged in 2.3.0. Since then I don't know about any problems after such change (but it essentially expects that if block was compressed before - it will compress after too, so we MAY have problems if compressor update make worse results).

FWIW, we should not assume Compress(Decompress(Compress(X))) == Compress(X), and if we have such places - we need to rework it. The only case where it's not now (AFAIK) - is #9416 (comment) "recompress encrypted block on L2ARC write", and we already may have problems with it with QAT.

[gmelikov]

And, in "recompress on l2arc write" case looks like we already just skip such non-"recompressable idempotently" block https://github.com/gmelikov/zfs/blob/master/module/zfs/arc.c#L9273-L9280

[behlendorf]

The more specific claim is something like: ZFS assumes that Compress(Decompress(Compress(X))) == Compress(X).

Exactly, it's this assumption which can cause some minor problems in certain circumstances. The possible side effects really boil down to unexpected additional capacity usage and some optimizations being skipped which could impact performance. Let me try and summarize the state of things as of 2.4.0. @gmelikov @robn please chime in if I missed anything!

1. Uncompressed ARC + Compressed On-Disk + L2ARC

As @gmelikov mentioned this case has largely been addressed as of the 2.3.0 release. As long as the updated compressor code compresses blocks to the same size (or smaller) they will be written to the L2ARC. If re-compressed blocks are either 1) larger then before, or 2) encryption is enabled, and the blocks compress differently, then they aren't written to the L2ARC.

Mind you this is still only a problem when the compressed ARC is disabled. It's enabled by default, so I wouldn't expect many users to encounter this. Even if they do it's a minor performance penalty until those files get rewritten with the new compressor. I think we've done what we can to address this one.

2. NOP Writes

When a cryptographically secure checksum is used, such as SHA256, ZFS can detect if the new data being written is identical to the data on-disk for that block. If it is, the write can be skipped which avoids an unnecessary I/O and can save space in snapshots. If the data compresses differently with the new compressor the checksums won't match and this optimization will be skipped.

My sense is NOP writes are uncommon enough that this really wouldn't be a problem in practice.

3. Deduplication

Blocks compressed with the new compressor may not deduplicate with blocks compressed using the old compressor. For users who are expecting a high-degree of deduplication this could be surprising after an update. We do now that we have richer options for managing dedup (dedup_table_quota property, zpool ddtprune, and zfs rewrite) so any potential problems would be more manageable.

Given the above I don't think there are any major deal breakers. My main concern is the same as Rob's, we need to be able to verify (within reason) that no correctness issues were introduced with the updated version. One way to do that would be to support multiple versions of the compressor+decompressor and allow the version to be selected at run time. We already do something similar for the SIMD optimized checksum implementations to ensure they're all compatible. Since the zstd version is stored within the block header this would also potentially allow us to gracefully re-compress old blocks as needed. I'd be interested to know if others think it's worthwhile, it would definitely be a bit more work to maintain.

[jumbi77]

With RAM getting more expensive and also CPU/RAM getting faster and faster, would it make sense to finally drop uncrompressed arc (and l2arc) in next zfs version?

[gmelikov]

Looks like we have (a little bit patched) zstd v1.4.5 from May 22, 2020.

I've skimmed through their changelogs https://github.com/facebook/zstd/releases , looks like there is a ton of performance enhancements (especially for small blocks <=128K, as we like) and some pretty critical fixes there. v1.5.7 was released in Feb 20, 2025, so it's pretty mature to use safely.

I'd propose to just update our zstd as is without supporting multiple versions, BUT it would be best to do so only in major releases. It would be great to highlight notes about encrypted+uncompressed_arc+l2arc/NOP_write/dedup in our changelogs. I see these things as a necessity.

[alex-moch]

Thanks everyone for the input!

@behlendorf: I don't quite understand what you mean by "correctness" here. Do you refer to the idempotency property that @robn referred to and its potential effects in OpenZFS? To me, correctness with regards to compressor versioning would be Decompress_new(Compress_old(X)) == X, and this should hold.

Supporting multiple versioned Zstd implementations (or possibly other compressors) wouldn’t scale well across releases. Also, the January 2022 leadership meeting notes mention that

For ZSTD, which changes more frequently, we might not want to maintain multiple compressor versions.

In any case, I think a strategy regarding future compressor updates is needed. My suggestion would be: right after a .0 release is cut (e.g. now after 2.4.0), update master to the latest stable Zstd. Then, until the next release freeze, also pull in any Zstd patch releases that appear. For example, if v1.5.8 drops before the freeze for OpenZFS 2.5.0, I think it makes sense to upgrade.

@gmelikov: To clarify my original pull request message: the intention was to fix -Og builds in master by updating to Zstd v1.5.7, and address -Og in the stable branches with careful guards/workarounds around the existing Zstd v1.4.5 code. I fully agree on not porting this back to the stable branches.

Finally, I would like to drop the last commit with whitespace-only changes to zstd-in.c (vendor file), and instead add it to the checkstyle ignore list as part of the integration commit.

[behlendorf]

It sounds like we have general agreement on how to handle updates to zstd going forward. In short:

1. zstd may be updated (as needed) after a new .0 release is tagged
2. critical patches may be applied up until the next release freeze (potentially ever to a newer version)
3. the zstd version will not be upgraded within a major release
   (e.g. OpenZFS 2.4.x => zstd v1.4.5, OpenZFS 2.5.x -> zstd v1.5.7, etc)
4. multiple zstd versions are not supported concurrently within a release
5. release notes should highlight zstd updates and what to expect

This all sounds reasonable. @alex-moch my correctness concern above was was primarily around making sure updated versions of zstd are mature and stable before we pull them in. For me that means getting zstd updated early in our release cycle so we have time to develop confidence in the updated code. Well that and the interoperability testing as you've already done.

I would like to drop the last commit with whitespace-only changes to zstd-in.c (vendor file), and instead add it to the checkstyle ignore list as part of the integration commit.

@alex-moch if if you can update the PR to make these changes we should be all set to get this merged. Could you also update the module/zstd/README.md file with a version of our policy for when zstd can be updated. Thanks again for updating this!

[robn]

Late to the party here, but some comments. tl;dr I'm onboard with the upgrade plan/policy, great work! Lets go!

I did feel like some past PRs attempting compression upgrades showed substantially worse performance. Unfortunately I can't find an obvious PR to point to; the discussion of compression in the issue tracker is... plentiful. I might be thinking of an LZ4 upgrade though; anecdotally it seems like people general have gotta broadly better performance out of zstd 1.5.x over 1.4.x. It would be nice, I suppose, to have something that could more easily compare algorithm performance before/after change, but we're not there (yet) and see no reason to pause right now. With a long lead time we should be able to shake it out.

On running multiple versions, it would be way too difficult to do with the way things are currently put together, but I will note that I'm still working towards having a single mechanism for swapping algorithm implementations across ZFS (same idea as we have for eg AES, GCM, fletcher4, etc). I'm hoping to have more time for that kind of refactoring work this year, so hopefully the next time we tackle an upgrade we'll be able to just drop the new version into a different dir and compile them both in (or not, as we like).

Mind you this is still only a problem when the compressed ARC is disabled.

We've been dancing around zfs_compressed_arc_enabled=0 for a few years now, and I vaguely recall someone saying they weren't even sure if it still worked. I'm in no hurry, and obviously not for this PR, but just to put the idea your heads: is this the year we remove the option completely?

We do now that we have richer options for managing dedup (dedup_table_quota property, zpool ddtprune, and zfs rewrite) so any potential problems would be more manageable.

I feel like there's possibly some opportunities here too. No actual ideas, just a sense that there could be things worth exploring. eg from one angle, physical rewrite is a way of understanding that two apparently-different blocks are actually the "same" in a different context. Not directly useful here perhaps, but now that the idea is introduced there may be more we can do with it.

Anyway, none of that is directly related. I think that this PR is a good thing for us to do: we don't think there's any correctness issues, and its probably a good time in our history to be revisiting it and experiencing any difficulties first-hand (or not) so we can really be sure we know what we're dealing with. I'd love for (say) yearly math upgrades to be routine! First one is always scary :)

[alex-moch]

I just pushed the update:

• Added zstd-in.c to the checkstyle ignore list in the integration commit.
• Updated Zstd’s README.md with the new update policy plus a few minor cleanups.

[alex-moch]

One more thing that just occurred to me and might be worth explicitly adding to the update policy:

For vendor or library drops, the vendor import commit should be a clean, unmodified upstream import. Any OpenZFS-specific integration or local adjustments should go into follow-up commits.

This makes the import easy to verify against upstream sources during review and later by anyone auditing OpenZFS’ git history. It also reduces supply chain risk by ensuring we have clearly auditable provenance for third-party code.

[behlendorf]

For vendor or library drops, the vendor import commit should be a clean, unmodified upstream import. Any OpenZFS-specific integration or local adjustments should go into follow-up commits.

Good thought, let's add it. Then this PR should be ready to go.

[alex-moch]

Just added it. I went with stronger wording: must instead of should.

[jumbi77]

With RAM getting more expensive and also CPU/RAM getting faster and faster, would it make sense to finally drop uncrompressed arc (and l2arc) in next zfs version?

@behlendorf Since @robn opened this topic too in this answer above, whats your opinion? Is 2026 the time to bring back this topic and finally drop uncompressed arc?