Add support for erofs and squashfs layer media types

[rchincha]

OCI artifacts support has landed in various OCI specs v1.1.0 which allows for arbitrary artifact types, small and large.

Large artifacts (even existing container images) pose a particular challenge that:

1. it takes too long to download
2. it takes too long to unpack

This PR begins to address 2. above.

squashfs (an older read-only) and erofs (newer) filesystems are already compressed full filesystems, and overlayfs compatible.

[sudo-bmitch]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

[rchincha]

#1190

[rchincha]

opencontainers/distribution-spec#537
to be combined with ^

[rchincha]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

This PR just represents possibilities. Needs a discussion if OCI wants to "adopt" this a recommended option. Almost all known solutions end up being a clever fuse or remotefs wrapper around a layer - so why not make the layer a filesystem itself! No further machinery needed.

[hsiangkao]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

This PR just represents possibilities. Needs a discussion if OCI wants to "adopt" this a recommended option. Almost all known solutions end up being a clever fuse or remotefs wrapper around a layer - so why not make the layer a filesystem itself! No further machinery needed.

JFYI, EROFS already has a IANA-registered media type "vnd.erofs" if some format is referred.

[rchincha]

Could also be application/vnd.oci.image.layer.erofs and application/vnd.oci.image.layer.squashfs
just so that we indicate this is OCI blessed.

Open question, whiteout handling

[syed]

The problems I see with large binary files (eg AI models) is that they don't tend well to compression. I would like to see if we can improve the distribution spec to allow parallel chunked upload/download. Right now we can only upload/download sequentially

[sudo-bmitch]

Right now we can only upload/download sequentially

I get the impression that most registries allow this already and we're just formalizing it.

[rchincha]

https://stackerbuild.io/
https://github.com/project-stacker/stacker/releases/tag/v1.1.0-rc1
v1.1.0-rc1 can now natively build OCI images with erofs layers+dmverity

[rchincha]

https://stackerbuild.io/ https://github.com/project-stacker/stacker/releases/tag/v1.1.0-rc1 v1.1.0-rc1 can now natively build OCI images with erofs layers+dmverity

In this context ^
containerd/containerd#11163

[sajayantony]

Honestly not in favor or non-oci types here.

[rchincha]

Honestly not in favor or non-oci types here.

Just to clarify further - the image is still OCI; this is about the layers. If they are complete filesystems (squashfs, erofs), then they can just be mounted directly. This is an alternative approach to what the recently added containerd erofs snapshotter does today - convert tgz into erofs and cache.

The only implication wrt this PR is that if squashfs, erofs are made "official" layer types (not a MUST btw, image still OCI), then the negotiation with containerd community becomes easier. Else maybe they will accept a tunable/config to allow such types.

If squashfs,erofs+verity are produced and signed at source, then integrity is maintained end-to-end and becomes a much simpler deployment model - no tgz mutation gimmickry.

[hsiangkao]

Honestly not in favor or non-oci types here.

Just to clarify further - the image is still OCI; this is about the layers. If they are complete filesystems (squashfs, erofs), then they can just be mounted directly. This is an alternative approach to what the recently added containerd erofs snapshotter does today - convert tgz into erofs and cache.

The only implication wrt this PR is that if squashfs, erofs are made "official" layer types (not a MUST btw, image still OCI), then the negotiation with containerd community becomes easier. Else maybe they will accept a tunable/config to allow such types.

If squashfs,erofs+verity are produced and signed at source, then integrity is maintained end-to-end and becomes a much simpler deployment model - no tgz mutation gimmickry.

@rchincha One thing I might need to mention here is prefix+compression format, currently
application/vnd.oci.image.layer.v1.tar+gzip
application/vnd.oci.image.layer.v1.tar+zstd
are reasonable because they are wraped with real gzip and zstd formats and can be decoded with gzip and zstd tools, also see https://www.rfc-editor.org/rfc/rfc6839#section-3.6
But I think it's incorrect to use if the case is "a filesystem contains lz4, lzma, deflate or zstd compressed data", it cannot be directly decoded with any tool above. IOWs, only uncompressed filesystem + gzip/zstd wrapper can be marked with application/vnd.xxx+gzip/zstd, otherwise it should be just application/vnd.xxx without any suffix.
Also EROFS supports multiple algorithms in one filesystem/image, so such suffix is meaningless to me.

[hsiangkao]

...

@rchincha One thing I might need to mention here is prefix+compression format, currently application/vnd.oci.image.layer.v1.tar+gzip application/vnd.oci.image.layer.v1.tar+zstd are reasonable because they are wraped with real gzip and zstd formats and can be decoded with gzip and zstd tools, also see https://www.rfc-editor.org/rfc/rfc6839#section-3.6 But I think it's incorrect to use if the case is "a filesystem contains lz4, lzma, deflate or zstd compressed data", it cannot be directly decoded with any tool above. IOWs, only uncompressed filesystem + gzip/zstd wrapper can be marked with application/vnd.xxx+gzip/zstd, otherwise it should be just application/vnd.xxx without any suffix. Also EROFS supports multiple algorithms in one filesystem/image, so such suffix is meaningless to me.

Also I try to enable EROFS metadata compression in the kernel 6.16 cycle: Compressed metadata is just placed in a special on-disk inode, so the metadata compress algorithm can be different from data algorithms, which could be useful to maximize runtime performance for metadata (lz4, zstd, or uncompressed) and leave high compression ratios for data (zstd, lzma, deflate, etc.)

[rchincha]

Pushed two test multi-layer OCI images with erofs layers (with and without dm-verity)

https://hub.docker.com/repository/docker/ramchi879/test-erofs-verity/tags
https://hub.docker.com/repository/docker/ramchi879/test-erofs-no-verity/tags

To download them,
skopeo copy --dest-oci-accept-uncompressed-layers docker://ramchi879/test-erofs-verity:c-erofs oci:erofs:test
skopeo copy --dest-oci-accept-uncompressed-layers docker://ramchi879/test-erofs-no-verity:c-erofs oci:erofs:test

[sudo-bmitch]

I'm a NACK on this:

• Defining the variables in specs-go is not adding it to the list of supported layer media types in the spec. I don't believe this does anything for the underlying goal.
• We defined the OCI layer media types because they have a content defined in layer.md and are more than a tar+gzip (if they were, we could have delegated to IANA).
• These media types are / should be registered with IANA and have their contents specified there.
• Whether implementations must/should/may support these media types would be specified in manifest.md.

[rchincha]

I'm a NACK on this:

• Defining the variables in specs-go is not adding it to the list of supported layer media types in the spec. I don't believe this does anything for the underlying goal.

Of course not. This was only meant to start a discussion about possibilities.

• We defined the OCI layer media types because they have a content defined in layer.md and are more than a tar+gzip (if they were, we could have delegated to IANA).
• These media types are / should be registered with IANA and have their contents specified there.
• Whether implementations must/should/may support these media types would be specified in manifest.md.

Wondering if there a circular argument here?

Firstly, only the following appears to be registered with IANA.
application/vnd.oci.image.manifest.v1+json

So to add a new layer type, a PR modifying layer.md? But to accept that it must first be registered with IANA?
https://www.rfc-editor.org/rfc/rfc6838.html#section-3.2

But that can only be done by OCI (since vnd.oci)? which must first be ready to accept the PR to layer.md?

[rchincha]

https://github.com/awslabs/soci-snapshotter/blob/main/docs/index.md

"mediaType": "application/vnd.amazon.soci.index.v1+json" (image "config", but still)
"mediaType": "application/octet-stream" (layer)

However, only the following registered with IANA and above not in layer.md.
application/vnd.amazon.mobi8-ebook

Perhaps this ^ pattern?

[sudo-bmitch]

Wondering if there a circular argument here?

Firstly, only the following appears to be registered with IANA. application/vnd.oci.image.manifest.v1+json

So to add a new layer type, a PR modifying layer.md? But to accept that it must first be registered with IANA? https://www.rfc-editor.org/rfc/rfc6838.html#section-3.2

But that can only be done by OCI (since vnd.oci)? which must first be ready to accept the PR to layer.md?

Lets start with a question, what is OCI specific about the new OCI media types you'd like to define?

Currently with the tar layers, there is ordering, white out files, limitations on hard links, and other requirements that apply when extracting multiple tar files as layer diffs, so we aren't using application/gzip or application/x-tar (perhaps not the best example because why the heck hasn't IANA defined an application/tar media type, but I digress). So lets start with what would be different in the OCI implementation of erofs and squashfs from the upstream standards?

If they aren't different, then we defer to upstream specs. If they are different, then to start defining that we'd need to go from the other direction, not from adding a new Go variable, but creating a working group, getting projects to buy in with some implementations to work out the various edge cases and compatibility issues, and then submitting the new OCI media type as an entry in something like layers.md.

If we want to document that implementations support a specific set of externally defined layer media types, we can work on changes to manifest.md to list those, and whether they are must/should/may for implementations to support.

[rchincha]

https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar
https://groups.google.com/a/opencontainers.org/g/dev/c/Zk3yf45HIdA?pli=1

... without any massive overhaul