performance: what can dist-spec do to improve downloads of large images/layers?

[rchincha]

Things we already allow/do:

1. Parallel download of layers

Things we can likely improve:

1. For large layers, range-based downloads - download sections of a large file using Range Header and stitch them back together?

Things known to the community:

1. https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md
   ^ fuse based filesystem solutions - copy individual things when referenced

[samuelkarp]

For streaming/lazy loading:

• GKE image streaming
• https://github.com/awslabs/soci-snapshotter

Multipart layer downloads with range requests:

• Multipart layer fetch containerd/containerd#10177
• https://github.com/awslabs/amazon-ecr-containerd-resolver#parallel-downloads

Direct mounts of compressed tars (saves on extraction time):

• [Feat] erofs snapshotter and differ containerd/containerd#9362 (comment)
• https://github.com/containers/composefs

[sudo-bmitch]

What changes are needed in distribution-spec to support this? Is a pointer to the HTTP specs documenting range requests enough?

[rchincha]

opencontainers/image-spec#1190

[rchincha]

What changes are needed in distribution-spec to support this? Is a pointer to the HTTP specs documenting range requests enough?

IMO, what server/client side optimizations can be enabled by dist-spec changes? demonstrably?

Just like conformance, we should write benchmark code for this.

[sudo-bmitch]

Just like conformance, we should write benchmark code for this.

Is that an OCI requirement, or something implementations should be doing?

[rchincha]

Just like conformance, we should write benchmark code for this.

Is that an OCI requirement, or something implementations should be doing?

Not an OCI requirement - our conformance suite should already ensure conformant registries can handle range-based blob pulls.

#537 (comment)
^ will be interesting to see data from various registries. If viable, clients should move to this model advised by blob size.

[rchincha]

#546