Skip to content
This repository was archived by the owner on Mar 24, 2026. It is now read-only.
This repository was archived by the owner on Mar 24, 2026. It is now read-only.

Analysis fails when container image has uncompressed layers #1080

Open
#1082
Open
Analysis fails when container image has uncompressed layers#1080
#1082
Labels
bugtriaged

Description

@tsaarni
tsaarni
opened on May 27, 2021

Is this a BUG REPORT or a FEATURE REQUEST? (choose one): BUG REPORT

Version of Anchore Engine and Anchore CLI if applicable:
Anchore Engine v0.9.4
Anchore CLI v0.9.1

What happened:
Analysis fails for images that contain uncompressed layers.

That is, layers with media type application/vnd.docker.image.rootfs.diff.tar instead of application/vnd.docker.image.rootfs.diff.tar.gzip.

What did you expect to happen:
Analysis should have succeeded.

Any relevant log output from analyzer:

Analyzer raises following exception and analysis fails with status analysis_failed.

[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.skopeo_wrapper/download_image()] [DEBUG] command succeeded: cmd=/bin/sh -c skopeo   copy --remove-signatures --src-tls-verify=false   docker://docker.io/tsaarni/uncompressed-layer-demo@sha256:cb7e1611d9ff0e9090a13c5bdf343bb0cde9a8e30e60c9892e47c83a4ac97634 oci:/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw:image stdout=b'Copying blob sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116\nCopying config sha256:cdce9ebeb6e8364afeac430fe7a886ca89a90a5139bc3b6f40b5dbd0cf66391c\nWriting manifest to image destination\nStoring signatures\n' stderr=b''
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/squash()] [DEBUG] Layers to process: ['sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116']
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/squash()] [DEBUG] Pass 1: generating layer file timeline
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/squash()] [DEBUG] processing layer sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 - None
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/squash()] [DEBUG] Pass 3: closing layer tarfiles
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/delete_staging_dirs()] [DEBUG] keep_image_analysis_tmpfiles is enabled - leaving analysis tmpdir in place {'unpackdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89', 'copydir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw', 'rootfs': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/rootfs', 'outputdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/output', 'cachedir': None}
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/delete_staging_dirs()] [DEBUG] keep_image_analysis_tmpfiles is enabled - leaving analysis tmpdir in place {'unpackdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89', 'copydir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw', 'rootfs': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/rootfs', 'outputdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/output', 'cachedir': None}
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/delete_staging_dirs()] [DEBUG] keep_image_analysis_tmpfiles is enabled - leaving analysis tmpdir in place {'unpackdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89', 'copydir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw', 'rootfs': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/rootfs', 'outputdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/output', 'cachedir': None}
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.clients.localanchore_standalone/delete_staging_dirs()] [DEBUG] keep_image_analysis_tmpfiles is enabled - leaving analysis tmpdir in place {'unpackdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89', 'copydir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw', 'rootfs': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/rootfs', 'outputdir': '/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/output', 'cachedir': None}
[service:worker] 2021-05-27 17:04:26+0000 [-] Traceback (most recent call last):
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 1092, in analyze_image
[service:worker] 2021-05-27 17:04:26+0000 [-]     imageSize = unpack(staging_dirs, layers)
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 876, in unpack
[service:worker] 2021-05-27 17:04:26+0000 [-]     squashtar, imageSize = squash(unpackdir, cachedir, layers)
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 330, in squash
[service:worker] 2021-05-27 17:04:26+0000 [-]     tarfiles[l] = tarfile.open(layertar, mode="r", format=tarfile.PAX_FORMAT)
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/lib64/python3.8/tarfile.py", line 1590, in open
[service:worker] 2021-05-27 17:04:26+0000 [-]     raise ValueError("nothing to open")
[service:worker] 2021-05-27 17:04:26+0000 [-] ValueError: nothing to open
[service:worker] 2021-05-27 17:04:26+0000 [-]
[service:worker] 2021-05-27 17:04:26+0000 [-] During handling of the above exception, another exception occurred:
[service:worker] 2021-05-27 17:04:26+0000 [-]
[service:worker] 2021-05-27 17:04:26+0000 [-] Traceback (most recent call last):
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/analyzer/analysis.py", line 333, in process_analyzer_job
[service:worker] 2021-05-27 17:04:26+0000 [-]     image_data = perform_analyze(
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/analyzer/analysis.py", line 193, in perform_analyze
[service:worker] 2021-05-27 17:04:26+0000 [-]     analyzed_image_report, manifest_raw = localanchore_standalone.analyze_image(
[service:worker] 2021-05-27 17:04:26+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/clients/localanchore_standalone.py", line 1125, in analyze_image
[service:worker] 2021-05-27 17:04:26+0000 [-]     raise AnalysisError(
[service:worker] 2021-05-27 17:04:26+0000 [-] anchore_engine.clients.localanchore_standalone.AnalysisError: failed to download, unpack, analyze, and generate image export (docker.io/tsaarni/uncompressed-layer-demo@sha256:cb7e1611d9ff0e9090a13c5bdf343bb0cde9a8e30e60c9892e47c83a4ac97634) - exception: nothing to open
[service:worker] 2021-05-27 17:04:26+0000 [-] [Thread-75] [anchore_engine.services.analyzer.analysis/process_analyzer_job()] [ERROR] problem analyzing image - exception: failed to download, unpack, analyze, and generate image export (docker.io/tsaarni/uncompressed-layer-demo@sha256:cb7e1611d9ff0e9090a13c5bdf343bb0cde9a8e30e60c9892e47c83a4ac97634) - exception: nothing to open

What docker images are you using:
docker.io/tsaarni/uncompressed-layer-demo:latest

The image is alpine:latest but it has been specifically crafted and uploaded in a way that it contains a layer that is uncompressed.

How to reproduce the issue:

$ docker run -e ANCHORE_CLI_USER=$ANCHORE_CLI_USER -e ANCHORE_CLI_PASS=$ANCHORE_CLI_PASS -e ANCHORE_CLI_URL=$ANCHORE_CLI_URL --network host -it anchore/engine-cli:v0.9.1 anchore-cli image add docker.io/tsaarni/uncompressed-layer-demo:latest

...

$ docker run -e ANCHORE_CLI_USER=$ANCHORE_CLI_USER -e ANCHORE_CLI_PASS=$ANCHORE_CLI_PASS -e ANCHORE_CLI_URL=$ANCHORE_CLI_URL --network host -it anchore/engine-cli:v0.9.1 anchore-cli image get docker.io/tsaarni/uncompressed-layer-demo:latest
Image Digest: sha256:cb7e1611d9ff0e9090a13c5bdf343bb0cde9a8e30e60c9892e47c83a4ac97634
Parent Digest: sha256:cb7e1611d9ff0e9090a13c5bdf343bb0cde9a8e30e60c9892e47c83a4ac97634
Analysis Status: analysis_failed
Image Type: docker
Analyzed At: None
Image ID: 6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec
Dockerfile Mode: None
Distro: None
Distro Version: None
Size: None
Architecture: None
Layer Count: None

Full Tag: docker.io/tsaarni/uncompressed-layer-demo:latest
Tag Detected At: 2021-05-27T17:04:19Z

Anything else we need to know:
The root cause is following:

(1) When image is added Anchore first runs skopeo inspect to download manifest

$ skopeo inspect --raw docker://tsaarni/uncompressed-layer-demo:latest | jq .{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1472,
    "digest": "sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 5879808,
      "digest": "sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116"
    }
  ]
}

Note that the layer sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 is uncompressed.

(2) Next, it uses skopeo copy to download image and store it as OCI directory

$ skopeo copy docker://tsaarni/uncompressed-layer-demo:latest oci:/analysis_scratch/...

(3) Skopeo will automatically compresses the layer on the fly during copy operation

This is the new manifest in the OCI directory

$ cat /analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256/f465bb97f5825dd1aae65c816886574984edcd22e908897648fcd26d819bd9a2 | jq .
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:cdce9ebeb6e8364afeac430fe7a886ca89a90a5139bc3b6f40b5dbd0cf66391c",
    "size": 585
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f12e03b5ef6655f2e0c3cc99f5f940e9be541a3b04703a27a1baf72325412af6",
      "size": 2902398
    }
  ]
}

Note that the layer sha256:f12e03b5ef6655f2e0c3cc99f5f940e9be541a3b04703a27a1baf72325412af6 is now a compressed version of the layer in the original image and that the digest changed due to compression.

(4) Analyzer will still use the manifest from skopeo inspect where the layer digest was for the uncompressed layer.

Therefore analyzer tries to open file /analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256/b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 (digest from original manifest) which does not exist. Exception is raised.

It should have opened /analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256/f12e03b5ef6655f2e0c3cc99f5f940e9be541a3b04703a27a1baf72325412af6 (digest after skopeo copy did the compression)

Here is contents of the work directory:

$ find /analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/index.json
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/oci-layout
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/parent_manifest.json
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/manifest.json
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256/f12e03b5ef6655f2e0c3cc99f5f940e9be541a3b04703a27a1baf72325412af6
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256/cdce9ebeb6e8364afeac430fe7a886ca89a90a5139bc3b6f40b5dbd0cf66391c
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/raw/blobs/sha256/f465bb97f5825dd1aae65c816886574984edcd22e908897648fcd26d819bd9a2
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/rootfs
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/output
/analysis_scratch/cf296952-8c5f-4e5b-b807-35dc652edc89/docker_history.json

Here manifest.json and parent_manifest.json are the original manifests from skopeo inspect.

Note that media type application/vnd.docker.image.rootfs.diff.tar was never explicitly mentioned in the image manifest v2 spec but it is supported by runtimes and the image works just fine:

$ docker run --rm -it tsaarni/uncompressed-layer-demo:latest ash
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.13.5
PRETTY_NAME="Alpine Linux v3.13"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugtriaged

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions