Deploy in-cluster OCI registry (zot) for artifact transport between tasks

[vdemeester]

Context

Multi-task pipelines on the dogfooding cluster currently use PVC-backed workspaces to pass data between tasks. On OCI/OKE, every volumeClaimTemplate creates a 50 GiB block volume (OCI minimum).

For pipelines that can't be collapsed into single Tasks (#3379) — primarily release pipelines (7-13 tasks), bastion nightly tests, and CD deploy pipelines — we need an alternative cross-task data transport mechanism.

The tekton-experiments repo demonstrates using OCI artifacts (via oras push/pull) for data transport between tasks, eliminating PVCs entirely. This requires an OCI registry to push/pull intermediate artifacts.

Proposal: deploy zot as in-cluster registry

zot is a lightweight, OCI-native registry that fits this use case well:

Deployment

• Single pod, ~30MB memory footprint
• emptyDir storage — intermediate CI artifacts are ephemeral, no need for persistence
• ClusterIP service only (cluster-internal, no ingress needed)
• No authentication for cluster-internal use (or k8s service account tokens if needed)
• Namespace: tekton-pipelines or dedicated registry namespace

Cleanup / retention

• Built-in garbage collection with configurable retention policy
• Set retain.delay: "24h" to auto-delete unreferenced artifacts
• Tag retention policy: keep last N tags per repo
• No CronJob needed — GC runs as part of the registry process

Sample deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zot-registry
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: zot
          image: ghcr.io/project-zot/zot-linux-amd64:latest
          ports:
            - containerPort: 5000
          resources:
            requests:
              memory: 64Mi
              cpu: 50m
            limits:
              memory: 256Mi
              cpu: 200m
          volumeMounts:
            - name: data
              mountPath: /var/lib/registry
            - name: config
              mountPath: /etc/zot
      volumes:
        - name: data
          emptyDir:
            sizeLimit: 10Gi
        - name: config
          configMap:
            name: zot-config
---
apiVersion: v1
kind: Service
metadata:
  name: zot-registry
spec:
  type: ClusterIP
  ports:
    - port: 5000
  selector:
    app: zot-registry

Usage in pipelines

Tasks would use oras (or the create-oci-artifact / use-oci-artifact StepActions from tekton-experiments) to push/pull artifacts:

# Push task output
oras push zot-registry.tekton-pipelines.svc:5000/artifacts/${PIPELINE_RUN}:source ./source/

# Pull in next task  
oras pull zot-registry.tekton-pipelines.svc:5000/artifacts/${PIPELINE_RUN}:source

Trade-offs

• Pro: fast (no network egress), no rate limits, self-contained, no external dependency
• Pro: no auth complexity, artifacts auto-expire
• Pro: useful beyond artifact transport — local bundle resolution, test image caching
• Con: if pod restarts, in-flight pipelines may fail (acceptable for CI)
• Con: another component to maintain (though very low maintenance)

Future: TEP-0164

TEP-0164 (Tekton Artifacts Phase 2) will make OCI artifact transport native to the Tekton controller — no StepActions or explicit registry config needed. The in-cluster registry would still be useful as the backing store for TEP-0164, or could be retired once TEP-0164 ships with its own storage.

Related

• Collapse simple 2-task pipelines into single Tasks to eliminate PVC usage #3379: Collapse simple pipelines into single Tasks
• Add tekton.dev/auto-cleanup-pvc annotation to all PipelineRun templates #3377 / chore: add tekton.dev/auto-cleanup-pvc annotation to all PipelineRun templates #3378: Auto-cleanup PVC annotation
• Reduce pruner history limits on dogfooding cluster #3376: Reduce pruner history limits
• tekton-experiments: OCI artifact PoC

/kind feature

[vdemeester]

Security considerations

Content poisoning: mitigated by digest-based references

The tekton-experiments StepActions export artifact digests as Task results. Consuming Tasks pull by digest, not tag. Even if a malicious workload overwrites a tag, the digest reference is immutable — you get the exact original content or a "not found" error.

Additionally, artifact paths should include the PipelineRun UID or name (which has a random suffix) to avoid tag collisions between concurrent runs:

zot:5000/artifacts/${PIPELINE_RUN_UID}:source@sha256:...

Namespace isolation: supported natively via k8s SA tokens

Zot supports Kubernetes service account tokens as OIDC bearer tokens out of the box (since v2.1.14+, see examples/config-bearer-oidc-workload.json). No extra identity provider needed — zot validates tokens directly against the k8s API server's OIDC endpoint.

This enables per-namespace access control:

{
  "auth": {
    "bearer": {
      "realm": "zot",
      "service": "zot-registry",
      "oidc": [{
        "issuer": "https://kubernetes.default.svc.cluster.local",
        "audiences": ["zot-registry"],
        "claimMapping": { "username": "claims.sub" }
      }]
    }
  },
  "accessControl": {
    "repositories": {
      "artifacts/tekton-ci/**": {
        "policies": [{
          "users": ["system:serviceaccount:tekton-ci:*"],
          "actions": ["read", "create", "update", "delete"]
        }]
      },
      "artifacts/tekton-nightly/**": {
        "policies": [{
          "users": ["system:serviceaccount:tekton-nightly:*"],
          "actions": ["read", "create", "update", "delete"]
        }]
      }
    }
  }
}

The SA token username is system:serviceaccount:<namespace>:<sa-name>, so pipelines in tekton-ci can only access artifacts/tekton-ci/**. Zero extra infrastructure — the k8s API server is already an OIDC issuer.

On the client side, the projected SA token (/var/run/secrets/kubernetes.io/serviceaccount/token) is passed to oras as a bearer token.

Other considerations

• DoS (storage exhaustion): emptyDir.sizeLimit caps total storage. Zot's GC retention policy (24h) prevents accumulation. Per-repo size limits are also configurable.
• No TLS: cluster-internal ClusterIP service, unencrypted on pod network. Same as any internal service. Could add if needed.
• Pod restart: ephemeral storage means in-flight pipelines may fail on restart. Acceptable for CI artifacts — pipelines would just retry.

Recommendation

Start without auth (simplest, cluster-internal only). If namespace isolation becomes a requirement, enable the k8s SA token auth — it's a config-only change, no infrastructure to add.