Skip to content

⚠ Consolidate ClusterObjectSet pre-apply check failures under Ready=False/ValidationFailure #2985

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
perdasilva wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚠ Consolidate ClusterObjectSet pre-apply check failures under Ready=False/ValidationFailure #2985

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions api/v1/clusterobjectset_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@ const (
ClusterObjectSetTypeReady = "Ready"

// Condition Reasons
ClusterObjectSetReasonArchived = "Archived"
ClusterObjectSetReasonBlocked = "Blocked"
ClusterObjectSetReasonProbeFailure = "ProbeFailure"
ClusterObjectSetReasonRetryableError = "RetryableError"
ClusterObjectSetReasonAllObjectsReady = "AllObjectsReady"
ClusterObjectSetReasonArchived = "Archived"
ClusterObjectSetReasonProbeFailure = "ProbeFailure"
ClusterObjectSetReasonRetryableError = "RetryableError"
ClusterObjectSetReasonValidationFailure = "ValidationFailure"
ClusterObjectSetReasonAllObjectsReady = "AllObjectsReady"
)

// ClusterObjectSetSpec defines the desired state of ClusterObjectSet.
Expand Down Expand Up @@ -500,9 +500,9 @@ type ClusterObjectSetStatus struct {
// - True with reason AllObjectsReady: the revision has rolled out and all objects are at the desired state.
// - False with reason ProbeFailure: one or more objects are failing their readiness probes during rollout.
// - False with reason RollingOut: the revision is actively rolling out and has not yet become ready.
// - False with reason Blocked: the revision has encountered an error that requires manual intervention for recovery.
// - False with reason ValidationFailure: the revision failed a pre-apply check (an invalid revision/phase object or probe, preflight validation, a mutable referenced Secret, or changed referenced content) and requires manual intervention.
// - False with reason ProgressDeadlineExceeded: the revision did not roll out within spec.progressDeadlineMinutes.
// - False with reason Reconciling: the revision encountered an error that prevented it from observing the probes.
// - False with reason RetryableError: the revision encountered a transient error that may resolve on a subsequent reconciliation.
// - False with reason Archived: the revision has been archived and its objects have been torn down.
//
// Rollout completion is recorded separately by status.completedAt.
Expand Down
4 changes: 2 additions & 2 deletions applyconfigurations/api/v1/clusterobjectsetstatus.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions docs/draft/concepts/clusterobjectsets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ Indicates whether all objects in the revision have been successfully rolled out
| --- | --- | --- |
| True | `AllObjectsReady` | All objects are at the desired state |
| False | `ProbeFailure` | One or more probes failing |
| False | `ValidationFailure` | A pre-apply check failed (an invalid revision/phase object or probe, preflight validation, a mutable referenced Secret, or changed referenced content); requires manual intervention |
| Unknown | `RetryableError` | Error prevented probe observation |
Comment thread
perdasilva marked this conversation as resolved.
| Unknown | `Archived` | Objects torn down after archival |
| Unknown | `Migrated` | Migrated from existing release; probes not yet observed |
Expand Down
28 changes: 18 additions & 10 deletions docs/draft/concepts/large-bundle-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,10 +149,10 @@ Recommended conventions:

2. **Immutability**: Secrets must set `immutable: true`. The reconciler verifies
that all referenced Secrets have `immutable: true` set before proceeding.
Mutable referenced Secrets are rejected and reconciliation is blocked with
`Progressing=False, Reason=Blocked`. Additionally, the reconciler records
Mutable referenced Secrets are rejected and the ClusterObjectSet reports
`Ready=False, Reason=ValidationFailure`. Additionally, the reconciler records
content hashes of the resolved phases on first successful reconciliation
and blocks reconciliation if the content changes (e.g., if a Secret is
and reports the same condition if the content changes (e.g., if a Secret is
deleted and recreated with the same name but different data).

3. **Owner references**: Referenced Secrets should carry an ownerReference to
Expand Down Expand Up @@ -396,8 +396,8 @@ Before resolving individual object refs, the reconciler verifies that all
referenced Secrets have `immutable: true` set. After successfully building
the phases (resolving all refs), the reconciler computes a per-phase content
digest and compares it against the digests recorded in `.status.observedPhases`
(if present). If any phase's content has changed, reconciliation is blocked
with `Progressing=False, Reason=Blocked`. On first successful build, phase
(if present). If any phase's content has changed, the ClusterObjectSet reports
`Ready=False, Reason=ValidationFailure`. On first successful build, phase
content digests are persisted to status for future comparisons.

When processing a COS phase:
Expand All @@ -412,11 +412,19 @@ When processing a COS phase:

Under normal operation, referenced Secrets are guaranteed to exist before the
COS is created (see [Crash-safe creation sequence](#crash-safe-creation-sequence)).
If a referenced Secret or key is not found — indicating an inconsistent state
caused by external modification or a partially completed creation sequence —
the reconciler returns a retryable error, allowing the controller to retry on
subsequent reconciliation attempts. This handles transient issues such as
informer cache lag after Secret creation.
The reconciler distinguishes two failure modes:

- A referenced **Secret that does not exist yet** — for example, informer cache lag
after creation, or a partially completed creation sequence — is treated as a
transient error. The ClusterObjectSet reports `Ready=False, Reason=RetryableError`
and the controller retries with backoff.
Comment on lines +417 to +420

@perdasilva perdasilva Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is ok for this PR - but when I remove RetryableError, I'll probably fold this case into ValidationFailed (with error return for exp. backoff (as opposed to 10 seconds re-enqueue))

- **Invalid referenced content** — a missing key, a malformed manifest, or content
that has changed since first rollout — is a validation failure. The ClusterObjectSet
reports `Ready=False, Reason=ValidationFailure`. Because referenced Secrets are not
watched and the `ref` is immutable, the reconciler retries this case periodically so
the revision recovers automatically once the referenced Secret is replaced with
corrected content. Referenced Secrets are immutable, so correcting the content means
deleting and recreating the Secret (it cannot be edited in place).

Secrets are fetched using the typed client served from the informer cache.

Expand Down
4 changes: 2 additions & 2 deletions ...1/base/object-controller/crd/experimental/olm.operatorframework.io_clusterobjectsets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -561,9 +561,9 @@ spec:
- True with reason AllObjectsReady: the revision has rolled out and all objects are at the desired state.
- False with reason ProbeFailure: one or more objects are failing their readiness probes during rollout.
- False with reason RollingOut: the revision is actively rolling out and has not yet become ready.
- False with reason Blocked: the revision has encountered an error that requires manual intervention for recovery.
- False with reason ValidationFailure: the revision failed a pre-apply check (an invalid revision/phase object or probe, preflight validation, a mutable referenced Secret, or changed referenced content) and requires manual intervention.
- False with reason ProgressDeadlineExceeded: the revision did not roll out within spec.progressDeadlineMinutes.
- False with reason Reconciling: the revision encountered an error that prevented it from observing the probes.
- False with reason RetryableError: the revision encountered a transient error that may resolve on a subsequent reconciliation.
- False with reason Archived: the revision has been archived and its objects have been torn down.

Rollout completion is recorded separately by status.completedAt.
Expand Down
Loading
Loading