[WIP] OCPBUGS-65621: add dedicated service account to crb, cvo and version pod #1266

ehearne-redhat · 2025-11-28T09:42:22Z

What

Add dedicated service account to CVO and version pods in openshift-cluster-version namespace.
Add cluster role binding and attach dedicated service account to it.

Why

Default service account should not be used on OpenShift components.

coderabbitai · 2025-11-28T09:42:34Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Excluded labels (none allowed) (1)

do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci · 2025-11-28T09:43:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ehearne-redhat
Once this PR has been reviewed and has the lgtm label, please assign wking for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ehearne-redhat · 2025-11-28T12:45:14Z

/retest

ehearne-redhat · 2025-11-28T12:46:37Z

/test e2e-aws-ovn-upgrade

openshift-ci · 2025-11-28T12:46:44Z

@ehearne-redhat: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test e2e-agnostic-operator

/test e2e-agnostic-operator-techpreview

/test e2e-agnostic-ovn

/test e2e-agnostic-ovn-upgrade-into-change

/test e2e-agnostic-ovn-upgrade-into-change-techpreview

/test e2e-agnostic-ovn-upgrade-out-of-change

/test e2e-agnostic-ovn-upgrade-out-of-change-techpreview

/test e2e-aws-ovn-techpreview

/test e2e-hypershift

/test e2e-hypershift-conformance

/test gofmt

/test images

/test lint

/test okd-scos-images

/test unit

/test verify-deps

/test verify-update

/test verify-yaml

The following commands are available to trigger optional jobs:

/test e2e-agnostic-operator-devpreview

/test e2e-extended-tests

/test okd-scos-e2e-aws-ovn

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-operator

pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn

pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change

pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-out-of-change

pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview

pull-ci-openshift-cluster-version-operator-main-e2e-hypershift

pull-ci-openshift-cluster-version-operator-main-e2e-hypershift-conformance

pull-ci-openshift-cluster-version-operator-main-gofmt

pull-ci-openshift-cluster-version-operator-main-images

pull-ci-openshift-cluster-version-operator-main-lint

pull-ci-openshift-cluster-version-operator-main-okd-scos-images

pull-ci-openshift-cluster-version-operator-main-unit

pull-ci-openshift-cluster-version-operator-main-verify-deps

pull-ci-openshift-cluster-version-operator-main-verify-update

pull-ci-openshift-cluster-version-operator-main-verify-yaml

In response to this:

/test e2e-aws-ovn-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ehearne-redhat · 2025-11-28T12:52:20Z

/test e2e-aws-ovn-techpreview

ehearne-redhat · 2025-11-29T18:18:15Z

/retest

ehearne-redhat · 2025-11-29T18:35:39Z

install/0000_00_cluster-version-operator_03_1_roles.yaml

+roleRef:
+  kind: ClusterRole
+  name: system:openshift:scc:privileged
+  apiGroup: rbac.authorization.k8s.io


I'm unsure whether this is needed or not... I initially added it because I was looking at the wrong test e2e-upgrade-into-change, and did not see the update-payload-dedicated-sa service account on version- pod in must gather files. However, I did see them in e2e-upgrade-out-of-change .

I'm thinking it might be necessary as the updatepayload pod contains the openshift.io/required-scc: privileged annotation . So, I'll leave it for the reviewer to decide. I can test without too if that suits.

install/0000_00_cluster-version-operator_02_roles.yaml

ehearne-redhat · 2025-11-30T16:45:44Z

/retest

openshift-ci-robot · 2025-12-01T09:19:27Z

@ehearne-redhat: This pull request references Jira Issue OCPBUGS-65621, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

WIP - do not merge.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

ehearne-redhat · 2025-12-01T09:20:04Z

/jira refresh

openshift-ci-robot · 2025-12-01T09:20:15Z

@ehearne-redhat: This pull request references Jira Issue OCPBUGS-65621, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @dis016

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-12-01T09:24:14Z

@ehearne-redhat: This pull request references Jira Issue OCPBUGS-65621, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @dis016

In response to this:

What

Add dedicated service account to CVO and version pods in openshift-cluster-version namespace.

Add cluster role binding and attach dedicated service account to it.

Why

Default service account should not be used on OpenShift components.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

wking · 2025-12-01T23:04:06Z

pkg/cvo/updatepayload.go

 			},
 		},
 		Spec: corev1.PodSpec{
+			ServiceAccountName:    "update-payload-dedicated-sa",


Default service account should not be used on OpenShift components

I think "Don't attach privileges to the default service account in OpenShift namespaces" makes a lot of sense, but I'm less clear on the downsides of Pods using the default service account if that account comes with no privileges. This version Pod does not need any Kube API privileges. In fact, it doesn't need any network communication at all, it's just shoveling bits around between the local container filesystem and a volume-mounted host directory. Can we leave it running in the default service account, or is there a way to request no-service-account-at-all? Maybe that's effectively what automountServiceAccountToken: false does?

Or maybe this is what you're talking about in this thread, and some layer is using the system:openshift:scc:privileged ClusterRole to decide if a Pod is allowed to have the privileged Security Context Constraint? Not clear to me how that would be enforced though. Are there docs on the system:openshift:scc:* ClusterRoles walking through this?

Poking around in docs turned up these sections dancing in this space, but I'm still not clear if system:openshift:scc:* ClusterRole access is checked on something in the Pod-creation path, or checked against the ServiceAccount that is about to be bound to the Pod being created. I also turned up a directory with release-image manifests for many of these ClusterRoles, but no README.md or other docs there explaining who is expected to use them how, or exactly how the guard that enforces them works.

@wking sorry for the late response - service accounts need to be attached to a pod, whether we specify one or not. If we specified no service account, the default one would be used in its place.

Thanks for letting me know about minimum permissions here. We can create a service account without specifying permissions. This way, we don't elevate permissions but we also don't allow default service account usage. I can see if this is all that is required for this deployment.

Thanks for looking into further docs on this topic. We can test without to see if the pod genuinely requires elevated permissions or not.

wking · 2025-12-01T23:28:40Z

install/0000_00_cluster-version-operator_03_1_roles.yaml

@@ -8,7 +8,21 @@ metadata:
 roleRef:
  kind: ClusterRole
  name: cluster-admin
+  apiGroup: rbac.authorization.k8s.io


huh, did we not need this before for some reason? Maybe we'd been relying on there only being one group that contained a ClusterRole kind? 😅 If you have any context on how this worked without this line, having that context in a commit message for future devs would be nice. Seems like there should be something in-cluster alerting on (Cluster)RoleBindings that leave off such an important property.

I don't have too much context on how it worked without this line. I added it to see if it would resolve test errors I got from cluster-version-operator pods. It seemed like an important property, so I added it back. I can also test without this line too.

namespace/openshift-cluster-version node/ci-op-yy1vhb3d-0bd36-c4ngb-master-2 pod/cluster-version-operator-5b954c7498-xprpt uid/51a23f9d-39b8-4a29-bebd-30fb7de19b57 container/cluster-version-operator restarted 30 times at: non-zero exit at 2025-12-01 15:56:52.358476306 +0000 UTC m=+612.085122721: cause/Error code/255 reason/ContainerExit ernalversions/factory.go:125" type="*v1.FeatureGate" E1201 15:56:22.959953 1 reflector.go:205] "Failed to watch" err="failed to list *v1.FeatureGate: featuregates.config.openshift.io is forbidden: User \"system:serviceaccount:openshift-cluster-version:cvo-dedicated-sa\" cannot list resource \"featuregates\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="github.com/openshift/client-go/config/informers/externalversions/factory.go:125" type="*v1.FeatureGate"

I keep getting this error, for example, in e2e-agnostic-ovn-upgrade-into-change and e2e-agnostic-ovn-upgrade-out-of-change .

install/0000_00_cluster-version-operator_02_service_account.yaml

wking · 2025-12-01T23:33:42Z

install/0000_00_cluster-version-operator_02_service_account.yaml

+  namespace: openshift-cluster-version
+  annotations:
+    kubernetes.io/description: Dedicated Service Account for the Cluster Version Operator.
+    include.release.openshift.io/self-managed-high-availability: "true"


I haven't looked into HyperShift, but I expect we'll need a ServiceAccount in the hosted-Kube-API there too?

I can look further into this and get back to you. :)

They appear to be using their own service account and setup --> https://github.com/openshift/hypershift/blob/main/control-plane-operator/controllers/hostedcontrolplane/v2/assets/cluster-version-operator/deployment.yaml

If I can't get my changes to work it might not be a bad idea to replicate their own setup. Thanks for pointing me in the right direction! :)

install/0000_00_cluster-version-operator_02_roles.yaml

ehearne-redhat · 2025-12-03T18:58:53Z

/test e2e-hypershift

ehearne-redhat · 2025-12-03T18:59:20Z

Will address failing tests tomorrow.

openshift-ci · 2025-12-05T18:13:20Z

@ehearne-redhat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agnostic-ovn-upgrade-out-of-change	`0c875cc`	link	true	`/test e2e-agnostic-ovn-upgrade-out-of-change`
ci/prow/e2e-agnostic-ovn-upgrade-into-change	`0c875cc`	link	true	`/test e2e-agnostic-ovn-upgrade-into-change`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

add dedicated service account to crb, cvo and version pod

bf44d47

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 28, 2025

ehearne-redhat added 2 commits November 28, 2025 09:45

add new line to keep linter happy

29bce17

add sa to testrendermanifest

2982ece

add dedicated sa for update payload

ca25977

ehearne-redhat commented Nov 29, 2025

View reviewed changes

add cluster-admin clusterrole

825ac61

ehearne-redhat commented Nov 29, 2025

View reviewed changes

install/0000_00_cluster-version-operator_02_roles.yaml Outdated Show resolved Hide resolved

ehearne-redhat added 2 commits November 30, 2025 20:05

remove cluster-admin role from file

446f19b

add new cluster role with watch feature gate

bb1602f

ehearne-redhat changed the title ~~WIP: add dedicated service account to crb, cvo and version pod~~ [WIP] OCPBUGS-65621: add dedicated service account to crb, cvo and version pod Dec 1, 2025

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Dec 1, 2025

openshift-ci bot requested a review from dis016 December 1, 2025 09:20

re-add cluster-admin

d7e4cd0

wking reviewed Dec 1, 2025

View reviewed changes

install/0000_00_cluster-version-operator_02_service_account.yaml Outdated Show resolved Hide resolved

wking reviewed Dec 1, 2025

View reviewed changes

install/0000_00_cluster-version-operator_02_roles.yaml Outdated Show resolved Hide resolved

ehearne-redhat added 2 commits December 3, 2025 13:11

rename roles to ensure service account is added first

5641681

rename cvo-dedicated-sa to cluster-version-operator

5fe9198

ehearne-redhat added 4 commits December 4, 2025 10:02

add default sa crb back to test into and out of change test failures

6310c45

add back featuregate read role + remove default crb

b39df2b

add new reader permissions

7d69c50

add leases role and role binding

0c875cc

[WIP] OCPBUGS-65621: add dedicated service account to crb, cvo and version pod #1266

Are you sure you want to change the base?

[WIP] OCPBUGS-65621: add dedicated service account to crb, cvo and version pod #1266

Uh oh!

Conversation

ehearne-redhat commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Uh oh!

coderabbitai bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci bot commented Nov 28, 2025

Uh oh!

ehearne-redhat commented Nov 28, 2025

Uh oh!

ehearne-redhat commented Nov 28, 2025

Uh oh!

openshift-ci bot commented Nov 28, 2025

Uh oh!

ehearne-redhat commented Nov 28, 2025

Uh oh!

ehearne-redhat commented Nov 29, 2025

Uh oh!

ehearne-redhat Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ehearne-redhat commented Nov 30, 2025

Uh oh!

openshift-ci-robot commented Dec 1, 2025

Uh oh!

ehearne-redhat commented Dec 1, 2025

Uh oh!

openshift-ci-robot commented Dec 1, 2025

Uh oh!

openshift-ci-robot commented Dec 1, 2025

What

Why

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ehearne-redhat commented Dec 3, 2025

Uh oh!

ehearne-redhat commented Dec 3, 2025

Uh oh!

openshift-ci bot commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

ehearne-redhat commented Nov 28, 2025 •

edited

Loading

coderabbitai bot commented Nov 28, 2025 •

edited

Loading

ehearne-redhat Nov 29, 2025 •

edited

Loading