Skip to content

OPRUN-4054: [OTE] add catalog tests from openshift/origin #419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 21, 2025
Merged

OPRUN-4054: [OTE] add catalog tests from openshift/origin #419

merged 1 commit into from
Aug 21, 2025

Conversation

camilamacedo86
Copy link
Contributor

@camilamacedo86 camilamacedo86 commented Aug 1, 2025 •
edited
Loading

This commit migrates the olmv1_catalog set of tests from openshift/origin
to OTE as part the broad effort to migrate all tests.
We are also doing some improvements to avoid flakes by ensuring cleanup and unique names

See that pass in the aggregation as well: https://pr-payload-tests.ci.openshift.org/runs/ci/48ad6920-7da7-11f0-93ef-7fa1591d2e23-0

@openshift-ci openshift-ci bot requested review from grokspawn and tmshort August 1, 2025 10:13
@camilamacedo86 camilamacedo86 force-pushed the add-catalog-tests-only-ote branch 3 times, most recently from 544842d to 5ac974e Compare August 1, 2025 11:00
@camilamacedo86
Copy link
Contributor Author

/retest-required

@camilamacedo86 camilamacedo86 changed the title [OTE] add catalog tests from openshift/origin OPRUN-4054: [OTE] add catalog tests from openshift/origin Aug 4, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 4, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 4, 2025 •
edited by openshift-ci bot
Loading

@camilamacedo86: This pull request references OPRUN-4054 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

Summary of Changes

This PR migrates the OLMv1 catalog tests from openshift/origin/test/extended/olm/olmv1-catalogd.go to the OpenShift Test Extension (OTE) framework. The new implementation replaces the use of the oc command-line interface with a Go-native, strongly-typed client, and leverages the OLM API directly for resource management.

The migration offers several key improvements:

  • Go-native API Interaction: Replaces oc command executions and manual JSON parsing with direct calls to the Kubernetes API using the controller-runtime client. This provides a more robust, type-safe, and efficient way to interact with cluster resources.
  • Programmatic Resource Management: Instead of relying on testdata YAML files, test resources like ClusterCatalog and Job objects are now created and managed programmatically using Go structs. This eliminates the need for temporary files and enhances test reliability.
  • Improved Polling and Assertions: Switches from manual polling loops (wait.PollUntilContextTimeout) and output parsing to declarative gomega.Eventually blocks. This makes the tests more readable and less fragile when verifying asynchronous state changes.

This change aligns the OLMv1 tests with modern Go testing practices, improves maintainability, and removes the dependency on the oc CLI for test execution.

Comparison of Methodologies

Feature Original Test (openshift/origin) New OTE Test (operator-controller/openshift)
Kubernetes Interaction oc CLI commands (oc get, oc apply) with manual JSON output parsing. Go client (controller-runtime) for direct, type-safe API calls.
Resource Creation Manually writing and applying YAML files from a testdata directory. Programmatic creation of Go structs (e.g., olmv1.ClusterCatalog).
Resource Cleanup Relies on AfterEach blocks with oc delete commands on temporary files. Uses ginkgo.DeferCleanup for a more robust and idiomatic cleanup mechanism.
Asynchronous State Checks wait.PollUntilContextTimeout function with manual logic to parse and check status conditions from oc output. gomega.Eventually with typed assertions against native Go objects.
Dependency Requires the oc CLI to be installed and available in the test environment. No dependency on external CLI tools. Relies solely on the Go test binary and a valid kubeconfig.

--- **Disclaimer:** This summary and comparison was generated by an AI assistant.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@camilamacedo86 camilamacedo86 changed the title OPRUN-4054: [OTE] add catalog tests from openshift/origin WIP OPRUN-4054: [OTE] add catalog tests from openshift/origin Aug 6, 2025
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 6, 2025
@camilamacedo86 camilamacedo86 force-pushed the add-catalog-tests-only-ote branch 6 times, most recently from be4f081 to 0e7e9e0 Compare August 8, 2025 22:02
@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 8, 2025 •
edited by openshift-ci bot
Loading

@camilamacedo86: This pull request references OPRUN-4054 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

This commit migrates the olmv1_catalog set of tests from openshift/origin
to OTE as part the broad effort to migrate all tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
@@ -1,4 +1,104 @@
[
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLM][Skipped:Disconnected] OLMv1 Catalogs should be installed",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1994

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLM][Skipped:Disconnected] OLMv1 openshift-community-operators Catalog should serve FBC via the /v1/api/all endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L2000

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLM][Skipped:Disconnected] OLMv1 openshift-certified-operators Catalog should serve FBC via the /v1/api/all endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1998

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLM][Skipped:Disconnected] OLMv1 openshift-redhat-marketplace Catalog should serve FBC via the /v1/api/all endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smae name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L2002

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLM][Skipped:Disconnected] OLMv1 openshift-redhat-operators Catalog should serve FBC via the /v1/api/all endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L2004C3-L2004C146

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLMCatalogdAPIV1Metas][Skipped:Disconnected] OLMv1 openshift-community-operators Catalog should serve FBC via the /v1/api/metas endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1970

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLMCatalogdAPIV1Metas][Skipped:Disconnected] OLMv1 openshift-certified-operators Catalog should serve FBC via the /v1/api/metas endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1968

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLMCatalogdAPIV1Metas][Skipped:Disconnected] OLMv1 openshift-redhat-marketplace Catalog should serve FBC via the /v1/api/metas endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name; https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1972

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLMCatalogdAPIV1Metas][Skipped:Disconnected] OLMv1 openshift-redhat-operators Catalog should serve FBC via the /v1/api/metas endpoint",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1974

camilamacedo86
camilamacedo86 commented Aug 8, 2025
View reviewed changes
openshift/tests-extension/.openshift-tests-extension/openshift_payload_olmv1.json
"environmentSelector": {}
},
{
"name": "[sig-olmv1][OCPFeatureGate:NewOLM][Skipped:Disconnected] OLMv1 New Catalog Install should fail to install if it has an invalid reference",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same name: https://github.com/openshift/origin/blob/7e8f07d709c533890f1c2f29fddea27fe4acffae/test/extended/util/annotate/generated/zz_generated.annotations.go#L1996

@camilamacedo86
Copy link
Contributor Author

/test e2e-aws-techpreview-olmv1-ext

@camilamacedo86
Copy link
Contributor Author

/test openshift-e2e-aws

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 20, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2025
@kuiwang02
Copy link

@camilamacedo86 FYI.
the case is parallel, while you use /payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview-serial 10, it only run Serial case. so the new case in this PR is not executed.
if you expect /payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview-serial 10 to approve the new case stable, it is not useless.

by the way, there is another job periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview which run parallel case, maybe you need to run it to see if the new case is stable.

@camilamacedo86
Copy link
Contributor Author

Hi @kuiwang02

Thank you,
In this case, it is a migration we would not need to run /payload-aggregate

Anyway, why not ensure 1000000% :-)

Thank you a lot !!!!

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview

@camilamacedo86
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview 5

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 20, 2025

@camilamacedo86: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/48ad6920-7da7-11f0-93ef-7fa1591d2e23-0

@camilamacedo86
Copy link
Contributor Author

/retest-required

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 20, 2025 •
edited by openshift-ci bot
Loading

@camilamacedo86: This pull request references OPRUN-4054 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

This commit migrates the olmv1_catalog set of tests from openshift/origin
to OTE as part the broad effort to migrate all tests.
We are also doing some improvements to avoid flakes by ensuring cleanup and unique names

See that pass in the aggregation as well: https://pr-payload-tests.ci.openshift.org/runs/ci/48ad6920-7da7-11f0-93ef-7fa1591d2e23-0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@camilamacedo86
Copy link
Contributor Author

/retest-required

1 similar comment
@tmshort
Copy link
Contributor

tmshort commented Aug 20, 2025

/retest-required

tmshort
tmshort reviewed Aug 20, 2025
View reviewed changes
@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 20, 2025
@camilamacedo86 camilamacedo86 force-pushed the add-catalog-tests-only-ote branch from c81b8de to 9d2b85f Compare August 21, 2025 07:32
@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 21, 2025
@camilamacedo86
UPSTREAM: <carry>: [OTE] add catalog tests from openshift/origin
11a51fb 
This commit migrates the olmv1_catalog set of tests from openshift/origin
to OTE as part the broad effort to migrate all tests.

Assisted-by: Gemini
@camilamacedo86 camilamacedo86 force-pushed the add-catalog-tests-only-ote branch from 9d2b85f to 11a51fb Compare August 21, 2025 07:38
@camilamacedo86
Copy link
Contributor Author

camilamacedo86 commented Aug 21, 2025 •
edited
Loading

Now, the issues faced in the e2e-aws no longer seems to be related: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_operator-framework-operator-controller/419/pull-ci-openshift-operator-framework-operator-controller-main-openshift-e2e-aws/1958433559538569216

Lets run once more:

/test openshift-e2e-aws

@camilamacedo86
Copy link
Contributor Author

/test openshift-e2e-aws

@camilamacedo86
Copy link
Contributor Author

Issue does not seems realted: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_operator-framework-operator-controller/419/pull-ci-openshift-operator-framework-operator-controller-main-openshift-e2e-aws-techpreview/1958433561212096512

/test openshift-e2e-aws

@tmshort
Copy link
Contributor

tmshort commented Aug 21, 2025

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 21, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 21, 2025

@camilamacedo86: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/openshift-e2e-aws-techpreview 11a51fb link false /test openshift-e2e-aws-techpreview

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot bot merged commit 0c4f430 into openshift:main Aug 21, 2025
12 of 13 checks passed
@openshift-bot
Copy link

[ART PR BUILD NOTIFIER]

Distgit: ose-olm-catalogd
This PR has been included in build ose-olm-catalogd-container-v4.21.0-202508212332.p0.g0c4f430.assembly.stream.el9.
All builds following this will include this PR.

@openshift-bot
Copy link

[ART PR BUILD NOTIFIER]

Distgit: ose-olm-operator-controller
This PR has been included in build ose-olm-operator-controller-container-v4.21.0-202508212332.p0.g0c4f430.assembly.stream.el9.
All builds following this will include this PR.

@camilamacedo86 camilamacedo86 deleted the add-catalog-tests-only-ote branch August 22, 2025 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@grokspawn grokspawn Awaiting requested review from grokspawn

@perdasilva perdasilva Awaiting requested review from perdasilva

@tmshort tmshort Awaiting requested review from tmshort

@kuiwang02 kuiwang02 Awaiting requested review from kuiwang02

Assignees

@tmshort tmshort

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@camilamacedo86 @openshift-ci-robot @tmshort @perdasilva @kuiwang02 @openshift-bot