Skip to content

Test lean CI PR run option #540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
+92 −14
Open

Test lean CI PR run option #540

Changes from 16 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
eab0cf1
Add an option to switch
rgsl888prabhu Oct 24, 2025
abc92ea
Test with lean on
rgsl888prabhu Oct 24, 2025
5586615
add midding job
rgsl888prabhu Oct 24, 2025
f7fe3f4
fix style
rgsl888prabhu Oct 24, 2025
3a79e12
fix style
rgsl888prabhu Oct 24, 2025
acf5df1
fix lean
rgsl888prabhu Oct 24, 2025
bdf1ba9
simplify
rgsl888prabhu Oct 24, 2025
bb5a483
fix
rgsl888prabhu Oct 24, 2025
a73e916
fix
rgsl888prabhu Oct 27, 2025
b7a765d
fix
rgsl888prabhu Oct 28, 2025
7f01697
fix matrix
rgsl888prabhu Oct 28, 2025
f3394f3
Merge branch 'main' into add_lean_ci_option
rgsl888prabhu Oct 29, 2025
786b60f
fix minor changes
rgsl888prabhu Oct 29, 2025
eac0671
Merge branch 'add_lean_ci_option' of github.com:rgsl888prabhu/cuopt_p…
rgsl888prabhu Oct 29, 2025
6c746c7
enable full ci
rgsl888prabhu Oct 29, 2025
45bdd25
Merge branch 'main' into add_lean_ci_option
rgsl888prabhu Oct 29, 2025
56a5763
fix lean CI to trigger from label
rgsl888prabhu Oct 31, 2025
f982bc7
fix style
rgsl888prabhu Oct 31, 2025
45d03dd
Address review
rgsl888prabhu Oct 31, 2025
fc71ad2
update
rgsl888prabhu Oct 31, 2025
f37deb5
update
rgsl888prabhu Oct 31, 2025
a2a31f6
fix
rgsl888prabhu Oct 31, 2025
3c660c0
Merge branch 'main' into add_lean_ci_option
rgsl888prabhu Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 59 additions & 14 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,14 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
# Set to 'true' for lean CI (amd64, latest Python/CUDA only) or 'false' for full CI (all platforms)
LEAN_CI: 'false'

Copy link
Contributor

@gforsyth gforsyth Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is going to work.

First, you can imagine the headache of constantly switching this back and forth on a per-PR basis.

But more importantly, when PRs are opened from forks, the workflows aren't run from changes to that fork, they're run from the base repo to prevent malicious changes to CI.

These changes would only take effect after the PR is merged, which is too late.

If you want to have variable CI runs, I would recommend instead setting up some conditions where the matrix_filter gets adjusted if a specific label is applied to a PR.

Copy link
Collaborator Author

@rgsl888prabhu rgsl888prabhu Oct 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This is mostly meant for long running PRs which span across several weeks and also to keep the run short so that we don't spend too much time on running whole matrix.
  • This PR is created from fork and I was able to run my changes in this PR, I think authorized user is the key on running or stopping a PR
  • I like the idea on label, let me explore that. Thank you for the tip.

jobs:
pr-builder:
needs:
- compute-matrix-filters
- changed-files
- checks
- conda-cpp-build
Expand All @@ -44,6 +49,38 @@ jobs:
- test-self-hosted-server
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@main
compute-matrix-filters:
runs-on: ubuntu-latest
outputs:
conda_lean_filter: ${{ steps.set-filters.outputs.conda_lean_filter }}
conda_test_filter: ${{ steps.set-filters.outputs.conda_test_filter }}
wheel_lean_filter: ${{ steps.set-filters.outputs.wheel_lean_filter }}
mps_parser_filter: ${{ steps.set-filters.outputs.mps_parser_filter }}
libcuopt_filter: ${{ steps.set-filters.outputs.libcuopt_filter }}
cuopt_server_filter: ${{ steps.set-filters.outputs.cuopt_server_filter }}
cuopt_sh_client_filter: ${{ steps.set-filters.outputs.cuopt_sh_client_filter }}
steps:
- name: Set matrix filters
id: set-filters
run: |
if [ "${{ env.LEAN_CI }}" == "true" ]; then
echo "conda_lean_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.10\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
echo "conda_test_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.13\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
echo "wheel_lean_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.12\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
echo "mps_parser_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.12\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
echo "libcuopt_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.12\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
echo "cuopt_server_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.12\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
echo "cuopt_sh_client_filter=[map(select(.ARCH == \"amd64\" and .PY_VER == \"3.12\")) | max_by(.CUDA_VER | split(\".\") | map(tonumber))]" >> $GITHUB_OUTPUT
else
echo "conda_lean_filter=." >> $GITHUB_OUTPUT
echo "conda_test_filter=." >> $GITHUB_OUTPUT
echo "wheel_lean_filter=." >> $GITHUB_OUTPUT
echo "mps_parser_filter=group_by([.ARCH, (.PY_VER |split(\".\") | map(tonumber))])|map(max_by([(.CUDA_VER|split(\".\")|map(tonumber))]))" >> $GITHUB_OUTPUT
echo "libcuopt_filter=group_by([.ARCH, (.CUDA_VER|split(\".\")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(\".\")|map(tonumber)))" >> $GITHUB_OUTPUT
echo "cuopt_server_filter=map(select(.ARCH == \"amd64\")) | group_by(.CUDA_VER|split(\".\")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(\".\")|map(tonumber)), (.CUDA_VER|split(\".\")|map(tonumber))]))" >> $GITHUB_OUTPUT
echo "cuopt_sh_client_filter=[map(select(.ARCH == \"amd64\")) | min_by((.PY_VER | split(\".\") | map(tonumber)), (.CUDA_VER | split(\".\") | map(-tonumber)))]" >> $GITHUB_OUTPUT
fi

changed-files:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/changed-files.yaml@main
Expand Down Expand Up @@ -114,38 +151,41 @@ jobs:
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@main
with:
enable_check_generated_files: false

conda-cpp-build:
needs: checks
needs: [checks, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@main
with:
build_type: pull-request
script: ci/build_cpp.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.conda_lean_filter }}
conda-cpp-tests:
needs: [conda-cpp-build, changed-files]
needs: [conda-cpp-build, changed-files, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@main
#if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp
with:
build_type: pull-request
script: ci/test_cpp.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.conda_test_filter }}
conda-python-build:
needs: conda-cpp-build
needs: [conda-cpp-build, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@main
with:
build_type: pull-request
script: ci/build_python.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.conda_test_filter }}
conda-python-tests:
needs: [conda-python-build, changed-files]
needs: [conda-python-build, changed-files, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main
#if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
run_codecov: false
build_type: pull-request
script: ci/test_python.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.conda_test_filter }}
docs-build:
needs: conda-python-build
secrets: inherit
Expand All @@ -159,6 +199,7 @@ jobs:
container_image: "rapidsai/ci-conda:25.12-latest"
script: "ci/build_docs.sh"
wheel-build-cuopt-mps-parser:
needs: compute-matrix-filters
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
with:
Expand All @@ -168,37 +209,39 @@ jobs:
package-type: python
append-cuda-suffix: false
# need 1 build per Python version and arch (but CUDA version doesn't matter so choose the latest)
matrix_filter: 'group_by([.ARCH, (.PY_VER |split(".") | map(tonumber))])|map(max_by([(.CUDA_VER|split(".")|map(tonumber))]))'
matrix_filter: ${{ needs.compute-matrix-filters.outputs.mps_parser_filter }}
wheel-build-libcuopt:
needs: wheel-build-cuopt-mps-parser
needs: [wheel-build-cuopt-mps-parser, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
with:
# build for every combination of arch and CUDA version, but only for the latest Python
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
matrix_filter: ${{ needs.compute-matrix-filters.outputs.libcuopt_filter }}
package-type: cpp
package-name: libcuopt
build_type: pull-request
script: ci/build_wheel_libcuopt.sh
wheel-build-cuopt:
needs: [wheel-build-cuopt-mps-parser, wheel-build-libcuopt]
needs: [wheel-build-cuopt-mps-parser, wheel-build-libcuopt, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
with:
build_type: pull-request
script: ci/build_wheel_cuopt.sh
package-name: cuopt
package-type: python
matrix_filter: ${{ needs.compute-matrix-filters.outputs.wheel_lean_filter }}
wheel-tests-cuopt:
needs: [wheel-build-cuopt, wheel-build-cuopt-mps-parser, wheel-build-cuopt-sh-client, changed-files]
needs: [wheel-build-cuopt, wheel-build-cuopt-mps-parser, wheel-build-cuopt-sh-client, changed-files, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
#if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_cuopt
with:
build_type: pull-request
script: ci/test_wheel_cuopt.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.wheel_lean_filter }}
wheel-build-cuopt-server:
needs: checks
needs: [checks, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
with:
Expand All @@ -208,8 +251,9 @@ jobs:
package-type: python
pure-wheel: true
# Only need 1 package per CUDA major version. This selects "ARCH=amd64 + the latest supported Python, 1 job per major CUDA version".
matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))]))
matrix_filter: ${{ needs.compute-matrix-filters.outputs.cuopt_server_filter }}
wheel-build-cuopt-sh-client:
needs: compute-matrix-filters
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
with:
Expand All @@ -220,15 +264,16 @@ jobs:
append-cuda-suffix: false
pure-wheel: true
# only need 1 build (noarch package): this selects amd64, oldest-supported Python, latest-supported CUDA
matrix_filter: '[map(select(.ARCH == "amd64")) | min_by((.PY_VER | split(".") | map(tonumber)), (.CUDA_VER | split(".") | map(-tonumber)))]'
matrix_filter: ${{ needs.compute-matrix-filters.outputs.cuopt_sh_client_filter }}
wheel-tests-cuopt-server:
needs: [wheel-build-cuopt, wheel-build-cuopt-server, changed-files]
needs: [wheel-build-cuopt, wheel-build-cuopt-server, changed-files, compute-matrix-filters]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
#if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_cuopt_server
with:
build_type: pull-request
script: ci/test_wheel_cuopt_server.sh
matrix_filter: ${{ needs.compute-matrix-filters.outputs.wheel_lean_filter }}
test-self-hosted-server:
needs: [wheel-build-cuopt, wheel-build-cuopt-server, changed-files]
secrets: inherit
Expand Down