Skip to content

[CI][DevContainer] Add devcontainer configs for CPU / NPU A3 / NPU A2 and CI consistency workflow - #16891

Closed
Yunme wants to merge 18 commits into
vllm-project:mainfrom
Yunme:main
Closed

Yunme wants to merge 18 commits into
vllm-project:mainfrom
Yunme:main

Conversation

@Yunme

@Yunme Yunme commented Sep 18, 2026 •

Copy link
Copy Markdown

What this PR does / why we need it?

Adds devcontainer development environments that reproduce the PR gate pipeline locally:

  • .devcontainer/cpu — CPU unit tests (cpu-ut stage)
  • .devcontainer/npu — A3 two-card device tests (a3-2 partition)
  • .devcontainer/a2 — A2 910B one-card device tests (a2-1 partition)
  • .devcontainer/activate_npu.sh + select_npu.py — free-card detection
  • .github/workflows/devcontainer_ci_consistency.yaml — CI consistency check between gate and devcontainer

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Local devcontainer bring-up; CI workflow verification pending merge to base branch (GitHub does not run workflow changes from a PR).

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the development experience by providing standardized devcontainer configurations that mirror the project's CI gate pipeline. By automating environment setup and NPU device selection, developers can now reliably reproduce test failures and validate changes locally before pushing to the repository.

Highlights

  • DevContainer Configuration: Added dedicated devcontainer environments for CPU, NPU A3, and NPU A2 to locally reproduce the CI gate pipeline.
  • NPU Resource Management: Introduced activate_npu.sh and select_npu.py to dynamically detect and filter available NPU devices for development environments.
  • CI Consistency: Added a new CI consistency workflow to ensure parity between the gate pipeline and the local devcontainer configurations.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/devcontainer_ci_consistency.yaml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

  • A PR should do only one thing, smaller PRs enable faster reviews.
  • Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
  • Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Suggested PR Title:

[CI][Feature] Add Dev Container configurations for CPU and NPU (A2/A3) environments

Suggested PR Summary:

### What this PR does / why we need it?
This PR adds VS Code Dev Container configurations for CPU, NPU A2 (single card), and NPU A3 (dual card) environments to simplify local reproduction of CI gate tests. It also includes helper scripts (`select_npu.py` and `activate_npu.sh`) to automatically detect and assign free NPU cards.

Feedback on the PR:
- In `.devcontainer/select_npu.py`, `subprocess.run` should specify `check=True` to prevent silent failures when running `npu-smi info`.
- In both `.devcontainer/a2/postCreateCommand.sh` and `.devcontainer/npu/postCreateCommand.sh`, the `MAX_JOBS` environment variable is hardcoded (to 23 and 46), which can cause Out-Of-Memory (OOM) errors on machines with fewer resources. It is recommended to dynamically set this using `nproc`.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
The configurations replicate the environment and steps used in `.github/workflows/pr_test.yaml`.

Comment on lines +45 to +47
out = subprocess.run(
["npu-smi", "info"], capture_output=True, text=True, timeout=30,
).stdout

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The subprocess.run call does not specify check=True. If npu-smi info fails with a non-zero exit code (e.g., due to driver initialization issues or permission errors), subprocess.run will not raise an exception. Instead, it will return a CompletedProcess object with an empty stdout, causing the script to silently conclude that there are no free NPU cards available instead of reporting the actual error. Adding check=True ensures that any non-zero exit code raises a CalledProcessError, which will be properly caught and reported by the except block.

Suggested change
out = subprocess.run(
["npu-smi", "info"], capture_output=True, text=True, timeout=30,
).stdout
res = subprocess.run(
["npu-smi", "info"], capture_output=True, text=True, timeout=30, check=True
)
out = res.stdout

cd "${PROJECT_DIR}"
uv pip install -r requirements-dev.txt
uv pip install --force-reinstall --no-deps triton-ascend==3.2.2
export MAX_JOBS=23

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Hardcoding MAX_JOBS=23 assumes the host machine has at least 24 cores and sufficient memory to support 23 parallel compilation jobs. On developer machines with fewer cores or limited RAM, this can easily trigger Out-Of-Memory (OOM) crashes during kernel compilation. It is highly recommended to dynamically determine the number of jobs based on the available CPU cores, while allowing an override via environment variables.

Suggested change
export MAX_JOBS=23
export MAX_JOBS=${MAX_JOBS:-$(nproc)}

uv pip install --force-reinstall --no-deps triton-ascend==3.2.2

echo "[5/6] 带设备编译安装 vllm-ascend(编译 NPU 自定义内核)"
export MAX_JOBS=46

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Hardcoding MAX_JOBS=46 assumes the host machine has at least 48 cores and sufficient memory to support 46 parallel compilation jobs. On developer machines with fewer cores or limited RAM, this can easily trigger Out-Of-Memory (OOM) crashes during kernel compilation. It is highly recommended to dynamically determine the number of jobs based on the available CPU cores, while allowing an override via environment variables.

Suggested change
export MAX_JOBS=46
export MAX_JOBS=${MAX_JOBS:-$(nproc)}

zhangzhiqiang90 added 18 commits September 20, 2026 11:59
Adds .devcontainer setup with an all-in-one vllm-ascend image to reproduce the pr_test.yaml pre-commit and cpu-ut gate checks locally.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Split devcontainer into cpu/npu configs and add a workflow_dispatch
workflow that uses devcontainers/ci to run each config's
postCreateCommand, validating that CI and devcontainer environments
match for cpu-ut and a3 two-card device tests.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
devcontainer.json mounts ${localEnv:HOME}/.cache, which does not exist
on GitHub hosted runners, causing docker bind mount to fail. Create it
before running devcontainers/ci.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
…eateCommand

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
nightly-main image bundles vllm v0.28.0 tag, whose kv_cache_interface
(KVCacheTensor.shared_by) and attn_utils (_allocate_kv_cache) differ from
the main commit pinned in .github/vllm-main-verified.commit (layers /
allocate_kv_cache), failing 7 kvpp/attention cpu-ut tests.

Check out the gate pin in /vllm-workspace/vllm and rebuild with
VLLM_TARGET_DEVICE=empty before installing vllm-ascend, matching the
'Install vllm-project/vllm from source' step in _selected_tests.yaml.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
…ateCommand

The /vllm-workspace/vllm checkout inside the nightly-main image has its
origin remote pointing at the in-house gh-proxy.test.osinfra.cn mirror,
which GitHub hosted runners cannot reach (HTTP 418). Fetch the pinned vllm
commit from the official github.com URL instead, bypassing the broken
origin config.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The amd64 nightly-main image bakes
url."https://gh-proxy.test.osinfra.cn/https://github.com/".insteadOf
"https://github.com/" into /root/.gitconfig, which makes git rewrite
github.com fetch URLs to the unreachable in-house proxy (HTTP 418) on
GitHub hosted runners. Remove that url section before fetching the pinned
vllm commit.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The devcontainer CI consistency workflow now validates exclusively on
GitHub Actions, so drop the WSL2 squid-proxy fallback (USE_HOST_PROXY,
host.docker.internal:3128) from cpu postCreateCommand and stop passing the
now-unused USE_HOST_PROXY env into both devcontainers/ci jobs.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Mirror the existing npu (A3) devcontainer for the A2 910B single-card
runner so the PR gate a2-1 partition (one_card tests) can be reproduced
locally. Uses the nightly-main all-in-one image and reinstalls vllm from
the verified pin before compiling vllm-ascend, matching the cpu/npu
postCreateCommand scripts.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Align the device-specific devcontainer directory names with the other
targets (cpu / a2) by renaming npu -> a3, which matches the chip family
(A3) rather than the generic "npu" label.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Add the same pull_request trigger as pr_test.yaml (opened/synchronize/
reopened/labeled on main and *-dev/releases/v* branches) while keeping
manual workflow_dispatch. On PR events both cpu and npu jobs run with the
npu job pinned to the default linux-aarch64-a3-2 label; dispatch still
selects targets via inputs.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The a3 job runs on a self-hosted runner without npm, so devcontainers/ci
fails with `npm: command not found` when installing @devcontainers/cli.
Add actions/setup-node to the npu job before the devcontainer build step.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
setup-node downloads from nodejs.org/GitHub releases, which the
self-hosted a3 runner cannot reach, so the job hangs on download. Install
a pinned node arm64 tarball from repo.huaweicloud.com, matching the
schedule_main2main.yaml pattern.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The a3 self-hosted runner job runs as a non-root user, so mkdir /opt/nodejs
fails with permission denied. Install the node tarball under $HOME/nodejs
and prepend its bin dir to PATH via GITHUB_PATH.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Switch the cpu job from ubuntu-latest to the repo's linux-amd64-cpu-8-hk
runner so both jobs use vllm-ascend self-hosted runners. Make the node
install arch-aware (x64/arm64) and reuse it in the cpu job.

Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
pre-commit fails with UP031 (% format) on the error/status prints and a
forbidden `import re`, and ruff-format reformats the file. Replace `re`
with the repo-wide `regex as re` convention and percent formatting with
f-strings so the gate lint passes.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The node install step extracted the tarball and appended its bin dir to
GITHUB_PATH, which only propagates to subsequent steps. The same-step
`node --version` and `npm --version` then failed with "command not found"
(exit 127), failing both devcontainer jobs before devcontainers/ci even
ran. Prepend the extracted bin dir to PATH so the version checks resolve
the freshly installed binaries.

Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant