Repository navigation
[CI][DevContainer] Add devcontainer configs for CPU / NPU A3 / NPU A2 and CI consistency workflow - #16891
[CI][DevContainer] Add devcontainer configs for CPU / NPU A3 / NPU A2 and CI consistency workflow#16891Yunme wants to merge 18 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the development experience by providing standardized devcontainer configurations that mirror the project's CI gate pipeline. By automating environment setup and NPU device selection, developers can now reliably reproduce test failures and validate changes locally before pushing to the repository. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
There was a problem hiding this comment.
Code Review
Suggested PR Title:
[CI][Feature] Add Dev Container configurations for CPU and NPU (A2/A3) environmentsSuggested PR Summary:
### What this PR does / why we need it?
This PR adds VS Code Dev Container configurations for CPU, NPU A2 (single card), and NPU A3 (dual card) environments to simplify local reproduction of CI gate tests. It also includes helper scripts (`select_npu.py` and `activate_npu.sh`) to automatically detect and assign free NPU cards.
Feedback on the PR:
- In `.devcontainer/select_npu.py`, `subprocess.run` should specify `check=True` to prevent silent failures when running `npu-smi info`.
- In both `.devcontainer/a2/postCreateCommand.sh` and `.devcontainer/npu/postCreateCommand.sh`, the `MAX_JOBS` environment variable is hardcoded (to 23 and 46), which can cause Out-Of-Memory (OOM) errors on machines with fewer resources. It is recommended to dynamically set this using `nproc`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The configurations replicate the environment and steps used in `.github/workflows/pr_test.yaml`.| out = subprocess.run( | ||
| ["npu-smi", "info"], capture_output=True, text=True, timeout=30, | ||
| ).stdout |
There was a problem hiding this comment.
The subprocess.run call does not specify check=True. If npu-smi info fails with a non-zero exit code (e.g., due to driver initialization issues or permission errors), subprocess.run will not raise an exception. Instead, it will return a CompletedProcess object with an empty stdout, causing the script to silently conclude that there are no free NPU cards available instead of reporting the actual error. Adding check=True ensures that any non-zero exit code raises a CalledProcessError, which will be properly caught and reported by the except block.
| out = subprocess.run( | |
| ["npu-smi", "info"], capture_output=True, text=True, timeout=30, | |
| ).stdout | |
| res = subprocess.run( | |
| ["npu-smi", "info"], capture_output=True, text=True, timeout=30, check=True | |
| ) | |
| out = res.stdout |
| cd "${PROJECT_DIR}" | ||
| uv pip install -r requirements-dev.txt | ||
| uv pip install --force-reinstall --no-deps triton-ascend==3.2.2 | ||
| export MAX_JOBS=23 |
There was a problem hiding this comment.
Hardcoding MAX_JOBS=23 assumes the host machine has at least 24 cores and sufficient memory to support 23 parallel compilation jobs. On developer machines with fewer cores or limited RAM, this can easily trigger Out-Of-Memory (OOM) crashes during kernel compilation. It is highly recommended to dynamically determine the number of jobs based on the available CPU cores, while allowing an override via environment variables.
| export MAX_JOBS=23 | |
| export MAX_JOBS=${MAX_JOBS:-$(nproc)} |
| uv pip install --force-reinstall --no-deps triton-ascend==3.2.2 | ||
|
|
||
| echo "[5/6] 带设备编译安装 vllm-ascend(编译 NPU 自定义内核)" | ||
| export MAX_JOBS=46 |
There was a problem hiding this comment.
Hardcoding MAX_JOBS=46 assumes the host machine has at least 48 cores and sufficient memory to support 46 parallel compilation jobs. On developer machines with fewer cores or limited RAM, this can easily trigger Out-Of-Memory (OOM) crashes during kernel compilation. It is highly recommended to dynamically determine the number of jobs based on the available CPU cores, while allowing an override via environment variables.
| export MAX_JOBS=46 | |
| export MAX_JOBS=${MAX_JOBS:-$(nproc)} |
Adds .devcontainer setup with an all-in-one vllm-ascend image to reproduce the pr_test.yaml pre-commit and cpu-ut gate checks locally. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Split devcontainer into cpu/npu configs and add a workflow_dispatch workflow that uses devcontainers/ci to run each config's postCreateCommand, validating that CI and devcontainer environments match for cpu-ut and a3 two-card device tests. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
devcontainer.json mounts ${localEnv:HOME}/.cache, which does not exist
on GitHub hosted runners, causing docker bind mount to fail. Create it
before running devcontainers/ci.
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
…eateCommand Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
nightly-main image bundles vllm v0.28.0 tag, whose kv_cache_interface (KVCacheTensor.shared_by) and attn_utils (_allocate_kv_cache) differ from the main commit pinned in .github/vllm-main-verified.commit (layers / allocate_kv_cache), failing 7 kvpp/attention cpu-ut tests. Check out the gate pin in /vllm-workspace/vllm and rebuild with VLLM_TARGET_DEVICE=empty before installing vllm-ascend, matching the 'Install vllm-project/vllm from source' step in _selected_tests.yaml. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
…ateCommand The /vllm-workspace/vllm checkout inside the nightly-main image has its origin remote pointing at the in-house gh-proxy.test.osinfra.cn mirror, which GitHub hosted runners cannot reach (HTTP 418). Fetch the pinned vllm commit from the official github.com URL instead, bypassing the broken origin config. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The amd64 nightly-main image bakes url."https://gh-proxy.test.osinfra.cn/https://github.com/".insteadOf "https://github.com/" into /root/.gitconfig, which makes git rewrite github.com fetch URLs to the unreachable in-house proxy (HTTP 418) on GitHub hosted runners. Remove that url section before fetching the pinned vllm commit. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The devcontainer CI consistency workflow now validates exclusively on GitHub Actions, so drop the WSL2 squid-proxy fallback (USE_HOST_PROXY, host.docker.internal:3128) from cpu postCreateCommand and stop passing the now-unused USE_HOST_PROXY env into both devcontainers/ci jobs. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Mirror the existing npu (A3) devcontainer for the A2 910B single-card runner so the PR gate a2-1 partition (one_card tests) can be reproduced locally. Uses the nightly-main all-in-one image and reinstalls vllm from the verified pin before compiling vllm-ascend, matching the cpu/npu postCreateCommand scripts. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Align the device-specific devcontainer directory names with the other targets (cpu / a2) by renaming npu -> a3, which matches the chip family (A3) rather than the generic "npu" label. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Add the same pull_request trigger as pr_test.yaml (opened/synchronize/ reopened/labeled on main and *-dev/releases/v* branches) while keeping manual workflow_dispatch. On PR events both cpu and npu jobs run with the npu job pinned to the default linux-aarch64-a3-2 label; dispatch still selects targets via inputs. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The a3 job runs on a self-hosted runner without npm, so devcontainers/ci fails with `npm: command not found` when installing @devcontainers/cli. Add actions/setup-node to the npu job before the devcontainer build step. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
setup-node downloads from nodejs.org/GitHub releases, which the self-hosted a3 runner cannot reach, so the job hangs on download. Install a pinned node arm64 tarball from repo.huaweicloud.com, matching the schedule_main2main.yaml pattern. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The a3 self-hosted runner job runs as a non-root user, so mkdir /opt/nodejs fails with permission denied. Install the node tarball under $HOME/nodejs and prepend its bin dir to PATH via GITHUB_PATH. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
Switch the cpu job from ubuntu-latest to the repo's linux-amd64-cpu-8-hk runner so both jobs use vllm-ascend self-hosted runners. Make the node install arch-aware (x64/arm64) and reuse it in the cpu job. Signed-off-by: vllm-ascend-ci <vllm-ascend-ci@users.noreply.github.com> Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
pre-commit fails with UP031 (% format) on the error/status prints and a forbidden `import re`, and ruff-format reformats the file. Replace `re` with the repo-wide `regex as re` convention and percent formatting with f-strings so the gate lint passes. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
The node install step extracted the tarball and appended its bin dir to GITHUB_PATH, which only propagates to subsequent steps. The same-step `node --version` and `npm --version` then failed with "command not found" (exit 127), failing both devcontainer jobs before devcontainers/ci even ran. Prepend the extracted bin dir to PATH so the version checks resolve the freshly installed binaries. Signed-off-by: zhangzhiqiang90 <zhangzhiqiang90@h-partners.com>
What this PR does / why we need it?
Adds devcontainer development environments that reproduce the PR gate pipeline locally:
.devcontainer/cpu— CPU unit tests (cpu-ut stage).devcontainer/npu— A3 two-card device tests (a3-2 partition).devcontainer/a2— A2 910B one-card device tests (a2-1 partition).devcontainer/activate_npu.sh+select_npu.py— free-card detection.github/workflows/devcontainer_ci_consistency.yaml— CI consistency check between gate and devcontainerDoes this PR introduce any user-facing change?
No.
How was this patch tested?
Local devcontainer bring-up; CI workflow verification pending merge to base branch (GitHub does not run workflow changes from a PR).