diff --git a/docs/hooks/nav_titles.py b/docs/hooks/nav_titles.py index 369ec175e222..2ce9be045b92 100644 --- a/docs/hooks/nav_titles.py +++ b/docs/hooks/nav_titles.py @@ -185,11 +185,13 @@ "tutorials/models/Step-5-Preview.md": {"en": "Step-5-Preview (Experimental)", "zh": "Step-5-Preview(实验性支持)"}, "tutorials/models/gpt-oss-120b.md": {"en": "gpt-oss-120b", "zh": "gpt-oss-120b"}, "user_guide/configuration/additional_config.md": {"en": "Additional Configuration", "zh": "附加配置"}, + "user_guide/configuration/device_selection.md": {"en": "NPU Device Selection", "zh": "NPU 设备选择"}, "user_guide/configuration/encoder_cache_manager.md": { "en": "Score Encoder Cache Manager", "zh": "Score 编码器缓存管理器", }, "user_guide/configuration/env_vars.md": {"en": "Environment Variables", "zh": "环境变量"}, + "user_guide/configuration/runtime_config.md": {"en": "Runtime Configuration", "zh": "运行时配置"}, "user_guide/deployment_guide/using_mindie_motor.md": {"en": "Using MindIE Motor", "zh": "使用 MindIE Motor"}, "user_guide/deployment_guide/using_volcano_kthena.md": {"en": "Using Volcano Kthena", "zh": "使用 Volcano Kthena"}, "user_guide/feature_guide/Ai_QoS_introduction_en.md": {"en": "AI QoS", "zh": "AI QoS"}, diff --git a/docs/source/locale/zh_CN/LC_MESSAGES/user_guide/configuration/runtime_config.po b/docs/source/locale/zh_CN/LC_MESSAGES/user_guide/configuration/runtime_config.po index 690f6ead3ebc..a3f986e7e81f 100644 --- a/docs/source/locale/zh_CN/LC_MESSAGES/user_guide/configuration/runtime_config.po +++ b/docs/source/locale/zh_CN/LC_MESSAGES/user_guide/configuration/runtime_config.po @@ -18,8 +18,8 @@ msgstr "" "Plural-Forms: nplurals=1; plural=0;\n" "Generated-By: detect_po_changes.py\n" -msgid "# runtime_config" -msgstr "# runtime_config" +msgid "# Runtime Configuration" +msgstr "# 运行时配置" msgid "" "JSON schema for Runtime Guard. Default path: " diff --git a/docs/source/tutorials/features/pd_disaggregation_mooncake_single_node.md b/docs/source/tutorials/features/pd_disaggregation_mooncake_single_node.md index 88df07e3bd95..9f888f33aadf 100644 --- a/docs/source/tutorials/features/pd_disaggregation_mooncake_single_node.md +++ b/docs/source/tutorials/features/pd_disaggregation_mooncake_single_node.md @@ -8,6 +8,10 @@ Using the Qwen2.5-VL-7B-Instruct model as an example, use vLLM-Ascend {{vllm_asc For P/D transfer priority settings, see [QoS Configuration](pd_disaggregation_mooncake_multi_node.md#qos-configuration). +For disjoint P/D device assignments and the interaction between +`ASCEND_RT_VISIBLE_DEVICES` and `--device-ids`, see +[NPU Device Selection](../../user_guide/configuration/device_selection.md#prefill-decode-disaggregation). + ## Verify Communication Environment ### Verification Process diff --git a/docs/source/tutorials/features/ray.md b/docs/source/tutorials/features/ray.md index e0362afeddc9..463e879bbf72 100644 --- a/docs/source/tutorials/features/ray.md +++ b/docs/source/tutorials/features/ray.md @@ -103,6 +103,11 @@ Choose one machine as the head node and the others as worker nodes. Before proce Set the `ASCEND_RT_VISIBLE_DEVICES` environment variable to specify the NPU devices to use. For Ray versions above 2.1, also set the `RAY_EXPERIMENTAL_NOSET_ASCEND_RT_VISIBLE_DEVICES` variable to avoid device recognition issues. +`--device-ids` does not select devices with the Ray executor; vLLM prints a +warning and Ray placement groups control resource assignment. See +[NPU Device Selection](../../user_guide/configuration/device_selection.md#ray) +for the difference from multiprocessing launches. + Below are the commands for the head and worker nodes: **Head node**: diff --git a/docs/source/user_guide/configuration/device_selection.md b/docs/source/user_guide/configuration/device_selection.md new file mode 100644 index 000000000000..fef0f696e57f --- /dev/null +++ b/docs/source/user_guide/configuration/device_selection.md @@ -0,0 +1,127 @@ +# NPU Device Selection + +Use `ASCEND_RT_VISIBLE_DEVICES` to restrict the NPUs visible to a process, and +`--device-ids` to select devices for a vLLM multiprocessing executor. Setting +`--tensor-parallel-size` controls the number of workers; it does not select +particular NPUs. + +The examples below use integer device IDs, a single node, and the `mp` executor. +Use matching vLLM and vLLM Ascend versions from the +[installation guide](../../getting_started/installation.md). Ray has a different selection +mechanism; see [Ray](#ray) below. + +## How the two settings interact + +- Set `ASCEND_RT_VISIBLE_DEVICES` **before** starting vLLM. It remains supported; + explicit device selection does not require removing this environment variable. +- If the variable is unset, integer `--device-ids` values select NPU IDs directly + in the current environment. +- If the variable is set, integer `--device-ids` values are **zero-based indices + into that visible list**, in the specified order. They are not a second list + of host device IDs. +- If `--device-ids` is omitted, vLLM allocates workers from the available visible + devices according to the parallel configuration. + +For TP2 on a machine exposing IDs `0` through `7`: + +| `ASCEND_RT_VISIBLE_DEVICES` | `--device-ids` | Selected NPU IDs | Worker runtime devices | +| --- | --- | --- | --- | +| Unset | Omitted | `0,1` | `npu:0`, `npu:1` | +| Unset | `1,3` | `1,3` | `npu:1`, `npu:3` | +| `4,5,6,7` | Omitted | `4,5` | `npu:0`, `npu:1` | +| `4,5,6,7` | `1,3` | `5,7` | `npu:1`, `npu:3` | +| `4,5,6,7` | `3,1` | `7,5` | `npu:3`, `npu:1` | + +For example, `ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 --device-ids 3,1` selects entries +`7,5`, not devices `3,1`. Conversely, `ASCEND_RT_VISIBLE_DEVICES=1,3` combined +with `--device-ids 1,3` is invalid: index `3` is outside a two-device list. + +The runtime device number is relative to the process's visibility, so `npu:0` +does not necessarily mean host device `0`. Container device mounts can further +restrict or remap devices; use IDs available inside the container. Neither +setting exposes devices that the container has not been granted. + +## Tensor parallelism + +Select two non-contiguous devices without a visibility override: + +```bash +unset ASCEND_RT_VISIBLE_DEVICES +vllm serve Qwen/Qwen3-8B \ + --distributed-executor-backend mp \ + --tensor-parallel-size 2 \ + --device-ids 1,3 +``` + +Alternatively, restrict the visible set and omit `--device-ids`: + +```bash +ASCEND_RT_VISIBLE_DEVICES=1,3 vllm serve Qwen/Qwen3-8B \ + --distributed-executor-backend mp \ + --tensor-parallel-size 2 +``` + +Run these as alternatives, not simultaneously on the same devices. Adjust the +model and memory settings for your hardware. + +## Data parallelism + +For a single launch with two local DP replicas, each using TP2, provide the full +four-device list. vLLM splits it between the replicas; do not pass only one +replica's two-device slice to this launch. + +```bash +unset ASCEND_RT_VISIBLE_DEVICES +vllm serve Qwen/Qwen3-8B \ + --distributed-executor-backend mp \ + --data-parallel-size 2 \ + --data-parallel-size-local 2 \ + --tensor-parallel-size 2 \ + --device-ids 1,3,5,7 +``` + +Replica 0 uses `1,3`; replica 1 uses `5,7`. For this single-node DP/TP layout, +allocate `DP × TP` distinct devices. An equivalent visibility-only launch uses +`ASCEND_RT_VISIBLE_DEVICES=1,3,5,7` and omits `--device-ids`. + +Multi-node or separately launched DP ranks have different launch arguments and +local device allocations. Follow the model's deployment tutorial rather than +copying this single-launch layout unchanged. + +## Prefill-decode disaggregation + +P and D are separate vLLM instances. On the same host, assign them disjoint +device sets and different service/connector ports. For example, with TP2 on +each side: + +| Instance | Visibility-only selection | Alternative with visibility unset | +| --- | --- | --- | +| Prefill | `ASCEND_RT_VISIBLE_DEVICES=0,2`, omit `--device-ids` | `--device-ids 0,2` | +| Decode | `ASCEND_RT_VISIBLE_DEVICES=4,6`, omit `--device-ids` | `--device-ids 4,6` | + +These are device assignments, not complete PD launch commands. Keep the model, +KV connector, proxy, and communication configuration from the +[single-node PD tutorial](../../tutorials/features/pd_disaggregation_mooncake_single_node.md) +or the [multi-node PD tutorial](../../tutorials/features/pd_disaggregation_mooncake_multi_node.md). +On different hosts, the same numeric device IDs can refer to different NPUs. + +## Ray + +`--device-ids` has no effect with `--distributed-executor-backend ray`; vLLM +prints a warning. Ray placement groups determine which resources workers use. +Set each node's `ASCEND_RT_VISIBLE_DEVICES` before starting its Ray processes, +and follow the [Ray tutorial](../../tutorials/features/ray.md) for cluster and +environment configuration. Changing the variable in a later client shell does +not reconfigure an already running Ray cluster. + +## Check the allocation + +- Use distinct integer indices within the visible-list bounds. Duplicate IDs + and out-of-range indices are rejected. +- Supply enough devices for the local parallel layout. For example, TP2 with + `--device-ids 1` cannot start two workers. +- Check worker startup logs and the `npu-smi info` process table in the relevant + host/container environment. Account for visibility remapping when comparing + runtime device numbers with the selected NPUs. +- A successful HTTP response alone does not establish correct device placement. + For DP and PD, verify every replica or instance, not just the API server. diff --git a/docs/source/user_guide/configuration/env_vars.md b/docs/source/user_guide/configuration/env_vars.md index ba863791c73a..058e327e14b7 100644 --- a/docs/source/user_guide/configuration/env_vars.md +++ b/docs/source/user_guide/configuration/env_vars.md @@ -2,6 +2,10 @@ vllm-ascend uses the following environment variables to configure the system: +For `ASCEND_RT_VISIBLE_DEVICES` and its interaction with `--device-ids`, see +[NPU Device Selection](device_selection.md). The visibility variable is a runtime +setting and remains supported alongside explicit device selection. + **Note:** Some environment variables are being migrated to `--additional-config` options. These environment variables are still supported during the migration period, and it is recommended to use `--additional-config` for new deployments. See [Additional Configuration](additional_config.md) for details. {{ include_code('vllm_ascend/envs.py', start_after='begin-env-vars-definition', end_before='end-env-vars-definition') }} diff --git a/docs/source/user_guide/configuration/index.md b/docs/source/user_guide/configuration/index.md index 88b4551d74c5..53b0e7b41bee 100644 --- a/docs/source/user_guide/configuration/index.md +++ b/docs/source/user_guide/configuration/index.md @@ -1,3 +1,6 @@ # Configuration Guide This section provides a detailed configuration guide of vLLM Ascend. + +See [NPU Device Selection](device_selection.md) for `--device-ids`, its +interaction with `ASCEND_RT_VISIBLE_DEVICES`, and DP, PD, and Ray examples. diff --git a/docs/source/user_guide/configuration/runtime_config.md b/docs/source/user_guide/configuration/runtime_config.md index dc45f7561983..9329d2d7bf49 100644 --- a/docs/source/user_guide/configuration/runtime_config.md +++ b/docs/source/user_guide/configuration/runtime_config.md @@ -1,4 +1,4 @@ -# runtime_config +# Runtime Configuration JSON schema for Runtime Guard. Default path: `/runtime/config/runtime_config.json`. diff --git a/mkdocs.yml b/mkdocs.yml index 0b0d98d9dee7..152c2fdf755b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -294,6 +294,7 @@ nav: - user_guide/support_matrix/feature_matrix.md - Configuration: - user_guide/configuration/index.md + - user_guide/configuration/device_selection.md - user_guide/configuration/env_vars.md - user_guide/configuration/additional_config.md - user_guide/configuration/runtime_config.md