How can I run VLLM serving without an internet connection?

[ZPerling]

I cloned the model repository on Hugging Face to my local machine and used the --download-dir parameter to specify the directory. However, when running VLLM, it still tries to connect to Hugging Face, which doesn't work without an internet connection. Even after setting export HF_HUB_OFFLINE=1, offline mode doesn't seem to be working. Is it possible to run VLLM offline and if so, how can I achieve this?

[MarkoMFilip]

We're facing the same issue, any feedback here would be appreciated. Running vLLM online is currently not possible for us...

[Martyprod]

same issue, i'm offline 99% of the time and VLLM is not working offline :(.

[Moran232]

can you show the command you run vlllm?

[Martyprod]

I'm running it in Stability matrix, so can it run totally offline ? I can install it locally if the issue is related to stability matrix itself !
?

[HYZ17]

Using absolute path to load the model, rather than the name on huggingface, works for me. For example,

import os 
os.environ["CUDA_VISIBLE_DEVICES"] = "5"
from vllm import LLM

prompts = ["Hello, my name is", "The capital of France is"]  # Sample prompts.
#llm = LLM(model="/home/.cache/huggingface/hub/models--huggyllama--llama-13b/snapshots/bf57045473f207bb1de1ed035ace226f4d9f9bba",)
outputs = llm.generate(prompts)  # Generate texts from the prompts.
print(outputs)

[llmwesee]

It works but the generated responses are wrong most of the time.

[smahdavi4]

Setting the hugging face transformers module to offline mode (via a global environment variable) worked for me: TRANSFORMERS_OFFLINE=1

[juud79]

same...

[bzr1]

this is not working

[yw0nam]

It works in vllm 0.5.5 with docker. Thanks.

[nitsan-airis]

hey, 0.6.2 version here,

docker run --gpus all -p 8005:8005 -v /abs/path/to/cached/model:/app/model_data --env "TRANSFORMERS_OFFLINE=1" vllm/vllm-openai:v0.6.2 --model meta-llama/Llama-3.1-8B --port 8005 --download_dir /app/model_data

did not work. Specifically got
huggingface_hub.errors.OfflineModeIsEnabled: Cannot reach https://huggingface.co/meta-llama/Llama-3.1-8B/resolve/main/config.json: offline mode is enabled. To disable it, please unset the HF_HUB_OFFLINE environment variable.

[googlercolin]

Hi, I'm trying to run my DeepSeek distilled model (which I saved the weights in /home/users/ntu/hong0259/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-14B) offline, but it doesn't work.

I ran:
TRANSFORMERS_OFFLINE=1 HF_HOME=/home/users/ntu/hong0259/.cache/huggingface vllm serve "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"

But met with this error, how can I fix it?

INFO 02-21 16:53:20 __init__.py:183] Automatically detected platform cuda.
INFO 02-21 16:53:25 api_server.py:835] vLLM API server version 0.7.0
INFO 02-21 16:53:25 api_server.py:836] args: Namespace(subparser='serve', model_tag='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B', config='', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x15127683d300>)
WARNING 02-21 16:53:25 utils.py:2069] Found ulimit of 16384 and failed to automatically increasewith error current limit exceeds maximum limit. This can cause fd limit errors like`OSError: [Errno 24] Too many open files`. Consider increasing with ulimit -n
INFO 02-21 16:53:25 api_server.py:203] Started engine process with PID 2943603
INFO 02-21 16:53:46 __init__.py:183] Automatically detected platform cuda.
INFO 02-21 16:53:53 config.py:520] This model supports multiple tasks: {'embed', 'classify', 'generate', 'reward', 'score'}. Defaulting to 'generate'.
WARNING 02-21 16:53:53 arg_utils.py:1107] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
INFO 02-21 16:53:53 config.py:1483] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 02-21 16:54:01 config.py:520] This model supports multiple tasks: {'embed', 'generate', 'classify', 'score', 'reward'}. Defaulting to 'generate'.
WARNING 02-21 16:54:01 arg_utils.py:1107] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
INFO 02-21 16:54:01 config.py:1483] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 02-21 16:54:01 llm_engine.py:232] Initializing an LLM engine (v0.7.0) with config: model='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=deepseek-ai/DeepSeek-R1-Distill-Qwen-14B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
ERROR 02-21 16:54:02 engine.py:387] invalid literal for int() with base 10: 'GPU-64642a09-f927-755a-07cb-3ee05fd5ceb2'
ERROR 02-21 16:54:02 engine.py:387] Traceback (most recent call last):
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
ERROR 02-21 16:54:02 engine.py:387]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 02-21 16:54:02 engine.py:387]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
ERROR 02-21 16:54:02 engine.py:387]     return cls(ipc_path=ipc_path,
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 73, in __init__
ERROR 02-21 16:54:02 engine.py:387]     self.engine = LLMEngine(*args, **kwargs)
ERROR 02-21 16:54:02 engine.py:387]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/engine/llm_engine.py", line 271, in __init__
ERROR 02-21 16:54:02 engine.py:387]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 02-21 16:54:02 engine.py:387]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/executor/executor_base.py", line 49, in __init__
ERROR 02-21 16:54:02 engine.py:387]     self._init_executor()
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/executor/uniproc_executor.py", line 38, in _init_executor
ERROR 02-21 16:54:02 engine.py:387]     self.collective_rpc("init_worker", args=([kwargs], ))
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/executor/uniproc_executor.py", line 49, in collective_rpc
ERROR 02-21 16:54:02 engine.py:387]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 02-21 16:54:02 engine.py:387]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/utils.py", line 2208, in run_method
ERROR 02-21 16:54:02 engine.py:387]     return func(*args, **kwargs)
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/worker/worker_base.py", line 549, in init_worker
ERROR 02-21 16:54:02 engine.py:387]     self.worker = worker_class(**kwargs)
ERROR 02-21 16:54:02 engine.py:387]                   ^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/worker/worker.py", line 81, in __init__
ERROR 02-21 16:54:02 engine.py:387]     self.model_runner: GPUModelRunnerBase = ModelRunnerClass(
ERROR 02-21 16:54:02 engine.py:387]                                             ^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/worker/model_runner.py", line 1063, in __init__
ERROR 02-21 16:54:02 engine.py:387]     self.attn_backend = get_attn_backend(
ERROR 02-21 16:54:02 engine.py:387]                         ^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/attention/selector.py", line 92, in get_attn_backend
ERROR 02-21 16:54:02 engine.py:387]     return _cached_get_attn_backend(
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/attention/selector.py", line 143, in _cached_get_attn_backend
ERROR 02-21 16:54:02 engine.py:387]     attention_cls = current_platform.get_attn_backend_cls(
ERROR 02-21 16:54:02 engine.py:387]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 172, in get_attn_backend_cls
ERROR 02-21 16:54:02 engine.py:387]     if not cls.has_device_capability(80):
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 68, in wrapper
ERROR 02-21 16:54:02 engine.py:387]     return fn(*args, **kwargs)
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 262, in has_device_capability
ERROR 02-21 16:54:02 engine.py:387]     return super().has_device_capability(capability, device_id)
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/interface.py", line 168, in has_device_capability
ERROR 02-21 16:54:02 engine.py:387]     current_capability = cls.get_device_capability(device_id=device_id)
ERROR 02-21 16:54:02 engine.py:387]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 68, in wrapper
ERROR 02-21 16:54:02 engine.py:387]     return fn(*args, **kwargs)
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 246, in get_device_capability
ERROR 02-21 16:54:02 engine.py:387]     physical_device_id = device_id_to_physical_device_id(device_id)
ERROR 02-21 16:54:02 engine.py:387]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387]   File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 57, in device_id_to_physical_device_id
ERROR 02-21 16:54:02 engine.py:387]     return int(physical_device_id)
ERROR 02-21 16:54:02 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-21 16:54:02 engine.py:387] ValueError: invalid literal for int() with base 10: 'GPU-64642a09-f927-755a-07cb-3ee05fd5ceb2'
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 389, in run_mp_engine
    raise e
  File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
    return cls(ipc_path=ipc_path,
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/engine/multiprocessing/engine.py", line 73, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/engine/llm_engine.py", line 271, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/executor/executor_base.py", line 49, in __init__
    self._init_executor()
  File "/home/users/ntu/hong0259/vllm/vllm/executor/uniproc_executor.py", line 38, in _init_executor
    self.collective_rpc("init_worker", args=([kwargs], ))
  File "/home/users/ntu/hong0259/vllm/vllm/executor/uniproc_executor.py", line 49, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/utils.py", line 2208, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/worker/worker_base.py", line 549, in init_worker
    self.worker = worker_class(**kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/worker/worker.py", line 81, in __init__
    self.model_runner: GPUModelRunnerBase = ModelRunnerClass(
                                            ^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/worker/model_runner.py", line 1063, in __init__
    self.attn_backend = get_attn_backend(
                        ^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/attention/selector.py", line 92, in get_attn_backend
    return _cached_get_attn_backend(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/attention/selector.py", line 143, in _cached_get_attn_backend
    attention_cls = current_platform.get_attn_backend_cls(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 172, in get_attn_backend_cls
    if not cls.has_device_capability(80):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 68, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 262, in has_device_capability
    return super().has_device_capability(capability, device_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/interface.py", line 168, in has_device_capability
    current_capability = cls.get_device_capability(device_id=device_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 68, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 246, in get_device_capability
    physical_device_id = device_id_to_physical_device_id(device_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/platforms/cuda.py", line 57, in device_id_to_physical_device_id
    return int(physical_device_id)
           ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'GPU-64642a09-f927-755a-07cb-3ee05fd5ceb2'
Traceback (most recent call last):
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/home/users/ntu/hong0259/vllm/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/entrypoints/openai/api_server.py", line 863, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/entrypoints/openai/api_server.py", line 133, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/miniconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/ntu/hong0259/vllm/vllm/entrypoints/openai/api_server.py", line 227, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

[sancelot]

download the model with this small script :

from huggingface_hub import snapshot_download

model_id="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"
model_path = snapshot_download(
    repo_id=model_id,
    local_dir="./models/"+model_id,
    max_workers=4  # Increase for faster parallel downloads
)

and then load it using
model = LLM(model=./models/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B)

[googlercolin]

Hmm, I previously downloaded the model in another machine and rsync-ed the huggingface files over to my remote machine (in /home/users/ntu/hong0259/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-14B) which has no internet connection.

However, I'm unable to load the the model via vllm serve.