Clarify usage of model-streamer with s3 compatible object-storage

[strigazi]

I am trying to create a dgd to run kimi-2.6 getting the model weights from our s3 object storage backed by Ceph's rados-gw.
The flows we want to test are:

1. S3 -> work pods
2. S3 -> first worker pod -> P2P for subsequent pods

While investigating 1. I based an example on: https://github.com/ai-dynamo/modelexpress/blob/main/examples/model_streamer_k8s/client/vllm/vllm-single-node-streamer-s3.yaml , I'm attaching the manifest.

It works as expected. With verbose log level for RUNAI_STREAMER_* I can see weights being fetched from s3.

s3-vllm-example.yaml.txt

However, when I use modelexpress with a dynamo image that has rust based client and following:

• https://github.com/ai-dynamo/dynamo/blob/main/docs/kubernetes/model-caching.md#modelstreamer-from-object-storage
• https://github.com/ai-dynamo/dynamo/blob/main/docs/kubernetes/modelexpress.md#stream-without-shared-storage
  downloading falls back to huggingface.

example configuration:

MX_MODEL_URI=s3://dynamo-models/moonshotai/Kimi-K2.6
# I tried a specific snapshot as well 
# MX_MODEL_URI=s3://dynamo-models/moonshotai/Kimi-K2.6/snapshots/7eb5002f6aadc958aed6a9177b7ed26bb94011bb
HF_HOME=/opt/verda/huggingface-cache/
VLLM_PLUGINS=modelexpress
HF_HUB_OFFLINE=0
MODEL_EXPRESS_URL=http://modelexpress-ml-staging.ml-modelexpress.svc.cluster.local:8001/
HF_MODULES_CACHE=/tmp/hf_modules
MODEL_EXPRESS_NO_SHARED_STORAGE=1

command: python3 -m dynamo.vllm --model=moonshotai/Kimi-K2.6 --load-format modelexpress --trust-remote-code --tensor-parallel-size 8 --dyn-tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --mm-encoder-tp-mode data

Output:

click to collapse

2026-07-09T11:15:27.229932Z DEBUG init.load_plugins_by_group: Available plugins for group vllm.general_plugins:
2026-07-09T11:15:27.230440Z DEBUG init.load_plugins_by_group: - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2026-07-09T11:15:27.230467Z DEBUG init.load_plugins_by_group: - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
2026-07-09T11:15:27.239538Z INFO args.update_engine_config_with_dynamo: Using kv_events_config for publishing vLLM kv events over zmq: None (use_kv_events=False)
2026-07-09T11:15:27.239634Z INFO args.update_engine_config_with_dynamo: Forward pass metrics enabled: scheduler_cls set to InstrumentedScheduler (port=20380)
2026-07-09T11:15:28.388333Z INFO dynamo_llm::hub: Successfully connected to ModelExpress server
2026-07-09T11:15:28.388405Z INFO modelexpress_client: Requesting model: moonshotai/Kimi-K2.6 from provider: HuggingFace
2026-07-09T11:15:28.398566Z INFO modelexpress_client: Model moonshotai/Kimi-K2.6: Model already downloaded
2026-07-09T11:15:28.398597Z INFO modelexpress_client: Model moonshotai/Kimi-K2.6 is now available
2026-07-09T11:15:28.398604Z INFO modelexpress_client: Model moonshotai/Kimi-K2.6 downloaded successfully via server
2026-07-09T11:15:28.398609Z INFO modelexpress_client: Shared storage disabled, streaming files from server for model moonshotai/Kimi-K2.6
2026-07-09T11:15:28.398615Z INFO modelexpress_client: Streaming model moonshotai/Kimi-K2.6 files to "/home/dynamo/.model-express/cache" with chunk size 32768 bytes
2026-07-09T11:15:28.401839Z WARN dynamo_llm::hub: Server download failed for model 'moonshotai/Kimi-K2.6': gRPC error: status: NotFound, message: "Model not found: Model snapshots for 'moonshotai/Kimi-K2.6' not found in cache", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 09 Jul 2026 11:15:28 GMT"} }. Falling back to direct download.
2026-07-09T11:15:28.401896Z INFO modelexpress_common::download: Downloading model 'moonshotai/Kimi-K2.6' using provider: Hugging Face
2026-07-09T11:15:28.402042Z INFO modelexpress_common::providers::huggingface: Using cache directory: "/opt/verda/huggingface-cache/hub"
LICENSE [00:00:00] [██████████████████████████████████████████████████████████████████████████████████████] 1.43 KiB/1.43 KiB 7.86 KiB/s (0s)TH

[athreesh]

thanks @strigazi - we're taking a look to replicate