Manifesto renders shareable Kubernetes manifests for LLM deployments.
It is built for the workflow where an engineer wants to describe a concrete vLLM deployment, render plain YAML, inspect or hand-edit it, and then apply exactly that artifact to a cluster.
Manifesto keeps intelligence in resolution, not in the artifact: model specs express intent, feature contracts supply required runtime details, and the result remains ordinary, minimal Kubernetes YAML.
The project is packaged as llm-manifesto, with manifesto as the Python
package and CLI command name.
Manifesto takes two YAML inputs:
- a model spec: model image, topology, roles, parallelism, routing, vLLM args
- a cluster profile: GPU shape, storage paths, fabric env, llm-d images
It emits raw Kubernetes manifests:
- Deployment or LeaderWorkerSet model-server workloads, depending on node count
- InferencePool and endpoint picker deployment
- standalone Envoy routing by default, or optional Gateway API objects
- per-pod monitoring sidecars
- instance-scoped names, labels, selectors, and cache paths
The rendered YAML starts with provenance comments:
# Generated by:
# uvx --from git+https://github.com/neuralmagic/llm-manifesto@<commit> manifesto render manifest \
# <(cat <<'MANIFESTO_MODEL'
# <expanded model YAML>
# MANIFESTO_MODEL
# ) \
# --cluster <(cat <<'MANIFESTO_CLUSTER'
# <expanded cluster YAML>
# MANIFESTO_CLUSTER
# )
# To set identity or placement, append --user USER and/or --namespace NAMESPACE.
# Source: https://github.com/neuralmagic/llm-manifesto/tree/<commit>
# Safe to edit before applying.
---
apiVersion: v1
kind: ServiceAccount
...That header is intentional: rendered manifests should be easy to paste into a PR, send to another engineer, edit manually, and regenerate later.
Licensed under the Apache License, Version 2.0. See LICENSE.
Bootstrap namespace prerequisites declared by the cluster profile (such as a shared cache PVC):
manifesto render bootstrap --cluster clusters/example-gb200.yaml --namespace "$MANIFESTO_NAMESPACE"
manifesto deploy bootstrap --cluster clusters/example-gb200.yaml --namespace "$MANIFESTO_NAMESPACE"Render a manifest:
uv run manifesto render manifest models/qwen/aggregated.yaml \
--cluster clusters/example-gb200.yaml \
--user "$USER"Render to a file, edit it, diff it, then apply it:
manifesto render file models/deepseek-v4/1P-EP8-1D-EP8.yaml
$EDITOR /tmp/manifesto.yaml
manifesto file diff
manifesto file applyDeploy directly:
manifesto deploy models/deepseek-v4/1P-EP8-1D-EP8.yaml
manifesto ready models/deepseek-v4/1P-EP8-1D-EP8.yamlStop the instance:
manifesto servers
manifesto stop # interactive picker
manifesto stop models/deepseek-v4/1P-EP8-1D-EP8.yaml
manifesto stop --instance "$USER-wide-ep-1p-ep8-1d-ep8"
manifesto stop models/deepseek-v4/1P-EP8-1D-EP8.yaml --nowmanifesto/ Python renderer implementation
cli.py CLI entrypoints
spec.py model/deployment spec schema
cluster.py cluster profile schema
instance.py instance-scoped names, labels, selectors, paths
resolve.py concrete env, paths, ports, and vLLM args per role
render/ Kubernetes object emitters
clusters/ cluster profiles
models/ model deployment specs
tests/ renderer and UX tests
Manifesto also reads a private user catalog from
~/.config/llm-manifesto (or $XDG_CONFIG_HOME/llm-manifesto). Keep local
cluster and model configuration outside the repository with this layout:
~/.config/llm-manifesto/
.env optional local defaults
clusters/
my-context.yaml
models/
model_provider/model.yaml
routing/
my-epp-profile.yaml
Catalog entries can be referenced by name, with the .yaml suffix optional:
manifesto render manifest model_provider/model --cluster my-contextIf a file under clusters/ is named for the current kube context or kube
cluster, render and deployment commands select it automatically, so
--cluster can be omitted. Explicit paths and the existing
MANIFESTO_CLUSTER and MANIFESTO_CLUSTER_MAP settings remain supported.
Set MANIFESTO_CONFIG_HOME to override the entire user configuration path.
User catalog entries take precedence over bundled entries with the same name.
Use the config commands to discover that effective catalog and validate a model/cluster pairing before rendering or deploying:
manifesto config home
manifesto config list models
manifesto config list clusters
manifesto config list routing
manifesto config resolve models deepseek-v4/1P-EP8-1D-EP8
manifesto config resolve routing wide-ep-lws-config
manifesto config edit models deepseek-v4/1P-EP8-1D-EP8
manifesto config export models deepseek-v4/1P-EP8-1D-EP8 -o model.yaml
manifesto config import models model.yaml --name experiments/model
manifesto config validate deepseek-v4/1P-EP8-1D-EP8 --cluster example-gb200config list reports whether each winning entry is from the user or bundled
catalog. Use --output name for shell scripts or --output json to also see
when a user entry shadows a bundled entry. config validate loads both schemas,
applies cluster defaults, and exercises the full renderer without contacting or
changing the cluster.
config edit opens $EDITOR (or vi) on the user-catalog copy. Editing a
bundled model first copies it and any relative extends parents into the user
catalog, so the bundled source remains untouched. A new name creates a new user
config. The edited file is schema-validated when the editor exits; an invalid
file is retained so it can be fixed with the same command.
config export writes a self-contained YAML document with model inheritance
flattened. config import validates and installs that portable form in the user
catalog. Both refuse to overwrite files unless --force is passed.
Model specs live under models/.
Examples:
models/qwen/aggregated.yaml
models/deepseek-v4/1P-EP8-1D-EP8.yaml
models/deepseek-v4/3P-EP8-1D-EP16.yaml
A spec chooses the topology, roles, parallelism, vLLM args, and routing behavior:
release: wide-ep
topology: pd
accelerator: gb200
model:
id: deepseek-ai/DeepSeek-V4-Pro
image_ref: vllm.standard
vars:
max_concurrency: 1024
mtp_size: 1
roles:
- name: decode
workload_name: vllm-ep8-decode
lws: {size: 4}
parallelism: {tp: 1, pp: 1, dp: 16, ep: true}
computed:
env:
MAX_TOKENS: max_concurrency
vllm:
max_num_batched_tokens: max_concurrency * mtp_size
max_num_seqs: max_concurrency
max_cudagraph_capture_size: max_concurrencytp, pp, and dp are global tensor-, pipeline-, and data-parallel sizes.
Each engine replica consumes tp × pp GPUs. Local model/DP groups, port fanout,
and per-pod launch arguments are derived from the workload size and GPUs per
pod. GPUs per pod is inferred from the parallel layout and the cluster profile;
set parallelism.gpus to override it. Single-node roles render as Kubernetes
Deployments; roles spanning multiple nodes render as LeaderWorkerSets. Set a
role's workload to deployment or leaderworkerset to override that default.
An explicit one-node LeaderWorkerSet can be useful when an admission controller
integrates with LeaderWorkerSet rather than Deployment. Multi-node roles cannot
select Deployment.
For example, this runs one TP2 × PP2 engine across four GPUs. Increase
lws.size and divide those model-parallel GPUs evenly across pods to span
nodes; Manifesto renders the native vLLM node-rank and headless-worker flags.
roles:
- name: decode
lws: {size: 1}
parallelism: {tp: 2, pp: 2, dp: false}Configure PP only through parallelism.pp. Manifesto rejects
pipeline_parallel_size in vllm:, computed vLLM arguments, or
vllm_raw_args so GPU allocation and the vLLM worker topology cannot disagree.
Unknown role keys are rejected at load time, so typos fail loudly instead of being silently ignored.
Keep EPP scheduling policy in the routing catalog, separate from model specs.
A routing profile is an EndpointPickerConfig, for example
routing/kv-aware.yaml:
apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- {type: approx-prefix-cache-producer, name: gpu-prefix-cache-producer}
- type: prefix-cache-scorer
parameters: {prefixMatchInfoProducerName: gpu-prefix-cache-producer}
schedulingProfiles:
- name: default
plugins:
- {pluginRef: prefix-cache-scorer, weight: 3}Choose the profile when rendering or deploying; the model remains independent of the routing policy:
manifesto render manifest deepseek-v4/1P-EP8-1D-EP8 \
--routing-profile wide-ep-lws-config
manifesto deploy deepseek-v4/1P-EP8-1D-EP8 \
--routing-profile wide-ep-lws-configProfiles resolve from the user catalog first and then bundled routing/ files.
Manifesto embeds the resolved profile in the rendered EPP ConfigMap, so the
Kubernetes artifact remains self-contained. Set MANIFESTO_ROUTING_PROFILE to
choose a default without repeating the flag. Advanced specs can still use
plugin_configs with plugins_config_file to bundle and select multiple files;
existing routing.plugin_config specs continue to render as plugins.yaml.
Routed specs use a standalone Envoy sidecar in the EPP pod by default. The EPP
Service exposes port 80, while the InferencePool remains the source of model
endpoints and rank ports. Select the Gateway API frontend explicitly when shared
Gateway infrastructure is required:
routing:
kind: pd
target_role: decode
frontend: gateway # standalone (default) or gatewaymanifesto deploy removes resources left by the previous frontend after a
successful apply. The low-level manifesto file apply command intentionally
applies only the saved file and cannot prune resources omitted from it; use
manifesto stop before that workflow when changing frontends.
workload_name is optional. When set, it controls the Deployment or
LeaderWorkerSet name and therefore the Kubernetes pod name prefix, while full
release-specific instance labels still scope routing and selectors.
Image versions are centralized in config/images.yaml. Model specs should use
model.image_ref, with vllm.standard as the shared vLLM image reference,
unless a spec needs an explicit one-off model.image.
Each cluster profile declares its available accelerator profiles and a
default. Specs inherit the selected cluster's default unless they set
accelerator. The selected entry controls accelerator allocation,
accelerator-specific cache paths, and the development build architecture.
For model authors, accelerator allocation is cluster-owned: model specs keep the same role GPU counts whether the cluster uses extended resources or Dynamic Resource Allocation (DRA). Only cluster operators select or change the backend.
An extended-resource profile declares that backend explicitly:
accelerators:
default: b200
profiles:
b200:
allocation:
extended_resource:
resource_name: nvidia.com/gpu
presence_label: nvidia.com/gpu.present
gpu_arch: b200
torch_cuda_arch_list: "10.0+PTX"To use DRA instead, the cluster operator changes only allocation:
b200:
allocation:
dra:
device_class_name: gpu.nvidia.com
presence_label: nvidia.com/gpu.present
gpu_arch: b200
torch_cuda_arch_list: "10.0+PTX"allocation must contain exactly one of extended_resource or dra.
A DRA profile renders one resource.k8s.io/v1 ResourceClaimTemplate for each
GPU-bearing role. Each template requests the role's GPU count with ExactCount,
and only the model container receives the resulting claim. CPU, memory,
ephemeral storage, and RDMA remain ordinary container resources. Template names
include a digest of the complete immutable claim specification, so a release
can safely reuse a stable workload name when its claim metadata changes.
The cluster must serve resource.k8s.io/v1 and provide the configured
DeviceClass. When Kueue is enabled, configure its DRA integration and a
deviceClassMappings entry for that class, then give the mapped logical
resource ClusterQueue quota. Manifesto verifies the DRA API and DeviceClass
before deployment; Kueue remains authoritative for mapping and quota accounting.
Manifesto applies shared vLLM arguments before role-specific vllm: and
computed arguments. By default, Uvicorn access logs omit the high-frequency
/health, /v1/models, and /metrics endpoints while preserving inference
request logs. A role can replace the excluded endpoint list with
vllm.disable_access_log_for_endpoints, or set it to null to omit the
Manifesto default entirely.
Role settings activate typed serving features before rendering. Data, pipeline, and expert parallelism, P/D, and llm-d are features. Connector and all-to-all implementations are reported separately as backends; workload kind and platform resources are derived consequences. For example:
routing.kind: disabledselects direct vLLM, whileload_awareorpdselects llm-d.- data parallelism uses external DP when llm-d is selected and internal DP for direct vLLM.
- P/D implies llm-d and a routing proxy for now.
- a NIXL connector backend supplies
VLLM_NIXL_SIDE_CHANNEL_HOSTfrom the Pod IP unless the role provides an explicit value; other connector backends are preserved without being treated as NIXL. - a parallel shape spanning multiple nodes selects a LeaderWorkerSet; single-node shapes select a Deployment.
- DeepEP is reported as the selected all-to-all backend for EP, and an IMEX claim is derived from the cluster platform capability when configured.
Dependencies are resolved transitively and the registry rejects implication cycles. They do not add framework annotations or dependency metadata to the rendered objects. Inspect the internal resolution separately:
manifesto explain models/deepseek-v4/1P-EP8-1D-EP8.yaml \
--cluster clusters/example-gb200.yaml \
--user "$USER"The explanation includes each role's enabled features, selected backends, workload and fabric profile, environment provenance, and resource claims.
Lower-level integrations can append exact command-line fragments with
vllm_raw_args. Unlike the vllm: mapping, these strings are not renamed,
quoted, or otherwise interpreted, and they are emitted after structured and
computed arguments. Each entry is trusted shell syntax, so quote spaces and
shell metacharacters within the entry when needed:
roles:
- name: decode
vllm_raw_args:
- --trust-remote-code
- --attention-config.backend=FLASH_ATTNSpecs can also extend a base YAML file and override only the fields that differ:
extends: wide-ep-base.yaml
release: wide-ep-2p-ep8-1d-ep8
roles:
prefill:
lws: {replicas: 2}Overrides deep-merge mappings. roles may be written as a map keyed by role
name in override files; each role override is merged into the matching base
role before normal schema validation.
Every deployment includes an idle-shutdown controller by default. Once all expected vLLM API servers are ready, it watches request counters and running or queued requests across all roles. After 45 idle minutes it scales the model workloads and endpoint picker to zero, deletes the instance Gateway when the Gateway frontend is selected so its platform-managed proxy replicas are released, and scales itself to zero. Applying or deploying the spec again restores all of those resources. Change the timeout or opt out in the runtime configuration:
runtime:
idle_shutdown:
timeout_minutes: 90
# enabled: falseMetric or Kubernetes API failures reset the idle timer, so the controller does not shut down an instance when it cannot establish that the instance is idle. For one-off operations, the CLI can override either setting without editing the model spec:
manifesto deploy models/qwen/qwen3-0.6b.yaml \
--context example-context \
--idle-timeout 15m
manifesto render manifest models/qwen/qwen3-0.6b.yaml \
--no-idle-shutdownDeepSeek V4 wide-EP specs use filenames that encode only the parallel layout:
<prefill-replicas>P-EP<width>-<decode-replicas>D-EP<width>.yaml. For example,
3P-EP8-1D-EP16.yaml means three prefill LWS replicas at EP8 and one decode
LWS replica at EP16. Backend choices stay inside the YAML because optimal
backends can change independently of the parallel layout.
Cluster profiles live under clusters/.
Three synthetic profiles are included for documentation and tests:
clusters/example-gb200.yaml
clusters/example-h200.yaml
clusters/example-stateless-b200.yaml
They deliberately omit real network, storage, scheduling, and provider details.
Put usable profiles in the private user catalog described above; clusters/*.yaml
is ignored by git except for files named example-*.yaml.
The stateless example intentionally declares no cache or logging filesystem. Its model pods omit persistent cache env, storage mounts, and the log tee wrapper; libraries use container-local defaults and logs stream to stdout. An external vllm-envs worktree cannot be selected unless its absolute path is covered by a model-pod volume mount.
Manifesto also omits Kubernetes fields whose defaults are sufficient. Set
pod_defaults.image_pull_secrets, image_pull_policy,
termination_grace_period_seconds, working_dir, or
container_security_context only when cluster or image policy requires an
explicit value. Pull secrets are existing Secret names in the workload
namespace; Manifesto references them but does not create credential data. They
are attached to every generated PodSpec, including routing and idle-shutdown
pods. For example:
pod_defaults:
image_pull_secrets:
- example-registry-credentialsA dedicated model-server ServiceAccount is generated only when an OpenShift SCC must be bound to it.
Cluster profiles own environment that should not be repeated in every model spec:
- available accelerators and the cluster default
- GPUs per node
- shared and local volume mounts
- optional namespace bootstrap configuration for shared PVCs
- pod annotations, scheduling constraints, extra devices, and security context
- RDMA extended-resource requests and OpenShift SCC authorization
- user, log, and cache path templates
- llm-d release
- UCX/NCCL/NVSHMEM/IMEX fabric env profiles
- optional Kueue LocalQueue selection for GPU LeaderWorkerSets
When a role omits resources.cpu or resources.memory, Manifesto derives the
request from the role's inferred GPUs per pod (N) using built-in defaults:
- CPU:
6 + 2N - memory:
max(64Gi * N, 128Gi)
For example:
| GPUs per pod | CPU | Memory |
|---|---|---|
| 1 | 8 | 128Gi |
| 2 | 10 | 128Gi |
| 4 | 14 | 256Gi |
| 8 | 22 | 512Gi |
Set either value directly on a role when a model needs a different request:
roles:
- name: decode
parallelism: {tp: 4}
resources: {memory: 224Gi}Explicit role resource values always win independently, so the example keeps the built-in 14-CPU request while using exactly 224Gi of memory.
Controllers that submit other GPU workloads can reuse the cluster-owned part of a Manifesto profile without depending on model topology or routing:
from manifesto.cluster import load_cluster
from manifesto.workload import workload_settings
settings = workload_settings(load_cluster("clusters/my-cluster.yaml"))
accelerator = settings.accelerator("gb200")The portable projection contains accelerator resource or DeviceClass names and node selectors, the default Kueue LocalQueue, accelerator-attached platform claims such as an NVIDIA compute-domain channel, and pod placement defaults such as affinity, tolerations, DNS, annotations, and image pull policy. The projection carries only each claim's portable name and template reference; storage, the rest of the fabric configuration, and launch settings remain part of Manifesto's serving-specific cluster model.
Manifesto also owns a controller-neutral workload IR and its Kubernetes object lowering. This lets tools describe a pod template and lifecycle policy while Manifesto consistently applies cluster placement and Kueue metadata:
manifesto render workload examples/non-indexed-job.yaml \
--cluster clusters/my-cluster.yaml \
--accelerator gb200The job backend emits an ordinary, non-indexed batch/v1 Job by default and
can explicitly describe Indexed Job completion policy for distributed work. A
workload may optionally declare a normal or headless Service. The same IR
lowers Deployment and LeaderWorkerSet backends, including explicit LWS leader
templates; Grove is reserved as an extension point until its rendering
contract is implemented.
An explicit LWS leader template is intentionally same-shaped: Manifesto applies the workload's accelerator count, accelerator container, and cluster-owned claims to both leader and worker templates. It also removes Kueue queue labels from both Pod templates and keeps queue selection on the LeaderWorkerSet, as required by Kueue's LWS integration.
GPU roles can be admitted through a Kueue LocalQueue selected in the cluster profile:
kueue:
local_queue: example-gpu-queueWhen configured, Manifesto preserves each role's native workload kind. A
single-node role remains a Deployment and receives
kueue.x-k8s.io/queue-name on both the Deployment and its pod template. Each
Deployment pod is therefore admitted independently. A role whose topology
requires a LeaderWorkerSet receives the queue label on the LeaderWorkerSet,
whose complete leader/worker group is admitted atomically. The queue is
cluster- and namespace-specific operational configuration, so it belongs in
the private cluster profile rather than a shareable model spec. Omitting
kueue preserves the same workload kinds without admission metadata.
CPU-only supporting resources, including the in-cluster end-to-end probe Job,
do not receive Kueue metadata or suspension.
Before applying a queued workload, manifesto deploy verifies that the Kueue
APIs are served, the selected LocalQueue is
Active=True, and its referenced ClusterQueue is Active=True. It also prints
every init-container and container request in each rendered workload pod
template so quota requirements are visible before mutation, and rejects
request resource names that the ClusterQueue does not cover. Deployments do not
require the LeaderWorkerSet API; native LeaderWorkerSets are checked for it.
Changing queue admission on an existing LeaderWorkerSet is intentionally a recreate operation because its admitted group is atomic. Enabling, changing, or disabling its queue deletes it before applying the replacement. Deployment queue changes use the Deployment rollout and replace independently admitted pods without changing workload kind. Moving a same-named role between native Deployment and LeaderWorkerSet shapes deletes the obsolete controller first. LWS recreation and kind transitions interrupt serving; use a distinct release name for a parallel, zero-downtime rollout.
The workflow CLI requires a cluster profile for commands that render a spec. Set
MANIFESTO_CLUSTER directly, pass --cluster, or set MANIFESTO_CLUSTER_MAP in
.env as a local lookup table keyed by kube context or kube cluster name:
MANIFESTO_CLUSTER=clusters/example-gb200.yaml manifesto deploy models/qwen/aggregated.yaml
manifesto deploy models/qwen/h200-aggregated.yamlThe namespace defaults to the selected kube context namespace, falling back to
default when the context has no namespace. Use --context to select a context
without changing kubectl's current context, or set MANIFESTO_NAMESPACE in
.env to override the namespace.
Shared storage is configured without assuming a filesystem implementation:
storage:
shared_volume:
persistentVolumeClaim:
claimName: model-cache
shared_mount_path: /mnt/shared
local_nvme_path: /mnt/localshared_volume accepts any Kubernetes volume source, such as
persistentVolumeClaim, hostPath, csi, nfs, or emptyDir. Profiles
backed by separate host-local cache volumes can omit it entirely.
When Manifesto should provision a referenced shared PVC in each namespace,
declare storage.shared_claim with its storage class, access modes, and size,
inspect it with manifesto render bootstrap, then run manifesto deploy bootstrap.
The apply command uses kubectl apply, so it is safe to repeat. PVCs managed
outside Manifesto should omit shared_claim.
Set platform: openshift for OpenShift clusters. This adds a stable USER
fallback for Python libraries when containers run under an arbitrary UID,
without affecting standard Kubernetes profiles. It also omits the
node-exporter sidecar because its /sys and /proc hostPath volumes are
incompatible with OpenShift's restricted SCC; dcgm-exporter remains enabled
when requested. An SCC can be granted to each release-specific model service
account:
openshift:
scc: custom-sccA model role can override automatic fabric selection when its backend requires one explicitly:
roles:
- name: decode
fabric_profile: custom_epFabric profiles provide backend-specific environment settings:
fabric:
profiles:
deepep_decode:
env:
VLLM_USE_NCCL_SYMM_MEM: "1"Select a non-default Gateway API implementation in the cluster profile:
gateway:
class_name: data-science-gateway-class
service_type: ClusterIPCompiled-cache paths derive their cache key from the resolved model image tag or digest. Custom and dev builds can force a fresh namespace explicitly:
cache:
key: dev-build-42When persistent cache storage is configured, model pods clear their JIT and
compilation caches after a failed process exit, or when the same container
restarts after terminating without running its exit handler. Stateless pods
omit this machinery. FlashInfer autotuning results under
$VLLM_CACHE_ROOT/flashinfer_autotune_cache are preserved. Disable this default
for a deployment only when retaining possibly corrupt compiled artifacts is
intentional:
cache:
cleanup_on_crash: falseModel-server stdout/stderr is persisted by the generated launch script. Configure the backing PVC and root path in the cluster profile:
logging:
pvc: example-shared-cache
mount_path: /mnt/shared
root: /mnt/shared/{user}/logsThe rendered pods tee logs to {root}/{role}, for example
/mnt/shared/<user>/logs/decode and /mnt/shared/<user>/logs/prefill.
The file workflow is the preferred path when you want to share or tweak exactly what will be deployed:
manifesto render file models/deepseek-v4/1P-EP8-1D-EP8.yaml
$EDITOR /tmp/manifesto.yaml
manifesto file diff
manifesto file applymanifesto file diff compares the rendered file against the live cluster with
kubectl diff. manifesto file apply applies that file as-is. Use -o/--output
with render file, or set MANIFESTO_RENDER_OUT, to choose a different file.
The low-level manifesto render manifest command performs no cluster I/O when passed an
explicit --cluster and --namespace (or with MANIFESTO_NAMESPACE set).
Workflow commands such as deploy, ready, file diff, and file apply are the
cluster-touching commands.
manifesto deploy SPEC *ARGS # render and apply a full stack
manifesto servers # list live servers in the namespace
manifesto stop [SPEC] [--now] # discover and delete live objects
manifesto stop --instance ID # stop by live instance identity
manifesto ready SPEC # wait for pods and routing frontend
manifesto test e2e SPEC # fresh-namespace dev + inference integration test
manifesto deploy routing SPEC *ARGS # update routing onlymanifesto stop is stateless: it finds live objects through their Manifesto
instance labels and does not depend on a stored manifest or deployment
inventory. With no target it opens an fzf picker, or a numbered picker when
fzf is unavailable. Selecting an entry starts teardown immediately. Explicit
spec and --instance targets remain noninteractive and are safe for scripts.
Teardown removes routing, model workloads, supporting objects, and remaining
instance pods. It does not drain active requests or delete shared logs, caches,
or model storage. --now retains the force-delete behavior.
Enable parser-driven completion for commands, flags, model configs, cluster and routing profiles, accelerator names, live deployment instance IDs, and local paths:
source <(manifesto completion zsh) # zsh
source <(manifesto completion bash) # bash
manifesto completion fish | source # fishFor scripts, manifesto servers --output name prints one instance ID per line,
and --output json includes the exact resources associated with each instance.
manifesto servers and manifesto stop find live objects by listing every
Manifesto-managed resource type. Those lists run concurrently, one request per
type, so discovery costs roughly one API round trip of wall-clock time instead of
one per type. There is no kubectl api-resources preflight: a type the cluster
does not serve simply lists as zero objects, and pruning the set in advance would
not save any wall clock once the lists run in parallel.
Discovery reads are bounded and retried on transient faults. Tune them when an API server is unusually slow or flaky:
MANIFESTO_KUBECTL_TIMEOUT=300 # seconds per read attempt; 0 disables the bound
MANIFESTO_KUBECTL_RETRIES=4 # extra attempts after a transient failure
MANIFESTO_TRACE=1 # log every kubectl invocation and its durationMANIFESTO_TRACE=1 is the fastest way to find which call is slow:
MANIFESTO_TRACE=1 manifesto stop --instance IDShell completion of live instance IDs uses a short, non-retrying timeout, so a slow cluster yields no candidates instead of stalling the prompt.
Examples:
manifesto deploy models/qwen/aggregated.yaml
manifesto ready models/qwen/aggregated.yaml
manifesto test e2e models/qwen/qwen3-0.6b.yaml --cluster my-cluster
manifesto deploy models/deepseek-v4/1P-EP8-1D-EP8.yaml \
--vllm-env /mnt/shared/$USER/vllm-envs/feature
manifesto deploy routing models/deepseek-v4/1P-EP8-1D-EP8.yaml
manifesto stopmanifesto test e2e owns the complete test lifecycle. By default it creates a
fresh namespace, replaces profile storage with emptyDir volumes, deploys the
vLLM bundled in the model image, waits for it to become ready, and creates an
unprivileged Job that calls /v1/models followed by a real /v1/completions
request. The Job targets the standalone router Service or generated Gateway
when routing is enabled, and the model Service otherwise.
Pass --vllm-env /absolute/worktree/path to test an existing vllm-envs
worktree instead. That mode preserves the cluster profile's normal storage so
the worktree remains visible; its volume references must be usable in the fresh
test namespace. Manifesto never creates, synchronizes, or modifies the worktree.
The namespace is deleted at the end, including on failure. Pass --namespace
to choose its name (the command refuses an existing namespace),
--keep-namespace to retain all test resources for inspection, --timeout to
change the inference deadline, or --image to use a mirrored Python image.
HF_TOKEN and a cluster profile are required, as with normal deployments.
Every rendered object is scoped by the release-based instance identity:
{release}
Names, labels, and selectors derive from that identity. User-scoped storage
paths and the llm-d.ai/owner label still retain the user. Deployments must have
unique release names within a namespace. Clusters whose users share release
names can enable user-prefixed identities in their profile:
naming:
user_prefix: trueUse vllm-envs to create, synchronize,
build, and remove development environments. Manifesto only points model pods at
an existing vLLM worktree whose .venv has already been initialized:
manifesto deploy models/deepseek-v4/1P-EP8-1D-EP8.yaml \
--vllm-env /mnt/shared/$USER/vllm-envs/featureThe path must be absolute and covered by a volume mount in every model pod.
Manifesto fails before launching vLLM if the worktree or .venv/bin/activate is
missing. Set the same pointer declaratively with runtime.vllm_env in a model
spec. Without either setting, the deployment uses vLLM from the model image.
The monitoring/ directory contains namespace-scoped Prometheus and Grafana
configuration. Per-pod exporters such as DCGM exporter are part of rendered
model manifests and can be toggled from the model spec.
Local tools:
uvkubectl
Expected .env values:
HF_TOKEN=replace-me
KUBECONFIG=/path/to/kubeconfig
MANIFESTO_CLUSTER_MAP=my-context=clusters/example-h200.yaml
MANIFESTO_NAMESPACE=workload-ns # optional, defaults to current kube namespacemanifesto deploy and manifesto file apply
create or update the namespace's hf-secret from HF_TOKEN before launching
pods. The token is sent to kubectl apply over stdin and is not included in
rendered workload files. Pure manifesto render commands remain side-effect
free; manage the secret explicitly when applying rendered stdout with kubectl.
uv run --extra test pytest
manifesto --helpManifesto is intentionally not a controller, operator, CRD, or matching engine.
It is a client-side renderer:
model spec + cluster profile + user -> raw Kubernetes YAML
The YAML is the artifact. Inspect it, edit it, share it, apply it.