[IlyasMoutawwakil]

What does this PR do?

Stacked on #50029, which adds ExportedModel. This PR makes it run: an export is called like the model it came from, on its backend's runtime.

from transformers.exporters import ExportedModel

model = ExportedModel.from_pretrained("my-org/my-model", subfolder="onnx", device="cuda")
outputs = model(**processor(images=images, return_tensors="pt"))
outputs.pooler_output  # the source model's own output class

Why

• No glue code. Converting tensors for the backend, matching kwargs to graph inputs, picking a device, rebuilding outputs: every user of an export writes this today. sentence-transformers needs it for its ONNX and OpenVINO backends (sentence-transformers#4178).
• Drop-in for the model. Same kwargs and same ModelOutput class as the source model, caches included.

How

• Each component gets a runner for its backend (exporters/runners/runner_<backend>.py), registered with register_backend(exporter, runner).
• A fresh export builds its runners on its first call; from_pretrained(..., device=...) builds them while loading. .to(device, **runner_kwargs) rebuilds them elsewhere or with backend options: providers and session_options for ONNX Runtime, config and torch_threads for OpenVINO.
• from_pretrained takes download options as named parameters, as PreTrainedModel.from_pretrained does, and runner options as kwargs, checked before any program file is downloaded.
• An ONNX Runtime session opened with CUDA graphs (enable_cuda_graph, trt_cuda_graph_enable) replays one: its inputs and outputs keep fixed device buffers. Capture needs a static export.
• The metadata records torch.export's input and output specs, so a runner rebuilds the model's output structure, and loading registers the cache classes it names.
• A call drops kwargs the export wasn't traced with, so a processor's whole output can be passed. Exports with several components raise when called; their runtimes come in follow-ups.

Backend	Runs through
Dynamo	ExportedProgram.module()
ONNX	an ONNX Runtime session, inputs and outputs bound in place
OpenVINO	the model compiled for the device
ExecuTorch	the program's forward

Also fixed: a GatherND at constant indices (the attention mask's) exports as a Gather, since ONNX Runtime's CUDA GatherND breaks graph capture (microsoft/onnxruntime#33875); ONNX input names recorded before a lazily initialized cache grows; bfloat16 convolutions run in float32 for ONNX (no bfloat16 Conv before opset 22); a saved program keeps EncoderDecoderCache.is_updated's int keys; 16-bit OpenVINO exports compile in their own precision, recorded on the model, instead of fp32 (bf16 EmbeddingGemma2 on CPU: 99 → 48 ms static); repeat_kv's expand broadcasts in numpy mode, which OpenVINO's CPU plugin keeps in 16-bit (73 → 52 ms dynamic); a bf16 model whose buffers are views exports to OpenVINO; a quantized ONNX export keeps its large constants when saved.

Performance

EmbeddingGemma2 text embeddings, 8 sentences of 128 tokens, like for like: dynamic exports against torch.compile(dynamic=True), and static ones replayed as CUDA graphs against torch.compile(mode="reduce-overhead"). Reproduce with exporters_runtime.py, a self-contained uv script (uv run exporters_runtime.py --device cuda [--dynamic]); its docstring has the methodology.

CUDA, A100-SXM4-80GB, every row fp16. Fused rows use OnnxConfig(ort_fusions="cuda") (#50309).

Dynamic shapes:

Static shapes, replayed as CUDA graphs (ONNX Runtime's enable_cuda_graph, TensorRT's trt_cuda_graph_enable), against torch.compile(mode="reduce-overhead"):

CPU, Xeon Platinum 8570, 53 physical cores of one socket, every row bf16.

Dynamic shapes:

Static shapes, against torch.compile(dynamic=False):

Tests

• test_runtime.py: every backend called fresh and loaded, against eager; runner options and their refusals; CUDA graph replay; caches loaded in a fresh process; the decode's static cache through the runner.
• test_save_load.py: ONNX exports from the Hub cache, in bfloat16 and past 2GB, run through the runtime.
• Forward export tests of Llama, BERT, T5, Gemma3, Qwen2-VL, CLIP, Whisper and Mamba on Dynamo, ONNX and OpenVINO, and of Llama and BERT on ExecuTorch.

🤖 Generated with Claude Code

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

• en/exporters
• en/exporters_extend
• en/exporters_quantization
• en/main_classes/exporters

[echarlaix]

Very cool work @IlyasMoutawwakil

[ArthurZucker]

ty

[IlyasMoutawwakil]

quick heads up, since the approvals a few things changed api-wise:

• runner files are now runners/runner_<backend>.py (same as quantizer_<backend>.py)
• from_pretrained takes the download args as named params like PreTrainedModel, and runner options as kwargs (e.g. providers=, session_options=), checked before anything downloads
• onnx: device or providers, not both. default is cuda only when a gpu is actually there
• onnx runner supports cuda graphs, opt-in via providers (enable_cuda_graph / trt_cuda_graph_enable), static exports only
• GatherND at constant indices (the mask's) now exports as Gather, ort's cuda GatherND breaks graph capture (CUDA graph replay of a graph with GatherND fails with an illegal memory access microsoft/onnxruntime#33875). no output changes across the export tests
• ExportMetadata: dtype/device are now export_dtype/export_device, the input/output name lists are gone (the specs and the programs have them), and the constants are recorded at export (in_constants for scalar kwargs, out_constants for non-tensor outputs like None hidden states)
• onnx and openvino share their graph i/o plumbing in exporters.utils now (output wrapping, feeds, post-conversion quantization), so openvino no longer imports from the onnx exporter

also fixed a few things the cpu benchmark surfaced (ort quantizer constants, openvino bf16 view buffers), each with a regression test, and benchmarks are updated (fp16, dynamic + static w/ cuda graphs)

and fixed what the onnx + openvino export sweeps through the runners turned up (non-tensor outputs, onnx bool outputs, onnx outputs returning an input as is, openvino broadcasts to 0, openvino bicubic antialias resizing, executorch precomputed kwargs, beam_idx with sliding caches). all the sweep failures pass now