onnxruntime
ONNX Runtime is Microsoft's cross-platform engine for running ML models exported to the ONNX format. You train in PyTorch, TensorFlow, or scikit-learn, export to a .onnx file, and ONNX Runtime executes it with graph optimizations and hardware-specific execution providers (CPU, CUDA, TensorRT, DirectML, CoreML, and more). The point is deployment without dragging the training framework along: a small C++ core with Python bindings that runs the same model file on a server, a Windows desktop, or a phone, usually faster than eager PyTorch inference on CPU.
The most portable serious inference runtime: one artifact, many backends, strong CPU performance, and Microsoft-grade maintenance. Its price of admission is the ONNX export step, so validate that your model converts cleanly before you architect around it, and use a specialized engine for large generative LLMs.
Use it if
- You deploy models where installing PyTorch is unreasonable: a .onnx file plus the compact onnxruntime wheel replaces a multi-gigabyte training stack
- You need one model artifact running across servers, Windows apps, mobile, and browsers via a family of runtimes sharing the same format
- CPU inference speed matters: graph fusion and quantization routinely beat eager PyTorch for classic CV, NLP encoder, and scikit-learn models
- You are on non-NVIDIA hardware: execution providers cover DirectML, CoreML, OpenVINO, and NPU targets that CUDA-only stacks ignore
- Your real problem is getting the model into ONNX: export is where projects die, with unsupported ops, dynamic control flow, and opset mismatches; if a clean export does not exist for your architecture, the runtime never gets a chance
- You serve large generative LLMs: dedicated engines like vllm or sglang handle KV cache management, continuous batching, and quantized attention kernels that plain ONNX Runtime sessions do not
- You can just stay in PyTorch: torch.compile and torch.export closed much of the inference gap, and skipping the conversion step removes a whole class of accuracy-drift debugging
- GPU packaging friction bothers you: onnxruntime and onnxruntime-gpu are separate packages that conflict when both installed, and the GPU wheel must match specific CUDA and cuDNN versions
Setup reality
The CPU wheel installs cleanly anywhere with only numpy, protobuf, and flatbuffers as dependencies; note that 1.28 requires Python 3.11+. GPU is where it gets annoying: you must uninstall onnxruntime before installing onnxruntime-gpu (they collide over the same import name), match the wheel to your CUDA and cuDNN versions, and pass an explicit providers list or you silently fall back to CPU. The real setup cost is upstream: torch.onnx.export with the right opset, dynamic axes, and a numerical parity check against the source model. Also worth knowing: the README states the project may collect usage data for Microsoft, with an opt-out.
Patterns
Load a model and run inferencerun-inference
import onnxruntime as ort
import numpy as np
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
x = np.random.randn(1, 3, 224, 224).astype(np.float32)
outputs = sess.run(None, {"input": x})
print(outputs[0].shape)None for output names returns all outputs in order; input dtype must match the graph exactly, float64 in place of float32 is the classic first error.
Discover input and output names and shapesinspect-model-io
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
for i in sess.get_inputs():
print(i.name, i.shape, i.type)
for o in sess.get_outputs():
print(o.name, o.shape, o.type)Dimensions exported as dynamic show up as strings like 'batch_size' instead of integers; feed dict keys must match these names exactly.
Run on GPU with CPU fallbackuse-cuda-gpu
sess = ort.InferenceSession(
"model.onnx",
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
print(sess.get_providers()) # verify CUDA actually loadedRequires the onnxruntime-gpu package with matching CUDA and cuDNN; if the CUDA provider fails to load, it silently falls back to CPU, so always check get_providers().
Export a PyTorch model and verify parityexport-from-pytorch
import torch
torch.onnx.export(
model, dummy_input, "model.onnx",
input_names=["input"], output_names=["output"],
dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
opset_version=17,
)
expected = model(dummy_input).detach().numpy()
got = ort.InferenceSession("model.onnx").run(None, {"input": dummy_input.numpy()})[0]
np.testing.assert_allclose(expected, got, rtol=1e-3, atol=1e-5)Without dynamic_axes the batch size is frozen at the dummy input's value; always assert numerical parity before shipping the export.
Control threading and optimization levelsession-options-threads
opts = ort.SessionOptions()
opts.intra_op_num_threads = 4
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
sess = ort.InferenceSession("model.onnx", opts, providers=["CPUExecutionProvider"])Default threading grabs all cores, which fights other workloads on shared boxes; ORT_ENABLE_ALL is already the default optimization level.
Shrink and speed up with dynamic quantizationquantize-model
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic("model.onnx", "model.int8.onnx", weight_type=QuantType.QInt8)Dynamic quantization needs no calibration data and works well for transformer encoders on CPU; conv-heavy vision models usually want static quantization with calibration instead.
Save the optimized graph to skip warmup workcache-optimized-model
opts = ort.SessionOptions()
opts.optimized_model_filepath = "model.opt.onnx"
sess = ort.InferenceSession("model.onnx", opts, providers=["CPUExecutionProvider"])Loading the pre-optimized file later cuts session creation time; the optimized graph can be provider-specific, so do not reuse it across different EPs.
Keep tensors on GPU with IO bindingio-binding-gpu
binding = sess.io_binding()
x_ort = ort.OrtValue.ortvalue_from_numpy(x, "cuda", 0)
binding.bind_ortvalue_input("input", x_ort)
binding.bind_output("output", "cuda")
sess.run_with_iobinding(binding)
result = binding.get_outputs()[0].numpy()Plain sess.run copies inputs and outputs through host memory every call; IO binding removes that transfer, which dominates latency for small models on GPU.
Serve a scikit-learn model via ONNXrun-sklearn-model
from skl2onnx import to_onnx
onx = to_onnx(clf, X_train[:1].astype(np.float32))
with open("clf.onnx", "wb") as f:
f.write(onx.SerializeToString())
sess = ort.InferenceSession("clf.onnx", providers=["CPUExecutionProvider"])
pred = sess.run(None, {"X": X_test.astype(np.float32)})[0]Needs the separate skl2onnx package; classifiers return two outputs (labels and per-class probabilities), so check get_outputs() rather than assuming one.
Profile per-op execution timeprofile-inference
opts = ort.SessionOptions()
opts.enable_profiling = True
sess = ort.InferenceSession("model.onnx", opts, providers=["CPUExecutionProvider"])
sess.run(None, {"input": x})
trace_file = sess.end_profiling()
print(trace_file) # chrome://tracing compatible JSONOpen the JSON in chrome://tracing or Perfetto to see which ops dominate; profile after a warmup run so one-time optimization cost does not skew results.
Convert a model to float16 for GPUfloat16-conversion
import onnx
from onnxconverter_common import float16
model = onnx.load("model.onnx")
model_fp16 = float16.convert_float_to_float16(model)
onnx.save(model_fp16, "model.fp16.onnx")Requires the onnxconverter-common package; fp16 roughly halves memory and speeds up recent GPUs, but validate outputs since overflow-prone models need op exclusions.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| openvino | PyPI | You deploy on Intel CPUs, iGPUs, or NPUs and want Intel's tuned kernels and tooling. |
| torch | PyPI | PyTorch is already in your deployment image and torch.compile gets you close enough without a format conversion. |
| tensorrt | PyPI | You are all-in on NVIDIA GPUs and want maximum kernel-level performance at the cost of portability. |