vllm-omni
Serving engine for multimodal, diffusion, speech, and action models
Featured by Willet
What it is
An extension of vLLM that runs inference for models producing text, images, audio, video, and robot actions, including diffusion transformers and other non-autoregressive architectures. It breaks models into pipelined stages that can run on separate hardware, can split work across GPUs by tensor, pipeline, data, or expert, and offers an OpenAI-compatible API with streaming and full-duplex realtime audio. ML engineers deploying models such as Qwen3-Omni, TTS systems, or video generators on GPU and NPU clusters use it to serve them at scale.