vLLM-Omni: serve image and audio models on your own GPUs

vLLM-Omni serves supported multimodal checkpoints through an extension to vLLM. It coordinates autoregressive and diffusion stages for tasks such as image generation and streaming speech. Choose the checkpoint and its supported pipeline first, then size the GPU session around that workload. The runtime is Apache 2.0; checkpoint licenses and access conditions belong to each model.

Original pipeline explanation. The stages and device allocation depend on the selected checkpoint.

On this page: Choose a workload · Install matching releases · Plan memory · Read a streaming benchmark · Measure cost · Sources and next steps

Choose a checkpoint and an endpoint

Use vLLM-Omni when you need a supported model’s image or audio pipeline behind a service, especially when multiple stages must run together. Its OpenAI-compatible server offers task-specific endpoints; compatibility does not imply that every model supports every endpoint. Start from the supported-model documentation and the corresponding example.

For a simple text-only assistant, compare ordinary vLLM or the local runtime directory. For a checkpoint too large to keep resident, AirLLM addresses a different constraint by streaming weight layers from disk. With an image workload, the Qwen-Image-2.1 guide covers model choice, licensing and hosted alternatives before you select a runtime.

Install matching releases on Linux

The fixed v0.30.0 quickstart requires Linux and Python 3.12 and pairs vLLM-Omni with vLLM 0.30.0. Pin both sides for a reproducible setup. The latest documentation has moved to vLLM 0.31.0; do not mix its dependency commands with the older tag. GPU platform requirements also differ between CUDA and ROCm.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm==0.30.0 --torch-backend=auto
git clone --branch v0.30.0 --depth 1 https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e .
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091

This CUDA example follows the tagged quickstart, adding a tag checkout to keep the runtime fixed. Model download and startup need time and storage before the server is ready. From another terminal, send a short request to the local image endpoint:

curl -s http://localhost:8091/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"a cup of coffee on the table","size":"1024x1024","response_format":"b64_json","seed":42}' \
  | jq -r '.data[0].b64_json' | base64 -d > coffee.png

A successful response contains encoded image data, which this command writes to coffee.png. Inspect server logs and the response before diagnosing a missing file: dependency mismatch, model access and insufficient device memory are separate failures. This local example binds a development service; an internet-facing deployment also needs authentication and network controls.

Use the current Drama 3 recipe for speech

Drama 3 has a newer serving path than the fixed image example above. Its official vLLM recipe requires vLLM 0.31.0 or later and a vLLM-Omni build with Drama 3 support, plus a CUDA GPU with at least 48GB. The reference run uses one H100 80GB. Stage-0 weights occupy 36.9GiB; the codec, caches and runtime allocations also need memory.

Use the recipe’s Drama 3 launch configuration and the current speech API guide for /v1/audio/speech, speaker tags and reference audio. The Drama 3 model guide covers its character-based API alternative and weight license.

Budget memory across the complete pipeline

There is no single minimum VRAM figure for vLLM-Omni. The checkpoint, precision, resolution, output length and concurrency determine the workload. Reserve space for resident weights, caches, activations, CUDA graphs and runtime buffers. Host memory and disk must also accommodate dependencies, downloads and staging.

The memory configuration guide describes a per-stage budget based on total device memory and gpu_memory_utilization, capped by free memory at initialization. Stages sharing a GPU need room collectively. Raising a utilization value cannot create capacity already occupied by another process.

Begin with the model’s supplied deployment configuration and one request. Confirm that every stage fits, then increase concurrency while watching memory and queue delay. If a request fails at a larger resolution or context, reducing concurrent work is a different intervention from changing model precision. Keep those settings with the measurement so a successful load is not mistaken for a useful production capacity figure.

What a developer’s streaming speech test shows

John Robinson’s Qwen3-TTS streaming demo reports a warm median time to first audio of 47ms for 0.6B CustomVoice and 59ms for a 0.6B Base voice-cloning path on an RTX 5090. Its setup pins vLLM and vLLM-Omni 0.30.0, uses Python 3.12.11 and documents Linux, CUDA 13 and at least 10GB free GPU memory. The default benchmark performs three warmups and three timed runs.

The repository also lists 2,300ms for a Transformers-native 1.7B Base benchmark. That uses a larger checkpoint and a different path, so dividing those times would mix model size with the streaming implementation. The useful demonstration is that the two-stage Talker-to-Code2Wav pipeline can return an audio chunk before the utterance finishes. First audio does not measure full synthesis time or sound quality. Open the repository’s WAV samples and benchmark script when evaluating the listening experience.

Price the GPU session and accepted outputs

The runtime has no per-token API tariff. For rented hardware, multiply the provider’s hourly rate by the complete billable session, including startup, model loading, warmup and idle time. Add storage and transfer charges when applicable. For owned hardware, include electricity and the machine cost appropriate to your usage.

For example, an assumed $2-per-hour GPU session lasting 45 minutes costs $1.50 before storage and transfer. If it produces 100 accepted images, GPU time is $0.015 per accepted image; if only 50 pass review, it is $0.03. These are arithmetic examples, not a provider quote or a measured throughput result. Revisions and failed generations consume the same session budget.

For speech, record both first-audio delay and complete generation time. The metrics guide distinguishes end-to-end request latency, first audio packet and real-time factor. Real-time factor is generation duration divided by audio duration; a value below one means generation is faster than playback. Queue time, aborted requests and stream underruns matter to users even when a warm first packet arrives quickly.

Compare a hosted API on the same accepted workload when demand is irregular or maintaining a GPU service would take more time than it saves. For a busy service, measure several concurrency levels before treating a warm single-request timing as your capacity.

Sources and next steps

Reviewed October 16, 2026. Read the official repository and license, fixed v0.30.0 quickstart, latest quickstart and v0.30.0 release notes for version-specific changes. Continue with the checkpoint’s example, or compare the local inference options before renting a device.