vLLM: build and budget a shared model endpoint

vLLM is an Apache-2.0 inference and serving engine for running supported models on your own hardware. It schedules requests, manages inference memory and exposes an API that applications can call. It is useful for a shared model service or a batch workload where concurrency matters. Download model weights separately and follow their license as well as the runtime’s.

A server that fits one short prompt may run out of memory under several long conversations. Choose the exact checkpoint, quantization, context and concurrent traffic together before comparing its hourly cost with a hosted API.

What changes with vLLM 0.33?

The 0.33.0 release adds models including Reflection Beam-1.0, Step-5 Preview and MiniCPM-OCR 1.0. Its NVIDIA release wheels and images move to CUDA 13.0; CUDA 12.9 wheels and the old -cu129 images are no longer published. The release also moves to PyTorch 2.14.0, torchvision 0.29.0 and Triton 3.8.0. Check driver and hardware compatibility before replacing a working environment.

The shorter vllm/vllm image name is available alongside vllm/vllm-openai. CPU, ROCm and XPU have their own artifacts. Use the backend-specific installation instructions rather than installing an NVIDIA wheel on a different device. A platform’s available package does not imply every model or optimization works on it.

Existing services should also check removed Medusa and MLP speculators, the removed legacy Ray compiled-graph executor and the NIXL 1.5 minimum for that connector. Requests arriving while generation is paused now receive HTTP 503. Keep a known-good deployment while checking your application’s model outputs, request handling and latency on the new version.

Start a small local endpoint

The following Linux NVIDIA example assumes a CUDA-13-compatible driver and enough memory for the selected Qwen3.8-27B checkpoint. The Qwen memory guide explains artifact size, cache and working memory.

uv venv --python 3.12
source .venv/bin/activate
uv pip install 'vllm==0.33.0' --torch-backend=auto
vllm serve Qwen/Qwen3.8-27B \
  --served-model-name local-qwen \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 4096 --max-num-seqs 2

The first launch downloads model files and initializes the engine. Keep the server bound to loopback for this test. Check its models, then make one request from a second terminal:

curl http://127.0.0.1:8000/v1/models
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-qwen","messages":[{"role":"user","content":"Give three checks for a saved backup."}],"max_tokens":256}'

Use the served name in the request. Model-specific chat templates and reasoning or tool parsers affect how text and tool calls are returned. Verify a plain answer before connecting an agent. For an application outside the machine, add an authenticated access path and network controls before exposing the service.

The LLM CLI can register this endpoint for repeatable prompts. Model serving and the client remain separate processes.

Reserve memory for live requests

The memory budget includes weights, KV cache, activation and execution buffers, plus any vision or audio components. A MoE model’s active parameter count describes computation per token; it does not reduce the expert weights to that same storage size. Quantization support depends on the checkpoint, backend and GPU generation.

Start with a bounded --max-model-len and --max-num-seqs. Increase them using representative traffic while watching allocation and latency. --gpu-memory-utilization sets the engine’s memory target; it cannot make an oversized model fit without an appropriate placement or quantization strategy. Tensor parallelism spreads work across devices but also adds communication and requires a compatible model and interconnect.

Record time to first token, generation speed and completed requests at the expected concurrency. A short single-user decode measurement does not predict the cost of serving long prompts to several users. Cache reuse, warm-up, output length and queued requests all affect a useful comparison.

Calculate the cost per completed task

A self-hosted service pays for allocated capacity, including idle time. Four GPUs at an assumed $2 per GPU-hour for six hours cost $48 before CPU, storage or network charges. If the service finishes 1,200 acceptable tasks, that GPU portion is $0.04 per task; 300 acceptable tasks makes it $0.16.

Include startup, retries and evaluation in a batch budget. For owned hardware, include measured system electricity and the hardware and operations assumptions appropriate to your project. The text calculator provides a capacity and cost starting point; benchmark the exact runtime configuration for throughput.

For personal use, compare the simpler setup options in Local inference. For image or audio generation pipelines, read vLLM-Omni; it has separate release compatibility and pipeline requirements.

Sources

Checked 2026-11-04: 0.33 release notes and artifacts, quickstart, GPU installation, OpenAI-compatible serving and source and license.