DeepInfer: build and budget a DeepSeek inference cluster

DeepInfer is DeepSeek’s open-source inference core for DeepSeek-V4.1-Flash. It gives infrastructure teams the model execution and scheduling code used behind DeepSeek’s services, with an offline entry point for running the model themselves. The supported platforms are NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Start here if you can build for one of those platforms and operate a cluster; for a first application, the DeepSeek model and API guide is a shorter path.

On this page: Open components · Hardware and memory · Installation · API integration · Community deployment reports · Benchmark units · Running costs · Sources

What the DeepInfer release includes

The Apache-2.0 code includes optimized kernels, model execution, speculative decoding and asynchronous scheduling. Its parallel execution paths distribute attention work and model experts across devices. The official scope also lists structured output and deterministic sampling, with platform-specific settings.

Much of the engine support needed for disaggregation is implemented, while the complete service components above remain outside this release. DeepSeek’s deployment Q&A explains that cache management and load balancing depend on its internal infrastructure and are not fully open-sourced. Moving an existing vLLM endpoint therefore involves service integration as well as a runtime build. Check the local inference directory when you need a ready serving workflow or a different hardware backend.

Plan hardware, memory and interconnect together

The documented software prerequisites are Linux, GCC 13 or newer and Python 3.12. NVIDIA builds use CUDA 13; Ascend builds use CANN 9.2. The hardware list names SM100 and Ascend 950 specifically. A GPU’s broad product generation is insufficient to establish compatibility with those kernels.

The published benchmark uses four SM100 GPUs for prefill and 64 for decode, each group within a single scale-up domain. Those are measurement configurations. The documentation does not give a universal minimum device count or total VRAM requirement for the supported checkpoint.

For the online architecture, the deployment Q&A recommends decode expert parallelism of 64 or more and describes 64 prefill GPUs plus 64 decode GPUs as its smallest efficient deployment unit. That is a recommendation for the described large-scale service, separate from the offline benchmark or a universal minimum for loading the model. Include its dedicated cache-storage and network needs when comparing that architecture with your own reservation.

Size the complete workload: resident weights, extra expert replicas, caches, activations and communication buffers, plus host RAM and checkpoint storage. Parallel layout, context length and concurrent requests change that allocation. The offline example sets a 32,768-token model limit, four sequences and 0.85 memory utilization; these are example settings, not a capacity guarantee for an arbitrary cluster. A model’s advertised context window does not establish that the example can serve it unchanged.

Before reserving machines, identify which devices share fast links and how traffic crosses nodes. Record active network interfaces and routing alongside device count. Two reservations with the same number of GPUs can spend very different amounts of time moving experts, activations and cached state.

Build a pinned revision and run an offline sample

The releases page has no packaged release as of October 23, 2026. This source-build example pins the reviewed main-branch commit, 9bd9cf2, rather than relying on a moving checkout. Prepare the platform toolchain above, then create an empty Python 3.12 environment:

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
git checkout 9bd9cf243fc53ef9afbf8f04d391d989117d6ab9
python3.12 -m venv .venv
source .venv/bin/activate
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Use ascend for the Ascend backend and keep DEEPINFER_ACCELERATOR exported when running the engine. The build script installs dependencies and builds backend-specific extensions; budget compilation time and disk space before model startup.

The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017. The first run downloads it into the Hugging Face cache. To use an existing copy, set DEEPINFER_HF_MODEL_DIR to its directory containing config.json and the safetensors files. An explicitly supplied --model_name takes precedence over that environment override.

python examples/offline_inference.py --help
python examples/offline_inference.py --ep 8 --check_structured_output

The second command uses the documented eight-way expert-parallel example. Choose a layout that matches your available devices and memory; eight is not a published minimum. The script generates bundled text and image samples and prints output-token counts, draft acceptance and decode perplexity. The added check flag also exercises regex, JSON and EBNF constraints and normal completion. Compare a failed check with the platform-specific baseline before starting a larger sweep. Completing this offline test establishes a starting point for engine integration; it does not start an HTTP service.

Connect API formats to the inference backend

deepseek-recipe is DeepSeek’s companion library for converting Messages, Chat Completions and Responses requests into a common conversation, rendering model prompts and parsing generated output. It includes Rust libraries and Python bindings. The caller supplies inference, tool execution and HTTP transport; the repository’s sample API servers use mock inference.

For an integration, trace a request through prompt rendering and tokenization, generation settings, engine output and streamed response parsing. Match the checkpoint and tokenizer, thinking mode, stop handling and output limits. With structured output, the converter can carry a constraint, while the inference backend must enforce it. These checks determine whether your application receives the intended behavior; accepting the same request format alone does not reproduce the hosted API.

What community deployment reports reveal

Missing build packages on a clean NVIDIA container

On October 21, contributor reyanshsolis reported a source build at the older 774ce24 revision in a CUDA 13.3.1 Ubuntu 24.04 container. Missing libdw-dev caused an elfutils/libdwfl.h failure in DeepEP; missing pciutils caused an lspci failure during DeepSelect setup. The author obtained a build and import after adding packages, on a build host without a GPU. That verifies compilation in that environment, not model execution.

Use those error signatures to inspect system dependencies when a build fails. The reviewed 9bd9cf2 commit subsequently changes the DeepEP dependency, so compare the failing vendor revision before applying an older workaround. The same report used a compiler spill-check bypass; it supplies no GPU performance validation for that bypass.

B200 and GB300 reproductions depend on topology

Contributor kzjeef’s October 21 benchmark reproduction proposal links separate four-B200 and four-GB300 runs. The B200 report uses a modified fork and EP4/DP4. The GB300 report attributes substantial waiting to a node with inactive ConnectX-8 links and traffic sharing one 100Gbps bond. The reports remain an open contribution rather than upstream deployment instructions.

The public cases here cover compilation without a GPU run and SM100 inference on modified forks. Independent confirmation of an unmodified upstream run on each supported platform remains limited in these records. Use their logs to investigate a matching setup, then run the upstream offline samples on your own platform before projecting capacity. Keep the hardware, parallel layout and service additions with the result.

Read stage benchmarks before calculating token cost

The benchmark documentation separates prefill throughput from decode steps per second. Its reported decode-token throughput uses full-acceptance accounting for speculative tokens, regardless of the actual draft acceptance rate. Keep that convention with any copied number; a production workload needs its measured accepted output.

Profiling also changes timing. Use the unprofiled measurement windows for a speed comparison and traces to inspect the work inside a step. Record checkpoint and code revisions, context and uncached input, concurrency, parallel layout, speculative settings and active interconnect. Then measure the complete request, including queueing and prefill, before comparing it with an API bill.

Budget the complete DeepInfer cluster session

DeepInfer has no hosted per-token tariff. For rented hardware, multiply the total cluster hourly rate by the billable reservation, including compilation, model loading, warmup, failed runs and idle capacity. Add CPU hosts, storage, networking and transfer where the provider bills them separately. For owned equipment, include the full machine and network cost, wall power, cooling and operating time.

For an arithmetic example, assume the entire reserved cluster costs $200 per hour. A three-hour trial costs $600 in compute before separately billed items and engineering time. If that trial completes 1,000 acceptable tasks, its compute cost is $0.60 per task. This is an assumed budget, not a provider quote or a throughput measurement. The time spent adding cache management, routing and monitoring belongs in the deployment budget too.

Compare that session with the DeepSeek API rates for the same input, output and accepted workload. The text cost calculator explains electricity and hardware-cost inputs for supported profiles; it has no measured DeepInfer cluster-capacity profile. Use your own complete-session measurements and the calculation method before projecting a monthly saving.

Sources and next steps

Reviewed October 23, 2026. Start with the pinned source revision, deployment Q&A, offline examples and evaluation instructions. Bring a working offline configuration into your serving integration, or return to the local inference options to choose another runtime.