Kolibri 1: Local Hardware, Reasoning Cost and Sovereign AI

Use Aleph-Alpha/Kolibri-1 for the FP8 checkpoint described here. Aleph-Alpha/Kolibri-1-BF16 is a separate weight format, and Kolibri Origin is an earlier model. Keeping the checkpoint name in your deployment record makes memory and quality comparisons easier to reproduce.

The 50-layer model routes each token to six of 384 experts per layer, alongside one shared expert. Active parameters describe the computation selected for a token. Loading the checkpoint still requires storage for the full expert set. This is why a model with roughly 3.46B active parameters has a much larger weight footprint.

Aleph Alpha lists about 78GB for the FP8 weights and about 156GB for BF16. Its minimum and recommended GPU arrangements are shown below. These are weight-format-specific deployment recommendations; reserve additional memory for the KV cache, runtime buffers and the concurrency you intend to serve.

Start with a context limit that matches your documents. Then measure peak memory and time to the first token before increasing simultaneous requests. A long context and several active users can change the useful capacity of the same GPU arrangement. A consumer-machine trial needs a compatible conversion and runtime, with its own memory measurements.

Community quantization offers a separate consumer-hardware path, with smaller files and runtime requirements described below.

Aleph Alpha offers downloadable weights and enterprise deployment options through its sales team. Its Kolibri product page and model card do not publish hosted per-token prices. The model card’s OpenAI-compatible localhost endpoint connects to the server you deploy. For a managed service, ask sales for the endpoint, deployment terms and pricing; for self-hosting, budget the GPU service, storage and operating time.

The aleph-alpha-inference package adds Kolibri support to vLLM and installs its supported version. The repository currently documents vLLM 0.29. Use its kolibri1 parsers for reasoning and tool calls. The command below follows the official FP8 setup; multi-GPU deployments also need the parallelism configuration appropriate to their hardware.

Once the server is listening on port 8000, send a short document-grounded request. This example selects low reasoning effort and the model card’s recommended sampling settings. In an application, the client executes requested tools and returns their results to the conversation; the model server supplies the generated calls.

Thinking is enabled by default. Select none, low, medium or high through chat_template_kwargs.reasoning_effort; enable_thinking=false also turns thinking off. Try the same set of tasks at all four levels. Record reasoning and answer tokens, time to the first token, total duration, tool-call success and whether the result passes your acceptance check.

Use task completion to choose the setting. A faster decode rate can still accompany a longer session if the model generates more reasoning or repeats failed tool calls. Keep retries in both the time and cost totals. For a document assistant, acceptance can require a supported answer and the correct source passage; for code, use tests against the resulting patch.

For an illustrative rental budget, suppose the complete GPU deployment costs $4 per hour. If 100 attempts consume 25 minutes of billed deployment time and 90 results are accepted, that batch costs about $1.67, or $0.0185 per accepted result. Add billed idle time, storage and application work when estimating a monthly deployment.

Reasoning uses the completion budget too. In an independent DGX Spark deployment, the author observed requests ending with content: null when max_tokens was exhausted while the model was still reasoning. Size max_tokens for thinking plus the visible answer, or disable/reduce reasoning when the task does not benefit from it.

Kolibri’s final long-context training stage used 262,144-token sequences. Aleph Alpha reports validation up to 1,048,576 tokens and recommends staying at or below 262,144 for latency-sensitive serving and complex tasks. Most attention layers use a 512-token sliding window; every fifth layer attends across the full context.

The official serving instructions extend the context by setting --max-model-len 1048576 and --hf-overrides with max_position_embeddings set to 1048576. Increase this after measuring your workload. For retrieval over a document collection, selecting relevant passages may give a shorter prompt and a lower waiting time than sending the whole collection on every request.

The DGX Spark measurements below show how first-token wait changes with context depth; they do not measure serving at one million tokens. Include prompt-processing wait time when comparing long-context deployments.

Aleph Alpha’s post-training table reports the scores below. The mathematical reasoning and code-generation results make those useful trial tasks. Its interactive terminal and multi-turn tool results warrant a separate test of the complete agent workflow. Each row uses its own benchmark protocol, so the numbers do not form one common success-rate scale.

Kolibri uses a 128,000-entry UniBPE vocabulary tailored to German and English. Tejas Kumar tested the complete German Basic Law and its official English translation using each model’s tokenizer. He reports 35,190 German tokens for Kolibri versus 41,482 for o200k_base, about 15.2% fewer in that document. The English counts were 39,875 and 39,737 respectively.

This is a count of one legal text, useful when sizing similar documents. Tokenize your own material before selecting chunk sizes and context limits. Fewer tokens let more text fit in a window and change prompt-processing work; the actual time and money saved also depend on the runtime, attention pattern and number of requests.

An independent NVIDIA Developer Forum report documents Kolibri 1 FP8 on a DGX Spark / GB10 with vLLM 0.29.0, CUDA 13.0 and aleph-alpha-inference 1.0.0. The author reports roughly 73.5 GiB of weights. The llama-benchy test used a 2,048-token prompt and 128 generated tokens, with two runs at concurrency 1; the table shows context depths of 0, 16K and 32K.

The same report describes an 18–25 minute cold start, with one weight-loading measurement around 1,087 seconds. Account for that startup wait when choosing between scale-to-zero and an always-warm endpoint.

These measurements come from one community deployment. The published compose file was configuration-checked rather than run end to end. Link to the original runtime and benchmark records when comparing your own results.

A community GGUF conversion provides Q4_K_M at about 47.5GB and Q8_0 at about 83.1GB. For one 67-token chat prompt, the conversion author reports top-1 agreement with an independent float64 reference at 67/67 positions for Q8_0 and 63/67 for Q4_K_M. The agreement figures compare conversion fidelity on that prompt.

The same project reports CPU-only Q4_K_M generation around 11.9–14.9 tokens/s on a Ryzen 7 7800X3D with 128GB RAM and eight threads for its smoke tests. This makes a non-datacenter experiment possible, but it is community evidence rather than an Aleph Alpha supported deployment.

The author documents a patch against llama.cpp commit 836d571. Stock llama.cpp and apps built on it need Kolibri architecture support to load these files. The reported CPU results use the patched implementation; CUDA, Vulkan, Metal, contexts above 8,192 tokens and multiple or streamed tool calls were not verified.

On October 3, HN user martianvoid reported roughly 170 tokens/s with FP8 weights on an RTX Pro 6000 setup. The same comment described lengthy reasoning even after the model had found an appropriate approach. It names the GPU and precision but leaves the runtime revision, prompt length, batch size and reasoning setting unspecified.

That observation gives a useful question for a local trial: does reducing reasoning effort shorten the total task while preserving an acceptable result? Keep the prompt, runtime and GPU configuration fixed during that comparison, and report completion time alongside tokens/s.

Aleph Alpha’s Merlin-Arthur training includes examples where relevant evidence is present and examples where it has been removed. The official table reports 85.6 on RGB Negative abstention, alongside 14.8 accuracy and a 44.0% non-hallucination rate on the public AA-Omniscience set. These results describe distinct evaluations of answering and abstaining.

HN user Glitch752 supplied an English question about an invented song and artist. Their October 3 comment reports that Kolibri fabricated an album and interpretation. For a document application, include an invented entity, a missing passage and two conflicting sources in your test set. Check the cited evidence and the decision to abstain as separate outcomes.

Aleph Alpha describes sovereignty through control of its training pipeline and customer deployment. Its launch article says the model was developed in Germany and trained on infrastructure in Germany and Finland. Apache-2.0 weights allow a team to host the model within its own system; document storage, retrieval, tools and access controls remain application choices.

The HN discussion questions how much of that control comes from downloadable weights. HotHotLava argues that maintaining a training pipeline matters if future suppliers stop releasing models. For a deployment decision, identify which parts you need to control: the checkpoint, inference service, retrieved documents and external tools.

The model card discloses 20T pre-training tokens, 3.44T mid-training tokens and 201B tokens for long-context training on 768 NVIDIA B200 GPUs. The reported 392,000, 90,000 and 10,000 GPU-hours add up to about 492,000 across those three stages. SFT and RL are separate, so that sum is only part of the development compute.

For an organization deploying the released checkpoint, budget the inference GPUs, utilization, document pipeline and maintenance. Use the training disclosure to understand how the model was made; use your task measurements to estimate the recurring cost of running it.