Mellum 2.1: GGUF Setup, Memory and Coding Costs

Mellum 2.1 fits a repository assistant or a smaller worker inside a coding system. Give it the relevant code, the failure and an acceptance test. A local chat runner is enough for an explanation; editing files and running tests require an agent harness that exposes those tools.

For a first trial, paste a short function and its failing test into chat. Check whether the explanation identifies the cause, whether the proposed patch changes unrelated behavior and whether the test passes after applying it. Keep automated tool execution separate until the model and runner behave as expected.

This is the Thinking checkpoint. Its responses can include <think> reasoning before the final answer, so allow output room for both and check how the client displays or parses them. It is a text model; use a vision model when your task starts from a screenshot.

The JetBrains GGUF repository contains single-file variants of the same Thinking model. The official files are available to download. Download one variant rather than the entire collection.

Q4_K_M is the variant used in the startup examples below. MXFP4_MOE is smaller, while Q6_K, Q8_0 and BF16 take more storage. The card’s quantization metrics compare logits on Wikitext-2 at 512-token context, not passed coding tasks. Check code quality and runtime support for your chosen format.

These decimal-GB figures are download sizes from the official file listing, not measured RAM or VRAM. Keep free disk space for the selected file and any download cache.

Use a current runner and backend that support the Mellum architecture and your selected quantization. The following paths use the official JetBrains artifact; an older runtime can reject a model even when its file fits on disk.

The full 12B model still needs a storage and memory-placement plan. The 2.5B active figure describes computation per token; it does not turn the downloaded checkpoint into a 2.5B-sized allocation. Runtime buffers, KV cache and other applications need memory beyond the selected file.

Load at a smaller context first and watch peak memory. Increase it only when the document or repository task needs it. The advertised 131,072-token context is a model limit, not proof that every laptop can serve that length. CPU offload can shift the allocation to system RAM and change latency.

Record system RAM, VRAM or unified memory, runner version, quantization, context, offload settings and concurrent requests. If loading fails, first check runtime support and available memory; avoid treating an unknown-architecture error as evidence that the quantization needs more RAM.

JetBrains’ own agent evaluation reports 47.0% on SWE-bench Verified for Mellum 2.1, compared with 2.0% for Mellum2 Thinking and 50.0% for Qwen3.5-9B. Those runs use Pi v0.73.1, shell/file tools, a 114K context and up to 16K output tokens per turn. They are publisher results, not an independent test of the GGUF you download.

The launch speed chart uses an H200 and includes multi-token prediction results. The MTP head remains described as forthcoming in the official model card. Do not use the advertised MTP speedup as the default throughput of a downloaded GGUF or an ordinary server without that head.

The main model card’s separate vLLM tool-use recipe selects qwen3 for reasoning parsing and hermes for tool-call parsing. Use the recipe for the original checkpoint when building that serving path, and validate tool calls through the client before giving an agent a larger repository task.

The weights use Apache 2.0 terms. Local inference has no model-provider token bill; include hardware, electricity, storage and elapsed time in the budget. A rented server or hosted endpoint has a separate tariff.

For an assumed 200W whole-system draw over three hours at $0.20/kWh, electricity is 0.2kW × 3h × $0.20 = $0.12. This is a budget example, not a measured Mellum run. Count loading, reasoning, tool waits and failed attempts in the time you allocate to the task.

For a cloud deployment, multiply the full allocated instance rate by its reserved hours, including startup and idle time. Compare that amount with accepted fixes over the same task set. File size, active parameters and a vendor H200 chart cannot supply an unmeasured completion rate for your own hardware.