Qwen3.8-27B: Local Memory, Context and Running Costs

The exact checkpoint is Qwen/Qwen3.8-27B. Its language model is dense: it uses the main language weights for each token, rather than routing through a small subset of experts. Qwen describes a 27B language model with a vision encoder. The calculator uses a 27.8B planning figure for the larger checkpoint footprint, including auxiliary components.

Its 64-layer backbone mixes Gated DeltaNet and full attention, with full attention on every fourth layer. The recurrent layers keep a compact state; the attention layers still retain conversation-dependent cache. This makes context memory different from an equally sized model with full attention at every layer, but you still need room for history, images and runtime buffers.

Ollama currently lists an approximately 18GB qwen3.8:27b download. Its llama.cpp artifact uses Q4_K_M language weights and a BF16 vision projector; the tag also lists an MLX artifact for supported hardware. Save the actual backend and artifact details with a benchmark, because the same tag can select a different execution path.

A 24GB GPU is a reasonable machine on which to try this download with a short conversation. The file size leaves several gigabytes for cache and execution, but the usable space also depends on display use, the backend and the image workload. On a 16GB card, that approximately 18GB artifact needs partial offload or a smaller compatible quantization. Use available memory, rather than the GPU’s advertised capacity, when choosing the first configuration.

Raw weight arithmetic gives another planning view: the calculator’s 27.8B figure is about 55.6GB at two bytes per parameter, 27.8GB at one byte and 13.9GB at half a byte. These are weight-only estimates in decimal GB. Mixed precision, quantization metadata, auxiliary weights and runtime allocation explain why a real 4-bit artifact can be larger than 13.9GB.

GGUF is a file format for storing model weights and metadata; it is not a quantization level or an inference engine. Q4_K_M is a particular 4-bit quantization used by the llama.cpp artifact in Ollama’s qwen3.8:27b tag. That tag also includes a BF16 vision projector and a separate MLX artifact, so the tag’s approximately 18GB download is not a generic GGUF size. Check the exact artifact and runtime your client selects.

For this calculator, choose GGUF Q4 when estimating the listed Q4_K_M local artifact. Choose another precision only when you plan to use a matching weight version. The estimate helps compare capacity and cost; the downloaded file and runtime determine the actual memory footprint.

Install a current Ollama release and run ollama run qwen3.8:27b to download the model and open a local chat. Begin with a short text question. Once that works, send the API request below from another terminal. It asks for three bug-fix checks, requests non-thinking output and starts with an 8K context budget.

The response includes the answer and timing counters. Keep it with the model tag, Ollama version and machine details. If the first response includes loading time, make a second request after the model is resident and label the two runs separately. A cold start and a warm follow-up feel different in a desktop app.

For an image task, add the image through a client that supports Ollama’s visual input format. Ask for a checkable detail such as a chart label, then compare the response with the original. A coding client also needs to execute returned tool calls; chat access alone supplies no file or shell operations.

Thinking is enabled by default in the official template. For a short extraction or formatting task, try the non-thinking mode first. For a difficult repair, compare the result with thinking enabled and a larger output allowance. Record both reasoning and answer tokens when evaluating waiting time, since a short final answer can follow a much longer generation.

Ollama exposes thinking through the think request field; /api/show reports the values supported by the installed model. The official model card’s OpenAI-compatible examples use chat_template_kwargs.enable_thinking. Copy the request format for your chosen server. Sending a setting in the wrong place can leave the default behavior unchanged.

For tool use, try a function with a known result before a repository session. Check the selected function, arguments, returned result and final answer. Then give the coding client a small task whose success is visible in the filesystem and tests. Preserve the server’s chat template and reasoning parser when updating it, because those choices affect how the client receives the model’s output.

The native 262,144-token window includes input and generated output. The one-million-token setting uses the documented RoPE scaling configuration. Choose that extension when your workload needs it, and test shorter prompts as well. A maximum context flag describes capacity; record the actual input length to make a speed result useful.

A widely discussed llama.cpp report initially described a severe slowdown beyond roughly 80K context. The author’s August 26 correction says that conclusion was wrong and the original cliff could no longer be reproduced. The slow measurements had divided generated tokens by total request time, including prompt processing. A rebuilt original commit produced 32.4 decode tokens/s at a 91K position.

Use separate counters when investigating your own delay. In Ollama, generated tokens/s is eval_count divided by eval_duration in seconds. Prompt processing uses prompt_eval_count and prompt_eval_duration; total_duration measures the complete request. Keep cached input separate when comparing repeated prompts. The full request duration remains useful for the person waiting, while decode-only timing helps identify the generation path.

tczank later tested a community IQ3_XXS-MTP artifact on an RX 7900 XTX with llama.cpp b11065 and local tuning patches. The comparison used the same machine, prompt positions and decode-only metric, with speculative decoding off. At a 91,842-token position, Vulkan reached 39.51 tokens/s and HIP reached 14.10. HIP processed the prompt faster in that comparison.

That split helps explain why two users can describe the same GPU as fast or slow. A fresh long document emphasizes prefill; an interactive follow-up emphasizes generation and cache reuse. For your test, use one uncached document request and one follow-up, then compare total waiting time on the backend you intend to keep.

MTP introduces another variable. A speculative decoder proposes tokens and verifies them against the main model. Its benefit depends on acceptance and implementation, with extra memory and work for the draft path. Compare MTP on and off on the same task, and read the runtime’s acceptance counters rather than assuming that loading the MTP weights enables a speedup.

For a model server, start from the vLLM 27B recipe and a compatible quantized checkpoint. One documented RTX 5090 setup uses Inferact’s NVFP4 conversion, a 32K maximum context, FP8 KV cache and eager execution. In that specific test, disabling CUDA graph capture was necessary for startup. Its memory settings belong to that conversion and runtime build.

The recipe also includes two-card RTX 5090 configurations, official FP8 weights and MTP acceptance measurements. Choose a setup according to concurrency and useful context, then measure requests/s, queue time and output rate under that load. Reserve visual input tests for the path that actually includes the vision encoder; a text-only launch deliberately removes that component.

Start with 27B when your tasks fit a quantized desktop setup and you want a straightforward runtime. Try Flash-Next with Strata when you have enough system RAM and a task that could benefit from the larger model. Run the same bug fix, document question and tool workflow in both, keeping the client and verification checks fixed.

HopefulConfidence0’s community comparison favored 27B Q8 over IQ3_XXS Flash-Next on a file-creation task. The latter reported files that the tool log showed it had never created. Other contributors favored their Flash-Next runs. Those differences make quantization and actual task completion worth comparing together: inspect the patch, output file or cited evidence before counting a successful answer.

Use the calculator’s exact 27B selection to compare its available precisions and deployment choices. Keep the hardware, context and measured wall power with the result. At an illustrative 250W and $0.15/kWh, two hours costs $0.075 in electricity; thirty such days cost $2.25. Add ownership and maintenance costs, then divide your complete workload cost by accepted tasks, including retries and review time.

For answers grounded in a document collection, use EmbeddingGemma 2 to index and retrieve relevant chunks, then supply those chunks to this model with their source identifiers. Keep retrieval quality and answer quality as separate checks: the correct passage can be found while the generated answer still misreads it.

Budget the first corpus embedding pass, later index updates, vector storage and generation inference separately. Changing embedding checkpoints requires a compatible new collection; lowering vector dimensions should be evaluated against labeled queries before it becomes a storage saving.