Strata Qwen: why local speed depends on your setup

Strata Qwen: running Qwen3.8-Flash-Next on a PC

The Qwen3.8-Flash-Next model guide covers the checkpoint, quantization choices, thinking controls and running costs. This section focuses on its PC Strata deployment.

PC Strata from Niko1221 specializes in Qwen3.8-Flash-Next MoE variants. Recent user reports describe useful generation speeds on consumer GPUs, including DDR4 systems. The practical question is whether its expert placement and CPU/GPU split improve the model and prompts you already run.

The PC engine supports specific Qwen3.8-Flash-Next variants. Its installation guide describes Windows 10/11 and Linux setup, an x86 CPU with AVX2, and NVIDIA or selected AMD GPUs. The recommended NVIDIA starting point is at least 12 GB VRAM; the guide describes 8 GB configurations as slow. Check the documented AMD card and ROCm path rather than assuming every Radeon card works.

System RAM is a separate requirement. The project’s model-selection table gives these starting choices:

Choose the exact artifact from the supported-model list. The technical guide recommends up to 128K context for regular IQ3 on a 64 GB PC. If you explicitly select a longer context during setup, Strata keeps that setting. Fitting the weights leaves a separate context budget. A large GPU can still struggle when RAM is full or the CPU path is slow.

Allow roughly 80 GB of free disk space as a starting budget, with extra room for optional components, backend downloads and conversion copies. Check the selected variant’s download size before starting. The standard NVIDIA engine uses driver 580 or newer. Version 0.1.39 also documents a separate CUDA 12.9 engine for selected older cards and drivers; follow its older-GPU guide for that experimental path. CPU instruction support, available RAM and driver compatibility are useful checks before a long download.

The local server offers OpenAI-compatible and Anthropic-compatible interfaces for chat or coding clients. Start with a short chat request, then test the tool calls and prompt lengths your coding client sends.

PC Strata memory: where do the weights go?

In the regular PC path, the GPU holds attention, routing and other shared work, plus a cache of frequently used experts. Expert weights live in system RAM; the CPU computes experts that are not served by the GPU cache. A separate SSD-resident n-gram lookup table supplies small retrieved rows.

This division can make a model usable with much less VRAM than its full weight size. It also makes CPU throughput, RAM bandwidth and cache placement relevant to each token. Context adds KV-cache pressure on top of weights. Leaving space for the operating system and your editor can matter as much as choosing a smaller quantization.

The project also documents alternative low-RAM and experimental SSD weight-paging modes. Treat each as a different configuration: record the artifact, expert-residency settings and storage device with the result. The architecture notes explain these paths. Measure CPU load and transfer time as well as disk reads when testing a paging mode.

Why can local Qwen feel so slow?

For the dense checkpoint, see the Qwen3.8-27B guide for its memory budget and the later correction to the long-context slowdown report.

For PC Strata, a small active expert count reduces work per token, while the full expert set still needs storage and a usable execution path. RAM capacity gets the model loaded; RAM bandwidth, CPU execution, GPU cache placement and PCIe transfers affect generation. A long coding prompt adds a separate prompt-processing wait.

A useful warning about measurement comes from llama.cpp issue #27623. For the separate Qwen3.8-27B model, cuiyuxin-gif withdrew a reported deep-context speed cliff after finding that a comparison included prefill time. Rebuilt runs gave 32.4 decode tokens/s at roughly 91K context. Keep prompt processing, generation and total request time as three measurements; each answers a different question about waiting.

Strata Qwen user reports: DDR4, AMD and multi-GPU setups

DDR4 users report faster output, with different results across setups

EmPips reports about 45–70 output tokens/s with IQ3_XXS on a DDR4 system and RX 7900 XTX, compared with a reported 22.5 tokens/s maximum in tuned llama.cpp on the same PC. In replies, the author describes working around 60K context and just under 800 prompt tokens/s. Coding output varies with MTP speculative acceptance.

In the same thread, DragonfruitIll660 reports about 20 output tokens/s with IQ3_S, 16 GB VRAM and 64 GB DDR4. The two users chose different quantizations, and neither provides a complete matching benchmark configuration. EmPips also describes hitching when the GPU was completely full, then suggests leaving VRAM headroom. Strata’s installation guide documents --vram-reserve-mib for that purpose. Choose the reserve for your desktop’s needs and measure its effect on the smaller expert cache.

A custom fork shows why versions and test workloads matter

RealEddoursul’s account starts with llama.cpp improving from 6 to 21 tokens/s after parameter changes, then 27 with IQ3_XXS, and 51 after switching to Strata. The first gain came from tuning the existing runtime; the later steps also changed the artifact or engine.

The author’s September 30 comparison uses a Ryzen 7 9700X, 160 GB DDR5-5600, an RTX 3090 and RTX 5070 Ti on Windows 11. These IQ3_S API results separate upstream engine 0.1.27 from the custom fork:

These are one run per API workload, with each engine generating its own text. The Strata columns use speculative decoding; that llama.cpp build has no MTP graph for this model. The custom fork changes expert placement across the two cards. Its upstream PR #245 was closed without merging, so use the fork’s installation and model-support instructions when reproducing those results.

Setup logs and unresolved slow configurations

A September 30 setup log by kun432 describes Ubuntu 22.04, an i9-13900F, 96 GB RAM and an RTX 4090 with 24 GB VRAM. The author ran the IQ3_XXS variant, used a 128K maximum context setting and logged an engine update to 0.1.29. The printed results included a 75.4 tokens/s calibration and about 78 tokens/s for a 1,718-token generation. Those are measurements from that setup and prompt, with speculative decoding settings included in the log.

The same account records setup friction: an older CMake version failed the build requirement, and a missing Linux binary led to compilation. This matters to a cost estimate because installation time can be substantial even when generation is fast. The log sets a 128K context limit; its measured generation contains 1,718 tokens.

In the October 3 issue #588, yannickloth tested an A3000 12 GB with Swift 1.5 IQ3_XXS and engine 0.1.35. Raising the PCIe-expert fraction from zero to 0.75 increased the reported warm GPU hit rate from 54.6% to 77.7%, while decode fell from 41.0 to 34.7 tokens/s. The author points out that the hit-rate denominator excludes PCIe experts. For this configuration, the larger displayed hit rate did not predict faster generation. The issue remains open.

An October 3 AMD report in issue #590 describes an RX 7900 XTX 24 GB, Ryzen 9 7900X and 96 GB RAM on Ubuntu 26.04, using a HIP 7.10 build and engine 0.1.34. The author reported about 0.76 decode tokens/s with custom settings. The maintainer asked for an update to 0.1.38, setup defaults and a llama.cpp HIP comparison, noting the difference from the documented ROCm environment. The author then retested 0.1.38 with setup defaults: a 40-token reply still took 2 minutes 4 seconds. A llama.cpp HIP test on the same machine reached 134 decode tokens/s, but used Qwen2.5-7B Q4_K_M, so it checks another ROCm workload rather than comparing the same Qwen model. The Strata issue remains open; the backend and host/GPU coordination need further investigation.

Another October 5 report, issue #922, records v0.1.39 IQ3_S on Windows with an RTX 5070 Ti 16 GB, Ryzen 9 9950X and 126 GB usable RAM. With a 128K context setting, roughly 2.85K uncached prompt tokens and 256 output tokens, annD-annD measured 9–10.6 decode tokens/s. The author could not identify the cause. Record the CPU and RAM path alongside the GPU when comparing this result with another machine.

For your own PC Strata benchmark, establish a working default, measure your task, then change one placement or cache setting at a time. Keep prompt length, quantization and runtime version with the numbers. Track prompt-processing time and completed tasks alongside the decode rate.

The v0.1.39 release, published October 4, adds multi-GPU helper optimizations and opt-in concurrent decoding. Its maintainer measured Q2_0 decode about 6% faster than 0.1.38 on one RTX 5070, using ten interleaved pairs and medians. The IQ3_XXS 32K-prompt gain was 18.5% with a 1,500-slot cache, but unchanged with setup defaults; Coder at 32K was about 8% slower.

More request slots can shorten queue waits while leaving less space for cached experts. On that RTX 5070 with Q2_0 and four requests, the last request began after 1.8 seconds instead of 11.2, while aggregate decode fell 11%. For several clients, compare waiting time and total completed work before increasing slots.

The release separates helper-card expert placement from layer splitting. Its multi-GPU guide explains setup for each.

Layer splitting assigns consecutive model layers to each card, with an expert cache per stage and a shared host expert arena. In the guide’s Coder test on an RTX 5080 plus RTX 3090 and Ryzen 9 9950X3D at 32K context, the best split produced 84 story and 110 code tokens/s; the 5080 alone gave 83–87 and 88–105. A slower third RTX 2080 Ti reduced the pair’s throughput. Budget host RAM, CPU work and transfers alongside each card’s expert cache. Use the exact card layout and quantization when comparing a two-card desktop with a multi-user server.

Which Strata LLM repository do you need?

Niko1221/Strata is a Windows/Linux engine for large Qwen MoE models on a PC. It combines GPU execution, CPU expert computation and system RAM. Ikaikaalika/strata explores memory-aware inference on Apple Silicon through MLX and custom execution paths. Each has its own maintainers and installation path.

Use the PC instructions for supported Qwen variants. On a Mac, follow the Apple project’s model-status list and MLX baseline before experimenting with memory management.

A third project, Amperor Strata, also discusses Apple MoE memory paging. Its BUSL-1.1 license has separate production-use conditions. It is maintained separately from the two MIT-licensed projects covered below.

Apple Silicon: an MLX baseline and research paths

Ikaikaalika/strata is an experimental MLX runtime with an MLX-LM loading path and a custom memory governor. Its Python package retains the ollm import name. The active backend has no Qwen3-Next adapter, so check the model-status list for your architecture.

Developers can run the small Llama and DeepSeek test models to check numerical behavior and cache handling. The governor tests exercise memory reservations, pinning, prefetch and eviction. Full-model validation and complete MoE expert paging remain development work, as the status table shows.

For a Mac you use daily, start with MLX-LM in the directory and a supported model that fits. Compare the research path only after that baseline works. Unified memory shares capacity across CPU and GPU, but the operating system and applications still need room.

Strata running costs: budget for the configuration you can use

Separate a machine you already own from one you plan to buy. For an existing PC, the immediate budget may be a RAM or SSD upgrade, electricity and setup time. For a new machine, include the CPU, RAM, storage, chassis and power supply alongside the GPU. For a Mac, price the actual unified-memory configuration you need; budget for weights, context and other applications in that shared memory pool.

Use measured wall power over representative work where possible. A GPU sensor omits the CPU and the rest of the system, which matters for a hybrid runtime. Longer generation, repeated cold starts and failed agent attempts can increase both energy and waiting time. Keep active time and idle time separate if the machine stays on between requests.

Use the text cost calculator for a supported model’s electricity, operations, system-cost and depreciation budget. Enter measured wall power and runtime for your machine. For Strata’s IQ3 variants or a Mac, take memory requirements from the runtime’s model guide; the calculator’s VRAM result applies to the precision selected there. The method guide and budget notes explain those inputs.

To compare against an API, run the same representative task, record input and output tokens, and count successful completions and retries. Divide the monthly budget by successful completions to compare cost per task. Include retries in the time and token totals.

Estimate a supported model’s running cost

Start Strata locally: repository to first request

For PC Strata, first check CPU instructions, driver, RAM and free disk space against the installation guide. Select the supported model variant before downloading. Use START-HERE.bat on Windows or the repository’s setup.sh flow on Linux, following its current instructions. Keep the default configuration for the first measurement.

The documented OpenAI-compatible base URL is http://127.0.0.1:8080/v1. Start on the loopback listener and send a short request from your local client. If you expose the service to other machines, follow the server’s authentication and network configuration instructions before changing the bind address. Then test a realistic prompt and record startup, prefill and decode time separately.

For Apple Strata, read MODEL_STATUS.md, the Python requirements and the example matching your model architecture. Start with the MLX baseline. Use the matching model example before changing the memory-governor settings. Choose the Apple Silicon directory options when you need a supported daily-use runtime first.

Strata alternatives for a different workload

Use Ollama or llama.cpp for a broader choice of local models and conventional setup paths. Use MLX-LM for Apple-focused generation and fine-tuning. For concurrent users and a shared API, compare vLLM and SGLang under your traffic pattern. For several existing devices, investigate exo with the actual network.

The model directory helps with model-specific uses and access. Check both the runtime’s supported architecture and the chosen weights’ license. An engine license does not change a model’s terms.

Strata LLM questions

Are the PC and Apple projects the same software?

They are independent repositories with different maintainers. Use Niko1221/Strata for its supported PC Qwen variants and Ikaikaalika/strata for the Apple research implementation.

Does PC Strata make system RAM unnecessary?

The regular expert path depends on system RAM and CPU computation. Match the exact variant’s RAM requirements; experimental low-RAM or SSD modes need their own measurements.

Will Strata Qwen be faster than llama.cpp on my PC?

The DDR4 report and dual-GPU comparison show gains on their setups. Compare the exact weights, actual prompt length and completed task on your machine. Upstream Strata and a custom fork can use different expert-placement and speculative-decoding paths.

Does a fast Strata result mean the coding task succeeded?

In the DDR4 discussion, HopefulConfidence0 reports that IQ3_XXS Flash-Next reported files as created while the logs showed no files were created, while Qwen3.8-27B Q8 performed better on that task. Other replies prefer the lower-bit Flash-Next variants. Verify the files, tool results and accepted output in your own client; include retries in time and cost.

Is Apple Strata ready to page a large MoE model from SSD?

The reviewed model-status document describes scaffolding and further routing/paging work. For a full-model Mac baseline today, use a supported MLX-LM path and measure the memory your workload needs.

How do I estimate Strata electricity costs?

Measure wall power during a representative task and enter it with your monthly runtime and local electricity rate in the calculator. Wall power includes the CPU, RAM and storage used by the hybrid engine.