Local inference

Choose a local LLM runtime for your hardware

Your machine, model and workload determine which runtime makes sense. These eight open-source projects cover personal PCs, Macs, GPU servers and clusters; two unrelated projects share the Strata name.

Reviewed 2026-10-21.

Choose a local LLM runtime for the machine and the job

For one person chatting or running a coding assistant, a convenient installer and reliable model loading may matter most. For an API used by several people, response latency, queueing and concurrent requests become part of the cost. For a model larger than GPU memory, ask where the remaining weights live and which processor evaluates them. Choose the runtime for that workload as well as for the available hardware.

Start on hardware you already own. Record RAM, VRAM or unified memory, operating system, driver and free disk space. Then choose a supported model artifact and a realistic context length. A quantized model can reduce weight storage, but a long prompt still needs working memory and time. Closing other applications may be enough for one configuration; another may need a smaller model or a different execution path.

The cards below cover each project’s role, model support and memory strategy. Follow its installation link for the supported devices and backend setup. Project licenses cover runtime code; model weights have separate terms. See model licensing before choosing weights for a commercial application.

Apple Silicon

For an Apple Silicon runtime, start with MLX-LM and a supported model that fits your Mac. Apple Strata is relevant when you want to investigate memory budgeting and the implementation itself.

MLX-LM

ml-explore/mlx-lm

Python tools for generating text and fine-tuning language models with MLX. A practical baseline for a supported model on a Mac.

Hardware / OS: Apple Silicon and the MLX ecosystem. Match the model architecture and quantized artifact to the installed MLX-LM version.
Supported models: Architectures supported by the installed MLX-LM release, with matching MLX weight artifacts.
Memory strategy: Uses the Mac’s unified memory, with quantized model support. The shared pool still has to accommodate weights, context, the operating system and other applications.
Current use: Run generation and fine-tuning through Python or the CLI, with quantization support for compatible MLX models.
Next step: Run a supported MLX model that fits your Mac, then measure startup, prompt processing and generation separately.

Installation MIT license

Strata · Apple Silicon

Ikaikaalika/strata

A research runtime exploring memory budgets and model execution on Apple Silicon, built around MLX and an oLLM-derived Python package.

Hardware / OS: Apple Silicon, Python and MLX, using the model adapters listed in MODEL_STATUS.md.
Supported models: Architecture-specific MLX-LM baseline and custom Llama / DeepSeek fixtures; active Qwen3-Next support is absent.
Memory strategy: Tests a memory governor, reservation and prefetch paths. Complete MoE routing with production SSD expert paging is still a development target.
Current use: Experimental MLX baseline and small correctness tests. Native Metal/ANE execution and production serving are development targets.
Next step: Choose a listed model adapter and start with the MLX baseline; the Apple guide explains the custom research paths.

Read this Strata guide Installation MIT license

PC GPUs

Choose a PC runtime: Ollama for model management, llama.cpp for backend and placement control, or PC Strata for its supported Qwen MoE variants. Check system RAM as well as VRAM.

Strata · PC / Qwen

Niko1221/Strata

A specialized engine for Qwen3.8-Flash-Next MoE variants, with a local API for chat and coding tools.

Hardware / OS: Windows or Linux on an x86 CPU with AVX2; NVIDIA and selected AMD GPUs. The model guide recommends IQ2_XS at 48–64 GB RAM; regular IQ3 variants need roughly 64 GB alongside GPU memory, with less room for context.
Supported models: Listed Qwen3.8-Flash-Next variants; Swift and pruned Coder families have separate artifacts and requirements.
Memory strategy: Keeps expert weights in RAM, computes some experts on the CPU and caches hot experts on the GPU. Its SSD lookup table and experimental weight-paging modes serve different purposes.
Current use: Runnable PC engine with independent setup logs and active troubleshooting. Model variant, CPU, backend and engine version strongly affect results.
Next step: Match the exact Qwen variant to your RAM and GPU, then compare the speed reports and slow-inference checks.

Read this Strata guide Installation Source snapshot MIT license

Ollama

ollama/ollama

A model manager and local API server for a personal machine. Pull a supported model and use the same service from a terminal or application.

Hardware / OS: macOS, Windows and Linux. GPU acceleration requires a supported device, driver and backend.
Supported models: Models listed in the Ollama library or supported import formats; check architecture support for custom weights.
Memory strategy: Manages model loading and GPU placement through its inference backends. Choose a model and quantization that leave room for the context and other applications.
Current use: Run local chat from the CLI or connect an application to its API; model loading and downloads are managed by the service.
Next step: Try one listed model on your existing machine before changing the server settings or buying a GPU.

Installation MIT license

llama.cpp

ggml-org/llama.cpp

A general inference engine and server for GGUF models, with explicit control over context, quantization and CPU/GPU placement.

Hardware / OS: CPU, Apple Silicon through Metal, and GPU backends including CUDA, HIP, Vulkan and SYCL. Build instructions and supported operations vary by backend.
Supported models: Supported architectures in GGUF format, including quantized conversions; model support is backend-dependent.
Memory strategy: Quantized weights and partial GPU offloading let you trade RAM, VRAM and speed. Context also consumes memory, so a model file smaller than RAM is only the first check.
Current use: Use its CLI or server to control context length and the layers offloaded to a GPU, or to compare different GGUF quantizations.
Next step: Use the documented backend for your machine and measure the same prompt and context before comparing a specialized engine.

llama.cpp v0.6.0 adds Clef text and vision decisions through /v1/systemone. Choose a matching GGUF and, for images, a projector. The full prompt must fit the physical batch. Its up-to-3× Metal result concerns specific matrix kernels; measure full request latency on your hardware. Clef GGUF selection and first local request

Installation MIT license

Self-hosted GPU servers

For a shared API, choose a serving engine and test the number of requests it can complete within your latency target. vLLM and SGLang address this serving workload with hardware-specific installation paths.

vLLM

vllm-project/vllm

A serving engine for higher-throughput deployments and concurrent requests, with an OpenAI-compatible server.

Hardware / OS: Hardware-specific installation paths for NVIDIA, AMD, Intel and CPU platforms. Check the supported model and accelerator combination before provisioning a server.
Supported models: Supported language and multimodal architectures, subject to the selected accelerator backend.
Memory strategy: Paged KV-cache management allocates memory for request state, while continuous batching schedules concurrent requests. These optimizations target shared-server throughput.
Current use: Deploy an OpenAI-compatible endpoint and benchmark request latency, queueing and throughput at your expected concurrency.
Next step: Benchmark your expected arrival rate, context and latency target, then budget for utilization and capacity at that service level.

Installation Apache-2.0 license

SGLang

sgl-project/sglang

A serving framework for language and multimodal models, including structured output and workload-aware scheduling.

Hardware / OS: Backend-specific recipes cover NVIDIA GPUs, AMD Instinct, Intel and Apple platforms, each with its own installation requirements.
Supported models: Supported language and multimodal models; check the installation and model-support recipe together.
Memory strategy: Repeated prompt prefixes can reuse cached work; traffic patterns and available cache memory determine the benefit.
Current use: Serve shared APIs with prefix caching, structured output and request scheduling on a supported backend.
Next step: Choose the documented backend, then compare response latency and completed requests under realistic concurrency.

Installation Apache-2.0 license

Multiple devices

A distributed runtime can pool resources from machines you already own. It also needs a network fast enough for the chosen model placement. Measure communication overhead before treating combined memory as one larger GPU.

exo

exo-explore/exo

A distributed runtime that discovers devices and splits model work across a local cluster.

Hardware / OS: Apple Silicon via MLX; the current Linux path supports CPU execution while Linux GPU support is under development. Networking and device capabilities matter alongside total memory.
Supported models: Models supported by the current distributed runner; architecture, device mix and placement determine viability.
Memory strategy: Distributes model work across machines. Combining memory also introduces communication, placement and synchronization costs that a single-machine fit estimate misses.
Current use: Run a local cluster with device discovery and model placement across its available machines.
Next step: Measure startup, generation and network traffic on your device mix, then compare the running cost with a single machine.

Installation Apache-2.0 license

CPU runtimes: a useful baseline, with a waiting-time trade-off

llama.cpp is a useful starting point for CPU-only inference and partial GPU offloading. Check your CPU instruction set and the model’s GGUF support. A smaller model can make a personal assistant more responsive than a much larger model that spends every token moving data or evaluating experts on the CPU.

Run the task you actually need: a long document, a coding conversation or a batch of short prompts. Record prompt-processing time and generation separately. A short question may look fast while a large repository prompt is slow. For a hybrid engine such as PC Strata, CPU and RAM bandwidth remain relevant even with a capable GPU.

Connect the runtime choice to your running cost

Use the text cost calculator for a supported model’s electricity, depreciation and operations budget. Enter your machine’s measured power and running hours. Take memory and placement requirements from the runtime’s installation guide, especially for CPU offloading, Mac unified memory or SSD paging. The calculation method explains the weight-based capacity estimate.

If you already own the machine, separate any upgrade from the electricity and time spent running it. If you are buying a system, include RAM, CPU, storage, power supply and chassis as well as the GPU. Use measured wall power where available: GPU power alone misses the CPU and the rest of the machine. The budget notes explain the available adjustments.

Measure how long the runtime takes to finish the task, including cold starts, prompt processing and retries. Add the time spent configuring the environment. Compare the same task and acceptable output quality before converting a reported token rate into a saving. For a shared server, run the benchmark with the expected concurrent traffic and measure utilization and response latency.

PS5LM: a console deployment case

PS5LM runs a native GPU inference app on a jailbroken PlayStation 5, with supported firmware listed as 7.00–13.60. It is a device-specific experiment for an already compatible console. Use the project’s console instructions and supported model files; a normal retail console cannot follow an ordinary desktop llama.cpp installation.

The maintainer’s README reports about 20 generated tokens per second for Qwen3.8 27B in UD-Q2_K_XL, with a 9.2GB model and a 64K context setting. Its separate console notes use UD-IQ2_XXS with a 4,096-token context: roughly 82 seconds to load, 35–50 prompt tokens per second, and around 21 decode tokens per second on a short prompt, falling to about 9 after the UI’s 371-token prompt. These are different configurations; the short-prompt result is not a 64K throughput measurement.

The GPU app’s unified heap can reach about 11.44GiB with the documented driver option, while the UI uses about 0.35GiB. The older CPU payload’s roughly 1.6GB allocation limit describes a different execution path. A reported 86W SoC figure excludes the rest of the console, so measure power at the wall before estimating electricity. Read the Qwen3.8 27B guide for model context, then compare completed tasks and loading time on hardware you already own.

Sources: maintainer README and console configuration notes, checked October 17, 2026.

Local inference setup and running-cost guides

Estimate a supported model’s running cost

Choose one runtime baseline before adding complexity

For a first local application, choose a supported model through Ollama, llama.cpp or MLX-LM and keep the configuration simple. For Qwen3.8-Flash-Next on a PC, read the Strata Qwen hardware and speed guide, including the DDR4 reports and slow-inference checks. For a shared endpoint, start with a serving engine’s supported backend. For several devices, test exo with the network you will actually use.

Save the model revision, quantization, runtime version and settings with your measurements. That makes a later upgrade or comparison useful. Browse the model directory for model-specific access and usage notes; runtime support and model availability are separate choices.

DeepInfer for DeepSeek cluster inference

DeepInfer is DeepSeek’s offline inference core for V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Read the DeepInfer setup and cluster-cost guide for supported toolchains, community build reports and the components needed for a complete service. The DeepSeek model guide covers application and hosted API choices. DeepInfer summary checked 2026-10-23.

TurboFieldfare for an Apple Silicon memory budget

TurboFieldfare streams Gemma 4 26B-A4B experts from SSD while keeping its shared core and request state in unified memory. Read the TurboFieldfare Mac setup and running-cost guide for platform requirements, the maintainer’s M2 measurements and the difference between process footprint and complete session cost. Checked October 26, 2026.