AirLLM: run larger models by streaming their layers

AirLLM is a Python inference tool that loads a model’s weights a layer at a time. The checkpoint stays on disk while each layer moves through memory for computation. This can let a larger open-weight model run on a GPU whose VRAM cannot hold all its weights together. The tradeoff is repeated data movement: budget SSD space, host memory and waiting time alongside the GPU.

Original explanation of layer streaming and the resources it uses.

On this page: What changes when AirLLM loads one layer at a time? · Install AirLLM and send a short request · Budget SSD space and host memory · What users found in timed AirLLM runs · What does an AirLLM session cost? · Compare AirLLM with a resident runtime or an API · Sources and next steps

Open the official AirLLM repository. For a runtime that keeps more of the model resident, compare the local inference guide and its Strata setup.

What changes when AirLLM loads one layer at a time?

The implementation splits the checkpoint into layer shards on disk. It loads the weights before a layer runs and frees them afterwards, with optional prefetching to overlap the next load with current computation. That reduces the weights resident on the GPU; it adds work transferring and staging them. Activations, caches and runtime buffers still need memory.

The project advertises a 70B model on a 4GB GPU and publishes model-specific setups. Read those as configurations to reproduce: the checkpoint, architecture, compression, prompt length and machine determine the workload. A model loading successfully answers a capacity question; completion latency and usable output answer whether it fits your task.

For occasional offline analysis, accepting a longer wait may be worthwhile when you already own a small GPU. For an interactive assistant or a server with many simultaneous users, compare a smaller resident model, a larger-memory machine or a hosted API on the same task. Keep the selected precision and serving behavior in the comparison.

Install AirLLM and send a short request

Use an isolated Python environment and follow the repository’s requirements for your checkpoint. The official quickstart uses AutoModel, which selects the implementation from the model configuration. A model requiring a particular Transformers version or additional CUDA packages needs that documented environment.

python -m venv .venv
source .venv/bin/activate
pip install airllm

The example below adapts the project’s Qwen3-32B quickstart for a CUDA machine. It downloads the checkpoint and creates layer shards on its first run. Prepare the cache and free disk space before executing it.

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
encoded = model.tokenizer(
    ["Explain why a delivery date might need manual review."],
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=128,
    padding=False,
)
result = model.generate(
    encoded["input_ids"].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True,
)
print(model.tokenizer.decode(result.sequences[0]))

First confirm that the run reaches a decoded response. Then increase the input and output lengths to match your real task. Separate the download and split time from later inference runs so your comparison reflects both initial setup and repeat use. Keep the checkpoint revision and package versions with the result.

Official quickstart and configuration options explain layer_shards_saving_path, prefetching, compression and supported architectures. Use the corresponding model guide to check its weight license and access conditions.

Budget SSD space and host memory

Both the original checkpoint and split shards can occupy disk during preparation. layer_shards_saving_path chooses where to store the split files. The delete_original option removes originals after splitting; retain them while checking a first setup or when other tools share that cache.

At two bytes per parameter, 70 billion raw weights occupy about 140 GB in decimal units. Keeping a second full-size split representation would bring the raw-weight total to roughly 280 GB, before metadata and temporary files. Use the actual download manifest and compression format for the checkpoint you choose. Large MoE models can store far more weights than their active-per-token parameter count suggests.

System RAM stages data and supports the runtime even when the GPU holds one layer. The exact architecture can add host-resident structures. In the project’s newer model examples, the README lists extra host RAM, disk and package requirements separately. Check those requirements before choosing a machine around VRAM alone.

The FAQ connects MetadataIncompleteBuffer during splitting with exhausted disk space. Inspect the free space and the integrity of partially created shards before rerunning. The project recommends AutoModel when a class mismatch causes max() arg is an empty sequence.

What users found in timed AirLLM runs

In an October 2, 2026 test on an M4 Mac mini, Kunal Vohra used 16GB unified memory and a Crucial X9 USB SSD with AirLLM 2.11.0, MLX 0.29.3 and Transformers 4.46.3. After fixes to that Mac setup, Llama 3.1 8B generated 20 tokens in about 6.5 minutes: 19.6 seconds per token. The results section and timing arithmetic specify tokens; the post’s 70B timings are projections, not a run of that checkpoint. Check the software/backend alongside the hardware when comparing a Mac result.

On August 4, 2026, TyrelCB compared compression modes on a GB10 using Qwen2.5-32B-Instruct, the same prompt and ten output tokens. The post reports 13.3 seconds/token without compression, 50.5 with 8bit and 106.7 with 4bit; split files shrank from 62G to 31G and 18G. For this setup, smaller files came with more processing time. Compare complete task time on your machine before choosing compression='4bit' or compression='8bit' for speed.

What does an AirLLM session cost?

The AirLLM repository is Apache 2.0 software. The selected model has its own license. Hardware, rented machine time, SSD capacity, electricity and setup work are the main deployment costs.

For an illustrative complete-machine rate of $1 per hour, a 30-minute billed session costs $0.50 and a two-hour session costs $2. Count download, splitting, inference and billed idle time in the rental session. Add any separately billed disk or transfer. On your own PC, compare the hardware purchase and whole-system electricity with the time the work occupies the machine.

Cost per accepted result is the total session bill divided by the outputs you keep. If a $2 session yields eight usable results, compute cost is $0.25 each. A lower hourly rate can be outweighed by longer processing or repeated attempts. Measure full-task time instead of choosing from the GPU’s hourly price alone.

The LLM cost calculator estimates ordinary supported deployments. Its model-memory profile does not describe AirLLM layer streaming. Use the session arithmetic above with measured runtime for AirLLM; see the calculator method for what its existing profiles include.

Compare AirLLM with a resident runtime or an API

Keep the task, prompt and acceptance check fixed across those paths. For local runs, record warm and first-run behavior separately, the precision, input/output lengths, peak VRAM, host RAM, free SSD space and wall time. Include unsuccessful attempts when comparing cost per accepted result.

Sources and next steps

Reviewed October 13, 2026. Start with the official README and model-specific requirements, layer-loading implementation, and Apache 2.0 software license.

Choose one checkpoint and a short task first. Once it runs, extend the test to your intended context and response length and compare its full cost with the alternatives. Browse model access and cost guides, local runtime options, and budget assumptions.