smolmaxxing: run larger local models on small devices

smolmaxxing adapts llama.cpp to run local language models on devices with very little free memory. The published examples include a Xiaomi TV Box S (3rd Gen) with about 1 GB of usable RAM and a jailbroken 2013 iPad Air with about 700 MB. Its 16B Ling-mini-2.0 run reaches 5.1 tokens per second on the TV box, according to the project author.

That result uses a custom 2-bit MoE build, a patched runtime and several gigabytes of flash storage for a repack cache. The model file itself is only about 0.47 GB, but the first run creates a 5.3 GB cache and uses about 1.13 GB of process memory. Treat this as an experiment for a supported device, not a drop-in setup for any TV box or tablet.

What smolmaxxing changes

The project patches llama.cpp with kernels tuned for the tested chips, prunes output-vocabulary rows to reduce memory, and streams or pages model weights from flash. For mixture-of-experts (MoE) models, only a subset of experts is used for each token. smolmaxxing keeps a pool of experts in RAM, reads other experts from flash as needed, and prefetches likely next-layer experts.

This reduces resident memory while adding storage traffic and specialized setup. The ready-made model files use layouts that stock llama.cpp cannot load correctly; the TV builds need the matching patched runtime. Some iPad MoE files also need the project's iOS 12 patch because another llama.cpp build can scramble their expert layout.

The published TV-box result uses Android 14 without root. The iPad result uses iOS 12.5.7 on a jailbroken iPad Air (2013). The setup commands below target the TV workflow; they do not turn an unmodified iPad into a supported device.

What fits on the tested TV box?

The Xiaomi TV Box S (3rd Gen) has an Amlogic S7D chip with four Cortex-A55 cores, 1.9 GB of RAM and eMMC storage. Android and background services leave roughly 1 GB available to the model process. In the author's measurements, Qwen3.5-2B is the everyday choice at 7.7 tokens/s; Ling-mini-2.0 is the largest model tested at a usable speed, at 5.1 tokens/s.

The quality column is the author's bits-per-byte score on a fixed 22 KB English text; lower is better. It is a narrow perplexity-style comparison, not a general capability rating. On this test, the 2B model scores better than the 16B build. Choose a model by testing your own prompts and checking whether its answers remain coherent, especially on long outputs.

The author's device speed varies by about 10% over a day. These are specific, sequential measurements on one TV box, with real chat requests through llama-server. They are useful starting points, not expected speeds for another device or a stock model file.

Install the patched TV-box runtime

You need an ARMv8.2 Android device with network ADB enabled. The host computer needs adb, Zig, CMake, Ninja, Python 3, curl and uv. Install the Hugging Face download CLI with uv tool install huggingface_hub. The build below also compiles the memory-guard helper used by the chat launcher. The commands below follow the project's TV-box setup and pin the llama.cpp revision used by its patch.

Clone the project and connect to the TV box, replacing the example address with the address shown by your device:

git clone https://github.com/hyperionaidev/smolmaxxing.git
cd smolmaxxing
git checkout 27b8b8a6b16f6b1ab3316c07324ae34c7548aa9b
export ADB_SERIAL=192.168.1.50:5555
adb connect "$ADB_SERIAL"

Build the patched runtime and copy its executables to the device:

git clone https://github.com/ggml-org/llama.cpp third_party/llama.cpp
cd third_party/llama.cpp
git checkout 8642541
git apply ../../patches/llama.cpp-v0.8.0-tvbox.patch
cmake -S . -B build-tv -G Ninja -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 \
  -DCMAKE_C_COMPILER="$PWD/../../zigcc/cc" -DCMAKE_CXX_COMPILER="$PWD/../../zigcc/c++" -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_EXE_LINKER_FLAGS=-static -DBUILD_SHARED_LIBS=OFF -DGGML_NATIVE=OFF \
  -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16 -DGGML_OPENMP=OFF -DLLAMA_OPENSSL=OFF -DLLAMA_BUILD_TESTS=OFF
ninja -C build-tv llama-server llama-cli llama-bench llama-perplexity llama-quantize
cd ../..
adb -s "$ADB_SERIAL" shell mkdir -p /data/local/tmp/llm/patched25 /data/local/tmp/llm/patched
adb -s "$ADB_SERIAL" push third_party/llama.cpp/build-tv/bin/llama-{server,cli,bench,perplexity} /data/local/tmp/llm/patched25/
adb -s "$ADB_SERIAL" push third_party/llama.cpp/build-tv/bin/llama-quantize /data/local/tmp/llm/patched/
mkdir -p bin
zig cc -target aarch64-linux-musl -O2 -static tools/tv-watch.c -o bin/tv-watch

Download the Qwen3.5-2B model file and start the included chat service:

hf download hyperionaidev/Qwen3.5-2B-tvbox-GGUF --local-dir models/
adb -s "$ADB_SERIAL" push models/w-q4k-h65p8.gguf /data/local/tmp/llm/
tv-llm-ui/start.sh

Open http://<tv-ip>:8090 from a device on the same network and send a short prompt. The launcher deploys the web page and memory guard alongside the model server. Stop the service with tv-llm-ui/start.sh stop. The guard stops the model when available memory falls below its configured reserve; check the device's server log if a request stops under memory pressure.

The 16B Ling experiment is a separate setup: its launcher selects patched42/llama-server, lm2-q2-h60.gguf, moe-warm-ling.txt and the MoE paging configuration. The quick build above produces patched25 for the 2B path, so it does not reproduce that 16B run. Use the model-specific experiment notes and launcher configuration when preparing a matching MoE runtime.

The author's MoE launcher enables headless mode, stopping the TV launcher and Google services and enabling stay-awake to free memory. Headless mode remains enabled after stopping that server. Restore the launcher and screensaver by running sh /data/local/tmp/llm/tv-llm-mode.sh off on the TV.

Why the 16B model needs extra disk and quality checks

For the 16B Ling-mini-2.0 build, the project pages 2-bit experts from flash and gives a small routing bonus to experts already in memory. In the author's measurements, cache-aware routing reduced expert misses from about 13% to about 2%. The routing bonus changes expert selection and has a measurable quality cost; the published setup uses a 0.3 bias for this particular test.

The same report says 2-bit experts can make long responses lose coherence after roughly 250–300 words, and that routing bias makes this worse. A repeat penalty of 1.15 over 256 tokens reduced the author's degenerate long answers from 12 of 13 to 1 of 13. Those figures describe the author's prompts and implementation. Test the exact setup on your own task and include long outputs if that is how you plan to use it.

Storage is easy to underestimate. The 16B GGUF takes about 0.47 GB, but its repack cache adds 5.3 GB on flash; keep at least that additional space free, plus room for the runtime and other files. The cache allows the repacked weights to be mapped on later starts. The TV has 1.9 GB physical RAM, yet only around 1 GB is available before headless mode. Check usable memory and free eMMC space rather than comparing the file size with the device's advertised RAM.

What does a local run cost?

smolmaxxing has no token-metered API charge. The software repository is MIT-licensed, while each downloaded model keeps its own license. The practical costs are the device, storage, electricity and time spent setting up, waiting or retrying.

On a TV box you already own, electricity per task is:

device power in watts × task hours ÷ 1,000 × electricity price per kWh

For example, a 10 W device running for 30 minutes at $0.30/kWh uses 0.005 kWh and costs about $0.0015 in electricity. This is an illustrative calculation, not a measured draw for the Xiaomi box. A rented machine's cost is its hourly price multiplied by the full reserved time, including model setup and cache creation. For a fair comparison, divide the complete cost by outputs that pass your checks, and include failed or repeated runs.

Use the cost methodology to add hardware and utilization assumptions. For disk-streaming and offload approaches, compare AirLLM; for a different local serving engine, see Strata. If you are evaluating a large model on a PC or GPU rather than a tiny device, compare the Edge0-145B/120B guide.

Sources and next steps

Sources checked November 13, 2026: the smolmaxxing repository, TV-box setup, TV-box results and measurements, memory and expert-paging design, model downloads, software license, and memory-guard source. The iPad measurements and setup are in the project's iPad Air guide.

Start with the 2B model on the supported TV workflow, confirm the chat page responds, then compare the larger-model experiments before preparing their separate runtime and flash caches. Browse the local inference directory for other hardware and runtime combinations.