Arete-27B: Local Setup, Memory and Qwen Comparison

Nous starts from Alibaba’s Qwen3.8-27B and applies supervised fine-tuning using Inkling as the teacher. The release describes a 27B model, an evaluated checkpoint at step 1,202 and text-only training at 64K context. All language-model weights were trained; the vision encoder was frozen. There was no architecture change, reinforcement learning or preference-optimization stage.

The intended change is viewpoint and response behavior. Nous focuses on seven categories including freedom of expression, democracy, privacy and state ideology. That makes Arete a useful candidate when those response choices matter to your application. For routine coding, the base model remains a relevant comparison; the release does not establish a general coding advantage.

The frozen vision encoder also means text-only post-training should not be read as proof that all inherited visual capability disappeared. If you need image input, use a conversion and runtime that explicitly support the matching vision projector; evaluate that path separately from ordinary text chat.

The original BF16 safetensors total about 54.7 GB. That is the weight download, before conversation state, activations and runtime buffers. A 24GB GPU cannot hold the original checkpoint entirely in VRAM. CPU offload or multiple devices change both memory placement and speed.

Bartowski’s community GGUF conversion lists Q4_K_M at about 17.20 GB. This is a more practical starting point for a 24GB desktop GPU, with space still needed for context and working memory. Treat the remaining capacity as a budget to test, rather than assuming a long conversation will fit because the file does.

For capacity arithmetic, 27 billion parameters at two bytes each are roughly 54 GB in decimal units; at four bits each they are roughly 13.5 GB before format metadata and higher-precision tensors. The actual 17.20GB GGUF is therefore more useful for planning a download than the idealized four-bit calculation. Neither number measures peak process memory.

Install a recent llama.cpp build with the llama command and the GPU backend appropriate to your machine. Bartowski’s conversion card uses release b12174 for quantization and recommends that release or newer when the architecture requires recent support. The command below downloads its Q4_K_M conversion and opens a local server; use the web interface at the address printed by the server for a first text prompt.

This is a community conversion, separate from Nous’s original BF16 repository. Begin with a short context and one request. If loading fails or a longer conversation exhausts memory, inspect GPU allocation, reduce the context setting through the runtime, or choose a smaller quantization from the same conversion card.

For a first comparison, ask both checkpoints to answer a question from a source document, then check each cited fact against that document. Keep the prompt, thinking mode, sampling settings and output limit fixed. Add tool calls or images only after the basic text path works.

Nous’s own framing evaluation contains 803 prompts. It uses Grok 4.7 as an LLM judge against dated source facts and a fixed rubric. Over five runs with a 32K-token cap, thinking disabled and temperature zero, Arete scores 2.98/5 on factual recall versus 2.32 for its starting model. Answers flagged by the publisher’s propaganda rubric fall from 22% to 0.9%. These results describe that evaluation and its definition of framing.

The release separately reports Hermes Bench at 65.60% for Arete versus 65.77% for Qwen and interprets the difference as noise. Terminal-Bench 2.1 is 70% versus 67%, but Nous explicitly does not claim credit for an apparent improvement. Use these as evidence that the publisher checked general capability, rather than a reason to promise better coding on every task.

The LocalLLaMA release discussion mostly debates the release’s aims and response behavior. It does not supply a sufficiently specified independent speed or memory benchmark. When choosing hardware, test the quantization you will actually deploy on representative prompts, including questions whose correct answer you can check.

Pratik Gandhi’s November 7 comparison uses an M5 Pro Mac mini with 64GB unified memory, community affine four-bit/group-64 conversions, temperature zero and thinking off. MLX 0.32.4/mlx-lm 0.32.2 and TensorFold 1.2.4 run without speculative decoding. At 512 input tokens, three-trial median Arete/Qwen generation is 17.75/17.06 tokens per second on MLX and 17.22/16.83 on TensorFold. A background inference app remained active; the author treats the speeds as comparable, not a clean ranking.

Quality was tested only on MLX: Arete passes 67 of 69 local checks, while Qwen passes all 69. Arete misses comma handling in a word-frequency function and exceeds a word limit by one word. Both pass five tool-format checks and eleven needle checks through 65,536 input tokens; no external tools execute. Different conversion provenance prevents attributing the two misses solely to fine-tuning. This small suite does not reproduce a long-running coding agent.

The downloadable checkpoint has no token bill of its own. Local costs come from hardware, electricity and time; a rented machine also bills loading and idle time. The Hugging Face model page currently lists no deployed Inference Provider, so the local setup above does not imply a hosted pay-as-you-go endpoint.

For an illustrative rental at $1 per hour, a 12-minute session costs $0.20 in allocated compute. Three such sessions to obtain an accepted result cost $0.60 before storage and other charges. Those are chosen budget assumptions, not Arete measurements or a GPU provider quote. Replace them with the full wall time and hourly price of your actual machine.

If you already own the GPU, an assumed 0.4kW whole-system draw over the same 12 minutes uses 0.08kWh. At an assumed $0.20/kWh, electricity is $0.016, excluding hardware ownership. A model that needs longer answers or more corrections can lose a low per-token advantage at the task level.

Nous labels Arete-27B under Apache-2.0. Preserve the license and required notices when redistributing the weights or a derivative. Check the conversion’s files and notices when distributing a GGUF rather than assuming every third-party package is identical to the original release.

Use the original Qwen guide when you need the baseline model’s broader deployment options. Inkling’s page covers the teacher’s own access and pricing; its API prices are not prices for Arete. Compare these routes by the accepted task and the hardware or API you will actually use.