Saluki 27B: GGUF Setup, Memory and Running Costs

Download Underdog-Saluki-27B-1.0-IQ2-mix.gguf for text. The separate projector is needed for image inputs; choose one projector rather than downloading both. Keep disk space for the model, projector and any duplicate download cache.

Use a current llama.cpp build with Qwen3.8 support. Download the exact IQ2-mix file, then start the server below. This example uses an 8,192-token allocation for a smaller first trial; the publisher’s 32,768-token command is an example setting, not a stated maximum context. Adjust GPU offload for your device.

Open http://127.0.0.1:8080 and ask it to explain a short function with a known bug. Check the answer against the code and run the proposed test. The local OpenAI-compatible API is at http://127.0.0.1:8080/v1. A model that emits tool calls still needs a client to execute them and return results.

The GGUF download is a storage figure. A loaded process also needs context cache, working buffers and, for vision, the projector and image-processing allocations. GPU offload changes how much sits in VRAM versus system RAM or unified memory.

Load a short text request first and record peak memory, context, runner version and offload settings. Increase context only as the job requires it. Keep the operating system and other applications in the budget; fitting the file on disk does not establish that an 8GB GPU can run your chosen configuration.

Michael Leung’s October 11 test used the IQ2-mix file on an RTX 4060 Ti 8GB, a 14th-generation i9 and 64GB RAM with llama.cpp b11540. Auto-fit placed 42 of 65 layers on the GPU. His short CUDA benchmark reached 12.11 tokens/s at 42 layers and 14.84 at 48, but the real server at 16K context produced 9.7 tokens/s; a longer conversation fell to roughly 3–5. Thinking consumed the context before completing code in two recorded attempts. File size, backend and conversation length all affected the usable result.

Dedene’s October 10 DGX Spark report uses 128GB unified memory, IQ2-mix, CUDA 13, F16 KV cache, flash attention and one slot configured at 262,144 tokens. Its weighted chat decode result is 23.9 tokens/s without a drafter and 52.7 with DFlash. The setup is pinned to llama.cpp b11515; the plain baseline ran on b11459, with a b11515 recheck at 23.8. The DFlash-plus-vision process holds 27.5GB. These are configured context and weighted decode measurements, not proof of an official maximum or uniform speed across languages.

Conway reports 88 of 120 passed tasks for Saluki versus 84 for full-size Qwen3.8-27B on its frozen BFCL v4 subset, with thinking disabled and temperature zero. That is a publisher evaluation on a modest sample. It supports testing tool calling, rather than treating compression as a uniform quality improvement.

The same card reports weaker competition-math retention and formatting slips in about a fifth of parallel-call replies. For an agent, validate arguments before execution and include retries and repairs in the measurement. Thinking is enabled by default; the publisher uses temperature 0.6, top_p 0.95 and top_k 20 for general reasoning, and disables thinking for the direct tool-call comparison.

For rented hardware, divide the billed session cost by accepted tasks. As a hypothetical example, a $1/hour machine used for 30 minutes costs $0.50; 20 accepted results cost $0.025 each. If only 10 are usable, that rises to $0.05. Include downloads, loading, idle time and failed attempts when they fall inside the rental period.

On hardware you own, measure wall power and elapsed time. At an assumed 200W for 30 minutes and $0.20/kWh, electricity is $0.02. These are illustrative assumptions, separate from hardware purchase and maintenance. A smaller download helps storage, but task quality and elapsed time determine whether it lowers your bill.

The publisher labels Saluki Apache 2.0 and credits Qwen3.8-27B and ISTA-DASLab’s compressed GGUF work. Retain the license and NOTICE files when redistributing the model.