Gemma 4 12B: QAT Downloads, Local Setup and Costs

The instruction-tuned identity is google/gemma-4-12B-it. The 12B Unified architecture is dense, with about 11.95B total parameters and a documented context capacity of 256K tokens. Its native multimodal architecture differs from the smaller E-series models with separate encoders. Runtime support still determines which input modes you can use.

Quantization-aware training prepares weights to tolerate reduced precision. An unquantized QAT checkpoint retains half-precision weights for further conversion or research. Select the ready-to-run Q4_0 GGUF for that four-bit deployment path; the word QAT alone does not mean the downloaded weights are four-bit.

Google’s verified 29d0977 revision lists gemma-4-12b-it-qat-q4_0.gguf at 6.98 GB and the accompanying mmproj file at 175 MB. These are artifact sizes. Running memory adds cache, buffers and any modality-specific allocations, and the split between GPU and system RAM depends on offloading.

Start with a context size suited to the actual document. A model’s 256K capacity is not a promise that every desktop can fill it. Leave room for the output as well as the input, and increase context only after checking memory consumption with the chosen cache precision.

Use a current llama.cpp build with Gemma 4 support. Google’s model page provides the repository selector below for llama-server. The first launch downloads the selected artifact, so allow storage and download time before evaluating generation speed. Start with a short text request and confirm that the server loads the intended Q4_0 file.

Open the local server’s web interface after it starts. Ask for a short answer with an objectively checkable result, such as extracting a date and amount from a paragraph. Record the runtime build, checkpoint revision, context size and GPU offload settings before changing one of them.

For visual input, follow the runtime’s multimodal instructions and load its matching components. A successful text-only session does not demonstrate that the same build accepts images or audio. Keep text-only memory measurements separate from multimodal sessions.

Murasan Lab’s July 27 report, updated August 4, uses an RTX 5060 Ti 16GB, llama.cpp build 9570 (3ac3c20c9), full GPU offload and a 65,536-token context setting. The target is the third-party QAT UD-Q4_K_XL GGUF, not Google’s Q4_0 artifact. Each fixed-length test generates 256 tokens five times with temperature 0, seed 42 and thinking disabled; mmproj is not loaded.

Without MTP, the f16-cache configuration averages 50.33 tokens/s and 8,248 MiB under load. With q8_0 KV cache plus a Q4_0 MTP draft and n-max=2, it averages 80.31 tokens/s and 8,076 MiB. Multiple settings change between those rows. The 65K value is configured capacity rather than a filled 65K prompt, and these generation rates are not complete API response latency.

The existing LLM calculator includes Gemma 4 12B with local and cloud deployment estimates. Choose GGUF Q4 for a four-bit planning scenario, then set your workload and deployment choice. Its generic capacity and throughput estimates are separate from the official QAT artifact and the Murasan configuration above.

For rented hardware, budget the entire billed session. A hypothetical two-hour session at $0.50 per hour costs $1. If it produces 200 accepted responses, compute spend is $0.005 per accepted response, before storage and transfer. Downloads, idle time and rejected answers reduce the amount of useful work that session buys.

For owned hardware, measure power at the wall. A hypothetical 200 W total system load for two hours consumes 0.4 kWh, costing $0.08 at $0.20/kWh. That is an electricity example; hardware purchase and maintenance remain separate. Use your measured power rather than a GPU’s advertised maximum as the input.

Google lists Apache 2.0 for this Gemma 4 release, including the QAT GGUF. Retain the applicable license and notices when redistributing it. Model licensing and any terms of a hosted inference service are separate parts of a deployment decision.