Nemotron 3.5 Bolt E4B: local setup and memory costs

This page covers nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB, based on Gemma 4 E4B. It accepts text and returns text. A voice assistant needs separate speech recognition and speech synthesis components, with their own memory and latency.

The older Bolt 74B/A9B is a different vision-language checkpoint. Super VL is another family member. Their projectors, memory recipes, context limits and API quotes do not apply to E4B.

The official Q4_K_M file is 6,180,446,208 bytes, about 6.18 decimal GB on disk. File size, resident device memory and complete system RAM measure different things. A runtime can place tensors across CPU and GPU; cache and work buffers add allocations after weights load.

Start with a short context and one concurrent sequence. Increase context only after checking memory use through prompt processing and generation. The 64K limit is the model’s supported ceiling, not a statement that it fits on every device. DFlash adds a separate draft checkpoint and runtime state, so compare its complete allocation with ordinary decoding.

For an owned device, estimate electricity from measured wall power and run duration. For a rented machine, charge the whole reserved period, including startup and idle time. Divide the bill by accepted tasks after retries; an isolated decode rate does not determine the cost of a completed conversation or tool action.

Use the official model card’s llama.cpp path. NVIDIA’s example uses the full 65,536-token context; the adapted command below shortens it to 4,096 tokens and one sequence in NVIDIA’s pinned CUDA 13 container. Prepare Linux, a compatible NVIDIA GPU driver, Docker and NVIDIA Container Toolkit first. For DFlash, also obtain the companion NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash checkpoint and follow its exact runtime build and launch flags.

Keep the model card’s chat-template settings with the checkpoint. enable_thinking controls reasoning, while conversational_mode requests shorter conversational replies. NVIDIA recommends temperature 1.0 and top_p 0.95. A tool-using application must parse and execute tool calls, then return results to the model; loading the weights alone does not provide that loop. Once the server finishes loading, /v1/models should list the nemotron-bolt alias. Check full memory before raising context or adding the drafter.

Hiroshi Morishige of Classmethod reported tests on October 24 with a DGX Spark and an 8GB Jetson Orin Nano Super. His DGX measurements use llama.cpp b11808, Q4_K_M and explicit concurrency settings. On the Jetson, the NVIDIA GGUF failed to load, while a different Unsloth quantization ran. Those are different artifacts; the alternative’s result does not prove the official file fits that device.

The report also compares decoding with and without DFlash. Keep the exact build, artifact, context and number of concurrent requests with any speed figure you use. For a purchase decision, reproduce the intended prompt and memory configuration rather than treating the 4GB label as the entire device requirement.

Atomic Agent authors Nadya Dudka and Andrew Dyuzhov report 4,863 MiB of GPU memory after loading NVIDIA’s official Q4_K_M GGUF on a 32GB RTX 5090. Their October 23 setup used llama.cpp build 11840, commit 36e36e63a, 64K context, one sequence, f16 KV cache and no DFlash. This is an after-load nvidia-smi reading, not peak runtime or system RAM, and it does not establish compatibility with every 6GB device.

In Atomic Agent 0.6.10, default name/template matching omitted Bolt’s reasoning channel; an alias enabled a second reasoning-mode test. Six tasks repeated three times yielded 12/18 passes in either mode. Default mode took 113 seconds total across 18 runs. Reasoning mode took 1,281 seconds including one 547-second request; the article’s 738-second headline excludes that request. Sampling was temperature 0.2 rather than NVIDIA’s recommended 1.0. Their 178-token/s default decode rate measures generation, not complete agent-task time.

NVIDIA distributes this checkpoint under OpenMDW 1.1. Read the agreement and preserve its required license and notices when redistributing covered materials. A runtime and any speech components carry their own terms and operating costs.