Nanbeige4.3-33B: agent setup and long-context memory

This guide covers Nanbeige/Nanbeige4.3-33B, not the separately linked Base checkpoint or a planned smaller student. The publisher presents 33B as the teacher for its smaller-model distillation work. Its benchmark comparisons are publisher evaluations, with different harnesses across tasks; use a representative accepted task to decide whether it suits your work.

The model card supplies Transformers, SGLang and vLLM examples. It does not provide a direct-vendor token tariff or a verified minimum GPU configuration. Choose self-hosting from an identified checkpoint and runtime rather than borrowing a price or throughput from another 33B model.

At two bytes per parameter, 33 billion BF16 parameters give roughly 66GB of weight data before runtime allocations. This is weight arithmetic, not a measured download or peak memory. A quantized release changes that weight budget, while its cache format, buffers and offload placement still need separate accounting.

The official configuration has 52 layers, eight KV heads and a head dimension of 128. For an ordinary full BF16 key/value cache, 2 × 52 × 8 × 128 × 2 bytes gives 208KiB per cached token per sequence. The table shows this ideal cache alone; allocator overhead, attention workspace, concurrent requests and the weights add to it. A runtime using another cache precision or storage scheme needs its own measurement.

The model’s 512K limit therefore does not describe a fixed device requirement. Start with the context your actual prompt needs, include room for generated tokens and measure peak GPU and host memory. Reducing cache precision can change capacity and output behavior; compare the same task before accepting that tradeoff.

Download the exact instruction checkpoint and use one of the card’s runtime recipes. Its SGLang and vLLM examples use tensor parallelism across four ranks and a 0.8 memory-utilization setting. They are launch examples, not proof that four GPUs are necessary or that every four-GPU machine fits the full context.

The tokenizer enables thinking by default. The publisher recommends preserve_thinking=false for general chat and true for multi-turn tool and coding workflows. It recommends XML tool-call format while also supporting JSON. Match the chat template, reasoning parser and tool parser in the chosen recipe before connecting an agent.

Begin with a short text question, check the response and record loading time and peak memory. Then try one repository task with a small test suite and a limited output budget. The official Transformers example enables trust_remote_code; review and pin the repository revision before using that option in your own environment.

Tonyd2wild publishes an AutoRound W4A16 recipe for two NVLinked RTX 3090 24GB cards with vLLM and GPTQ Marlin. The serving configuration caps context at 131,072 tokens and uses an FP8 e5m2 cache. This provides an Ampere-oriented route to try alongside the official full-precision examples.

The author reports 61–66 decoding tokens/s at concurrency one, with thinking disabled, temperature zero and 400-token outputs. The separate 69-scenario tool evaluation includes a failed money-transfer authorization case and ambiguous requests where the model guessed. Keep approval checks in the application for consequential actions; a tool-format score does not establish permission to execute them.

Use the repository’s exact launcher, weights and recorded test conditions when reproducing the result. Measure peak device and system memory for your own workload. A longer prompt, additional users or a reasoning-heavy request changes the serving budget; the short decoding test does not measure the duration of a repository task.

On a rented machine, bill the allocated session: model loading, prefill, generation, tool execution, retries and idle time. For illustration, a 30-minute session at an assumed $2 per GPU-hour costs $1 for one billed GPU, or $4 if four GPUs are each billed at that rate. These are budget examples, not a quote for hardware known to fit this model.

On a machine you already own, measure wall power over the full task. An assumed 400W system running for half an hour uses 0.2kWh; at $0.20/kWh, electricity is $0.04. Hardware purchase, storage and your time remain separate. Keep rejected attempts in the session total and divide by the number of results you accept.