Kimi-K3.1-Pro-Coder Fits in 775GB with Three-Tier Single-GPU Offloading

Summary:Independent developer Grawnch has compressed the 2.8-trillion-parameter Kimi-K3.1-Pro-Coder from 1.77TB to 775GB by assigning routed-expert weights three quantization tiers averaging roughly 1.92 bpw. The accompanying Strata runtime places hot experts in GPU memory, warm experts in pinned RAM, and cold experts on NVMe, using router lookahead to prefetch inactive experts before they are needed. On a single RTX PRO 6000 Blackwell, the author reports 36.8 tokens per second for batched evaluation a

---

Fitting a 2.8T Model into 775GB

Grawnch's Kimi-K3.1-Pro-Coder quantization reduces the storage requirement of the 2.8-trillion-parameter model from approximately 1.77TB to 775GB, a reduction of about 56%. The release includes 96 weight shards, the Strata runtime, and the dr-triton dequantization kernels.

Rather than applying one low-bit format to the entire model, the project exploits the uneven access frequency of MoE experts. The 384 routed experts in each layer are divided across quantization tiers of 2.81, 2.56, and 1.44 bpw, averaging roughly 1.92 bpw. Shared experts, attention, and embeddings remain in BF16/F32 to preserve precision in the model's critical paths.

The result is still far from a conventional desktop deployment. Users need about 775GB of download capacity, substantial system memory, and fast NVMe. Running on one GPU does not mean the entire model resides in VRAM; it depends on a coordinated hierarchy of GPU memory, CPU memory, and SSD storage.

How Three-Tier Offloading Avoids the PCIe Wall

Strata places experts according to their access temperature:

• Tier 1: frequently accessed experts remain resident in GPU VRAM;
• Tier 2: warm experts are pinned in system RAM;
• Tier 3: cold experts are accessed through NVMe mmap.

The decisive factor is not merely the number of tiers, but whether data can be transferred after routing decisions and before expert execution. Strata uses router lookahead prefetching to start loading cold experts before they are required. The author reports a 97.9% effective hit rate.

That figure should not be treated as an end-to-end cold-data hit rate. Incorrect routing predictions can still trigger useless PCIe or NVMe transfers, and high-level routing decisions may not align perfectly with low-level expert fetches. Actual throughput depends on random SSD reads, NUMA placement, PCIe bandwidth, memory bandwidth, and batch size.

dr-triton Keeps Dequantization on the GPU

Expanding compressed weights into BF16 on the CPU would greatly increase the amount of data crossing PCIe. dr-triton uses fused Triton kernels to transmit packed weights over the bus and dequantize them only after they reach GPU SRAM.

This approach reduces host-memory pressure and bus traffic, but it adds dequantization work and temporary GPU buffers on every loaded expert. On compute-constrained consumer GPUs, the I/O savings may not always outweigh kernel-launch overhead and the cost of materializing dequantized weights.

Accuracy and Benchmark Results

The author compared the full 93 layers and lm_head:

| Metric | 1.77TB base model | 775GB quantized model | Change | |---|---:|---:|---:| | AIME 2025, Pass@1 | 29/30 | 29/30 | Identical | | LiveCodeBench, 30 problems | 29/30 | 29/30 | Identical | | Top-1 token agreement | 100% | 95.20% | -4.80 percentage points | | Top-5 token agreement | 100% | 99.92% | -0.08 percentage points | | AIME 2025 PPL | 2.2135 | 2.2358 | +1.01% | | LiveCodeBench Hard PPL | 2.1136 | 2.1562 | +2.02% | | LiveCodeBench Overall PPL | 2.6911 | 2.7629 | +2.67% |

The identical benchmark scores are encouraging but do not establish lossless equivalence. Top-1 agreement falls to 95.20%, indicating that generation paths have changed on some inputs even when the final answers happen to match. With only 30 problems per benchmark, the result also has a wide confidence interval and should not be generalized beyond the tested distributions.

The LiveCodeBench sample contains 10 easy, 10 medium, and 10 hard problems rather than a complete rolling evaluation window. Sampling parameters, run scripts, and repeated-run confidence intervals are not provided. These results are better interpreted as stress-test evidence than as a comprehensive model-quality certification.

Measured Throughput on One Blackwell GPU

On a single RTX PRO 6000 Blackwell with 96GB VRAM and 176GB RAM in Colab, the author reports:

• 36.8 tokens per second for batched or prefill evaluation;
• 10.8 tokens per second for single-stream decoding.

Most figures for other configurations are estimates rather than direct measurements. For example, a 24–32GB RTX 4090 or 5090 with 128GB DDR5 and Gen4 NVMe is estimated at 11–14 tok/s for batched evaluation and 4–5.5 tok/s for single-stream decoding. Dual-GPU configurations are estimated at 18–22 tok/s for single-stream decoding. These values illustrate the expected relationship between tier capacity and throughput, but they should not be used directly for purchasing or capacity planning.

Batch evaluation and interactive decoding also optimize different bottlenecks. Larger batches amortize expert-loading costs, whereas interactive workloads depend on time to first token and tail latency. Even with a healthy average TPS, NVMe latency spikes may still produce an uneven user experience.

Deployment Guidance

The project provides three primary components:

1. the 775GB, 96-shard model weights on Hugging Face;
2. the Strata three-tier offloading and lookahead engine;
3. the dr-triton fused dequantization kernels.

Users can reconfigure how many experts reside in VRAM, RAM, and NVMe. A system with 512GB DDR5 can pin more cold experts in RAM, while a 24–32GB GPU requires a smaller hot tier and places greater pressure on RAM and NVMe.

The weights use a custom packed tensor layout and cannot be loaded directly into vanilla llama.cpp or vLLM. Teams with an existing PyTorch or Triton runner can import the standalone dr-triton and strata_coder modules and manage placement themselves; other users need Strata's scheduler.

Before deployment, operators should test expert hit rates under the production routing distribution, time to first token, P99 latency, cold-start I/O time, and multi-turn completion quality on real code repositories. For production systems, a 30-problem benchmark is not enough to justify replacing the original weights.

Engineering Value and Limitations

The main contribution is the coupling of quantization granularity with runtime expert popularity. Instead of assigning every expert the same bit width, the system keeps hot experts at higher precision, compresses colder experts more aggressively, and hides storage latency through predictive I/O.

The remaining constraints are substantial. A 775GB model is still impractical for most individuals. Single-stream decoding at 10.8 tok/s can support code assistance, but interaction remains bounded by waiting time. The KLD, token-agreement, and perplexity changes show that compression is not fully lossless. Most importantly, the measurements come from the author's environment; generalization across NVMe devices, memory capacities, CUDA/Triton versions, and routing distributions still requires independent reproduction.

---

Source: Reddit r/LocalLLaMA (2026-11-08)

— END —

Tags: #Kimi-K3.1-Pro-Coder Follow #MoE Quantization Follow #Three-Tier Offloading Follow #Triton Follow #Single-GPU Inference Follow