NVIDIA Tackles the AI Memory Wall with Full-Stack Codesign

Summary:NVIDIA is addressing the memory pressure created by larger models, longer contexts, and agentic inference through full-stack codesign across GPUs, CPUs, interconnects, storage, and software. The strategy combines NVFP4 quantization for weights and KV caches, model sharding across an NVL72 domain, and tiered context management with NVIDIA Dynamo, Vera CPU memory, and CMX shared storage. RAG indexes also remain outside HBM, reducing the data that competes with active inference. NVIDIA claims that

---

NVIDIA is redesigning the entire data path, not simply adding HBM

Long-context inference and agentic workflows are pushing AI systems toward the memory wall: model weights occupy HBM continuously, KV caches grow with context length and tool-call history, and RAG adds more data that must be managed. NVIDIA's answer is not merely to add HBM3e or HBM4 capacity. It is to co-design GPUs, CPUs, interconnects, storage, and serving software so that data resides on the tier that best matches its access pattern.

The central distinction is that capacity determines how much data a system can retain, while bandwidth determines how quickly the compute units can consume it. Once compute advances faster than HBM bandwidth, additional capacity can raise cost and power consumption without translating into useful throughput.

How agentic inference creates the memory wall

Two major consumers compete for HBM during inference:

• Model weights: Their resident footprint depends largely on parameter count and numerical format.
• KV cache: It grows with context length, concurrent requests, and tool-call turns because the Key and Value states for historical tokens are retained to avoid recomputing the prefix.

Agentic systems carry prior context across repeated model and tool calls, and some trajectories reach millions of tokens. During tool execution, a KV cache may remain resident even though it is temporarily inactive. The accelerator can therefore wait for a tool while the cache continues consuming scarce HBM capacity.

RAG adds a separate pressure. Embedding indexes are large and have different access patterns from model computation. Keeping them in HBM directly reduces the capacity available for active inference.

How NVIDIA compresses, places, and reuses data

1. NVFP4 for model weights

NVFP4 stores eligible weights in a 4-bit floating-point format, reducing their HBM footprint relative to FP8. NVIDIA cites NVFP4 checkpoints for FLUX.2 klein, Inkling, Beam 1.0, Meta Muse Spark, Step 5 Preview, and Nemotron 3.5 Super VL, indicating that 4-bit inference is moving into model-release workflows.

Within an NVL72 system, weights can also be sharded across as many as 72 GPUs. Each GPU then holds only part of the model, leaving more HBM for KV cache and concurrent sessions. The trade-off is communication: insufficient operator parallelism, uneven MoE routing, or All-to-All bottlenecks can prevent linear scaling. Small models may also lose to communication overhead.

2. Quantized and temporally reused KV cache

Applying NVFP4 to the KV cache lowers the storage required per active context, allowing more sequences or longer contexts to remain resident. This is not free. KV values preserve attention history, and low-bit representations can accumulate error over long contexts. Deployments should therefore validate perplexity, retrieval-position bias, time to first token, and multi-request throughput rather than relying on memory savings alone.

NVIDIA Dynamo manages the cache as a system-wide resource across GPU memory, CPU memory, and storage. Its KV Cache Reuse capability allows previously computed context to be reused across requests or nodes, reducing redundant prefill. The key benefit is not merely compression, but a higher reuse rate for already-computed context.

3. CPU memory and shared storage for paused context

When a tool call makes context temporarily inactive, the Vera CPU's large LPDDR memory can act as a waiting area. Faster tool execution reduces the idle period, while NVLink-C2C connectivity allows paused context to return quickly when needed.

Longer-lived inactive context can be offloaded by Dynamo to CMX shared storage. Compared with local SSDs, CMX can be shared across servers. Compared with conventional enterprise storage, it avoids overhead associated with long-term data protection that short-lived context does not require. This trade trades extra movement time for lower cost per bit and only works if the runtime can predict access well: premature eviction and repeated reloads can increase latency and storage traffic.

4. Keep RAG indexes outside HBM

NVIDIA Nemotron Retriever produces compact embeddings, while cuVS builds and serves vector indexes across CPU and storage. The knowledge base remains outside HBM, and only the passages selected for the prompt enter the model context.

This frees GPU capacity but shifts pressure to vector search, PCIe or network transfers, and data loading. The result depends on embedding quality, index scale, retrieval latency, Top-k recall, and concurrent query capacity.

Engineering trade-offs: throughput per dollar is not a hardware-only metric

NVIDIA says that on the SemiAnalysis AgentX DeepSeek V4 Pro workload, GB300 NVL72 achieves up to 10 times the throughput per dollar of H200 at 100 tokens per second per user. Vera Rubin NVL72 reportedly reaches up to 30 times the throughput per dollar of GB300 NVL72 at 125 tokens per second per user, despite only about a 3% increase in HBM capacity per GPU.

These figures require careful interpretation:

1. Vendor benchmark bias. NVIDIA is citing SemiAnalysis AgentX results; independent replication across models, concurrency levels, and context lengths remains necessary.
2. Service targets matter. Tokens per second per user is valid only at the specified rate. Throughput improvements can increase queuing and tail latency unless P50, P95, and P99 latency and service-level violations are also reported.
3. Cost definitions may differ. A meaningful dollars-per-token metric must clarify whether it includes GPU depreciation, power, networking, CPUs, storage software, operations, and redundancy.
4. Cross-generation comparisons include software variables. The gains may reflect the GPU, NVLink, CPU, interconnect, Dynamo, quantized checkpoints, and scheduler together—not hardware alone.
5. “Up to” is not a typical result. A 30-fold improvement cannot be assumed for every model, precision configuration, or context length.

More useful AI-factory metrics include effective tokens per second at a target P99 latency, useful tokens per watt, concurrent sessions per GPU, incremental gain from cache hits, and token cost under a fixed quality constraint.

Deployment recommendations

• Profile the KV cache first. Measure average and P95 context length, concurrency, prefill-to-decode ratio, idle time during tool calls, and prefix reuse before enabling tiered offload.
• Adopt precision changes incrementally. Establish a quality baseline with FP8 or mixed precision before evaluating NVFP4 weights and KV caches.
• Include reload latency in the SLO. Measure cold and warm retrieval latency from CPU and CMX at P95 and P99.
• Check whether communication becomes the new bottleneck. For NVL72, inspect NCCL traces, All-to-All traffic, MoE load balance, and inter-node congestion.
• Separate RAG retrieval from generation resources. Serve indexes through cuVS on CPU or storage, but give the vector service its own concurrency and tail-latency budget.
• Validate end to end. Track task success, answer quality, time to first token, end-to-end latency, tokens per second, tokens per dollar, tokens per watt, and KV-cache hit rate.

The memory hierarchy becomes an agent-infrastructure battleground

NVIDIA's main claim is not a new model or memory standard, but a systems approach: make weights smaller, keep active KV state in HBM, move paused context close to the GPU, leave RAG indexes outside the compute core, and reuse context through Dynamo.

The direction is technically credible, but it also increases complexity. A genuinely useful hierarchy must show that data movement, cross-tier scheduling, and software coordination are more than offset by higher reuse and concurrency. Otherwise, the memory wall simply moves from HBM to PCIe, the network, and storage control planes.

---

Source: NVIDIA AI News (2026-11-05)

— END —

Tags: #NVIDIA Follow #HBM Follow #KV Cache Follow #NVFP4 Follow #Dynamo Follow