Together Compiler Uses AI Agents to Fuse GPU Kernels, Boosting B200 Throughput 28%
Summary:Together AI says Together Compiler uses AI agents to rewrite and fuse GPU kernels in a model's forward pass, replacing a hand-written megakernel path with generated, specialized kernels. On an NVIDIA B200 running Llama 3.2 1B, it reports 2,118 tokens/s versus 1,649 for the hand-written baseline—a 28% gain. The result points toward a repeatable generate-compile-validate loop for kernel engineering rather than bespoke optimization for every model. Yet the disclosed evidence is narrow: one GPU, on
The Core Change
Together Compiler places AI agents inside the GPU kernel optimization loop: they generate or rewrite kernels for a model's forward pass and then merge eligible operators. Together AI's goal is no longer to maintain an ever-expanding hand-written megakernel, but to generate specialized kernels for a particular model structure, input shape, and target GPU. The public summary does not explain graph parsing, code generation, autotuning, or validation, so the result should not be interpreted as evidence that agents can perform arbitrary CUDA optimization without supervision.
Technical Mechanism: What Fusion Removes
GEMMs and attention operations in Transformer inference are generally more compute-bound, while adjacent elementwise chains involving normalization, activation, masking, and residual connections are often constrained by memory bandwidth. If each operator launches a separate kernel, intermediate tensors must be written to HBM and read back by the next kernel. For a tensor containing N elements at d bytes per element, every additional HBM round trip moves approximately 2Nd bytes.
Fusion can keep intermediate values in registers or shared memory when the dependency and hardware constraints permit, reducing HBM transactions, kernel launches, and scheduling overhead. More aggressive fusion is not automatically better: spanning matrix multiplications or attention can increase register pressure, shared-memory usage, and spill risk, reducing occupancy. Performance depends on operator partitioning, memory layout, thread mapping, and scheduling rather than kernel count alone.
The potential value of AI agents is their ability to expand the code-generation and search space while automating repetitive compilation, debugging, and tuning. A reliable pipeline would at minimum include graph capture, fusion-legality checks, code generation, compilation, numerical comparison, autotuning, candidate selection, and artifact caching. The available summary does not indicate which stages are agent-driven and which remain conventional compiler operations.
How to Read the 28% Result
The reported figures give 2,118 divided by 1,649, or approximately 1.284, consistent with the stated 28% gain and an absolute improvement of 469 tokens/s. The comparison target is a hand-written megakernel rather than an unoptimized operator library, which strengthens the result.
The evidence nevertheless covers only an NVIDIA B200 and Llama 3.2 1B. A small model can make fixed overhead and fusion benefits relatively visible, whereas larger models are often dominated by GEMMs, attention memory traffic, and communication, so the same percentage may not transfer. The post does not disclose batch size, input and output lengths, prefill/decode separation, KV-cache state, numerical precision, warmup, CUDA Graph usage, or variance across runs. A 28% increase in aggregate throughput also does not imply an equivalent reduction in TTFT, inter-token latency, or p99 latency for every request.
Engineering Trade-offs
- One-time cost versus steady-state gain: Agent search, compilation, and tuning may take minutes or longer. That cost must be amortized against sustained serving throughput and energy savings. If model shapes or CUDA versions change frequently, artifact-cache hit rates will fall.
- Numerical correctness: FlashAttention-style algorithms, fused normalization, or altered floating-point operation order can change rounding error. Generated code requires operator-by-operator regression against trusted implementations, not just a loose comparison of final outputs.
- Performance portability: Tiles, shared-memory usage, and instruction choices tuned for B200 may not transfer to H100, AMD GPUs, or other CUDA architectures. Production artifacts should be fingerprinted by hardware, driver, compiler, and model version.
- Maintainability and security: Hand-written kernels are easier to audit but slow to evolve. Generated kernels accelerate development but introduce requirements for code review, dependency locking, sandboxed execution, and rollback.
- Baseline fairness: It must be confirmed that the hand-written megakernel received comparable autotuning and that both paths use identical precision, batching, warmup, and timing boundaries. Otherwise, the 28% gain may partly reflect baseline-quality differences.
Deployment Guidance
Treat generated kernels as compiler artifacts that require strict validation. Teams should first pin model versions, weights, input shapes, parallelism strategies, and the CUDA toolchain, then generate and cache a separate artifact for every production shape. Each artifact should be checked against eager execution or a trusted compiler using layer-level and end-to-end allclose tolerances, including edge cases and long-context inputs.
Acceptance tests should not rely on tokens/s alone. They should cover batch size one and high-concurrency serving, prefill and decode phases, TTFT, inter-token latency, p95/p99 latency, memory footprint, power consumption, and energy per token. Production deployments should retain an automatic fallback to a stable kernel path and continuously track numerical deviation, compilation failures, and performance decay for newly generated artifacts.
Evidence Boundary
The most important aspect of Together Compiler is not the isolated 28% figure, but the attempt to bring agentic code generation into a kernel-compilation loop. If it can reproduce results across models and shapes while automatically validating every generated kernel, GPU optimization could evolve from a bespoke activity for a small group of experts into scalable infrastructure. At present, it remains a promising engineering proof rather than a universal replacement for mature inference stacks.
— END —Source: Together AI Blog (2026-11-16)
Tags: #Together AI #AI Compiler #GPU Kernel Fusion #B200 #Inference Optimization
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments