Together Compiler Uses a Continual Agent Loop to Accelerate AI Inference
Summary:Together AI has introduced Together Compiler (TKCC), a continual compilation system for production AI inference. TKCC combines a tiered persistent engine, a cost model for selecting parallelism configurations, and an agent hill-climbing loop to continuously optimize inference paths as models, architectures, and traffic patterns change. The paper reports up to 1.7x higher throughput than hand-written Megakernels on the Llama 1B workload and up to 2.6x TPS over leading open-source engines across
TKCC Turns Inference Compilation into a Continuous Optimization Loop
Together AI's paper introduces Together Compiler (TKCC), a system designed to keep optimized inference execution aligned with frequently changing models, architectures, and traffic distributions. Rather than treating compilation as a one-time artifact, TKCC lets an agent repeatedly measure the real runtime, propose candidate configurations, and perform hill-climbing optimization inside a persistent execution environment.
The work addresses a practical tension in inference optimization. Offline kernel-level optimization is fast to automate but may not capture data-dependent behavior such as MoE routing and dynamic batching. End-to-end engine optimization provides more faithful feedback, but repeatedly launching, compiling, and rebuilding engines is expensive.
The Tiered Persistent Engine Separates Launch and Compute Costs
The central mechanism is a Tiered Persistent Engine that decouples engine launch overhead from forward-pass computation. The runtime can survive across candidate configurations, allowing the optimizer to load model state, switch kernels or parallelization schemes, and compare execution paths without repeatedly starting a complete engine.
This improves measurement throughput, but persistent execution introduces operational complexity. It requires strict state isolation, memory-leak control, configuration rollback, and correct cache invalidation. If kernel, model, or compiler versions are omitted from cache keys, stale artifacts could produce incorrect results or silently reuse incompatible code.
A Cost Model Prunes the Parallelism Search Space
TKCC also uses a cost model to choose the initial parallelism configuration. Tensor parallelism, pipeline parallelism, expert parallelism, and memory layout interact in ways that create a combinatorial search space. Evaluating every combination directly would be prohibitively slow.
The cost model estimates expected gains, compilation time, and execution cost before the agent starts its search. This is closer to budget-aware, best-first optimization than an undirected search over all configurations. Its limitation is model error: if the estimate is systematically wrong for a particular GPU topology, the search can prematurely converge to a local optimum.
The Agent Loop Gains Fidelity from Real Forward Passes
TKCC places the agent inside a feedback loop. The agent reads the model and engine state, proposes a configuration or kernel change, measures an actual forward pass in the persistent runtime, and uses the result to guide the next search step.
Compared with offline kernel benchmarks, this approach can observe effects caused by expert selection, variable sequence lengths, dynamic batching, and KV Cache state. The paper also presents ablations on agent context management, indicating that the choice of retained observations, failures, and code state directly affects search efficiency.
An agent does not automatically imply a better optimizer, however. Excessive context can increase inference cost and amplify noisy measurements. Without reliable metric regression checks and configuration rollback, one bad candidate can contaminate later iterations. TKCC's contribution is therefore a high-fidelity optimization interface, not proof that a general-purpose agent can discover every optimal kernel autonomously.
Performance Claims and Evaluation Boundaries
The paper reports up to a 1.7x speedup over a highly optimized hand-written Megakernel on the Llama 1B workload. Across Kimi K3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek v4.1 Flash, TKCC reportedly achieves up to 2.6x higher TPS than leading open-source engines. At batch size 1, the authors report 1111 TPS for M3.1 Flash and 1205 TPS for DeepSeek v4.1 Flash.
These results suggest that automated compilation can outperform expert-written kernels, particularly when models and traffic evolve quickly. TPS, however, cannot be interpreted without token lengths, input-output ratios, batch size, sequence length, and hardware configuration. High-concurrency throughput, low-concurrency latency, time to first token, tail latency, and optimization time are different objectives, and improving one may degrade another.
The evaluations are also author-reported. The abstract does not disclose the complete hardware matrix, baseline versions, search budgets, warm-up policy, or experimental variance. Independent replication will need to verify compiler versions, fusion boundaries, quantization settings, and whether data processing is included in the end-to-end measurements.
How to Evaluate Continual Compilation in Production
For inference-platform teams, the most important aspect of TKCC is not an isolated kernel but the conversion of compilation from a release-time task into a continuously running optimization service. Deployment should test:
- Adaptation speed: Can the persistent engine invalidate obsolete plans and reconverge quickly after a model, hardware, or traffic change?
- Search economics: Do the GPU time, compilation time, and memory overhead of agent exploration produce a net gain in online throughput?
- Rollback safety: Can invalid configurations, numerical instabilities, or abnormal requests be isolated immediately?
- Observability: Metrics must be segmented by kernel, engine, batch size, sequence length, and hardware rather than relying on a single TPS figure.
- Isolation and security: Persistent state must prevent cross-tenant leakage, while cache keys should include model versions, quantization settings, compiler versions, and driver versions.
TKCC has reportedly been deployed in Together AI's production fleet to serve trillions of tokens, demonstrating more than a conceptual prototype. Production scale alone is not independent validation, however. Teams should run end-to-end A/B tests against their own latency SLOs, hardware configurations, and request distributions.
— END —Source: ArXiv AI (cs.AI) (2026-11-13)
Tags: #Together AI #Continual Compilation #LLM Inference #Agentic Optimization #Inference Engines
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments