Together Compiler Uses Agents for Continual AI Inference Optimization

Summary:Together AI has introduced Together Compiler, a continual optimization system that combines agents with persistent compilation engines to adapt open-model inference as traffic and deployment conditions change. According to the company, the system can reach up to 3.7 times higher throughput and complete an optimization cycle in under 24 hours. The key distinction is the feedback loop: rather than producing a one-time kernel plan, the system continuously reacts to model, hardware, and traffic con

---

Release Overview

Together AI has introduced Together Compiler, combining agent-driven code generation with persistent inference compilation engines to continually optimize open models. The company says an optimization cycle can complete in under 24 hours and deliver up to 3.7 times higher throughput under particular traffic conditions.

The central idea is “continual compilation.” Conventional inference optimization usually produces kernels and execution plans offline after the model, GPU, sequence-length range, and concurrency range have been fixed. Once real traffic changes, a static configuration can fall outside its best-performing region. Together Compiler feeds deployment telemetry back to agents, which modify or generate optimized implementations, while persistent engines compile, validate, and deploy those candidates.

Technical Architecture

Based on the public description, the system has three main layers:

1. The observability layer collects throughput, latency, request-length, concurrency, and runtime-error signals to determine whether the current execution plan still matches production traffic.
2. The agent layer uses those signals to modify inference code, generate operator implementations, or adjust compilation settings. This allows search across CUDA code, operator implementations, parallelization strategies, and memory layouts rather than tuning one kernel in isolation.
3. The persistent compilation layer maintains and validates candidate execution plans so that the system does not rebuild them for every request or short optimization cycle. It also provides the boundary between agent-generated proposals and executable production code.

The most important aspect is not the agent itself, but the optimization space under its control. Relevant techniques include operator fusion, quantization, KV Cache management, continuous batching, paged attention, operator substitution, memory-layout changes, and combinations of hardware-specific kernels. Physical limits still dominate. Memory capacity determines whether weights and KV Cache can remain resident, while memory bandwidth determines how quickly autoregressive decoding can read weights.

For throughput-oriented compilation, numerical accuracy requires careful attention. FP16 or BF16 scaling is usually less disruptive to general-purpose model quality than aggressive INT8/INT4 quantization, which is more sensitive to outliers, activation distributions, and small-batch requests. Fusion may also change floating-point operation order and produce results that differ from a reference implementation. Production deployments therefore need regression testing for every model, precision configuration, and hardware variant—not just a single throughput measurement.

Engineering Trade-offs

The reported maximum of 3.7 times higher throughput indicates meaningful optimization headroom, but it should not be treated as a universal improvement. Different inference phases have different bottlenecks:

• Prefill is generally compute-bound and depends on matrix multiplication, attention kernels, and long-context handling.
• Decode is often constrained by memory bandwidth and KV Cache access.
• Low-concurrency, short-request traffic may be dominated by scheduling, Python boundaries, and kernel-launch overhead rather than raw GPU compute.
• High-concurrency, long-context traffic is often governed by KV Cache memory and scheduling policy.

The same compilation plan may increase tokens per second in one batching interval while degrading tail latency. Continuous agent-driven tuning can also over-optimize average throughput through excessive batching or kernel substitution if it lacks explicit latency constraints.

Completing an optimization cycle in under 24 hours reduces deployment latency, but continuous operation is not maintenance-free. Generated code must pass sandboxing, compilation, static analysis, benchmarking, and regression validation before production promotion. A persistent engine should therefore function as a versioned execution system with approvals and rollback, not merely a background cache.

Benchmarking Caveats

The available announcement supports only a “up to 3.7 times” claim. A rigorous evaluation should include:

• Fixed model versions, input and output lengths, concurrency levels, sampling settings, and request-mix distributions;
• Comparisons against unoptimized deployments, hand-tuned systems, and established compiler baselines;
• Tokens per second, request throughput, time to first token, inter-token latency, and P95/P99 latency;
• Output-quality measurements such as perplexity or task accuracy to detect gains obtained through precision loss;
• Tests after model updates, driver upgrades, and traffic shifts to identify continual-optimization regressions.

If the 3.7-times figure comes from a particular length distribution, batch size, or low-precision configuration, it should be interpreted as an optimization ceiling rather than a general promise for every open model.

Deployment Guidance

Production teams can adopt the approach in four stages:

1. Select an internally important open model and record the unoptimized baseline together with the inference-stack version.
2. Cluster real traffic into representative request distributions instead of testing only fixed synthetic lengths.
3. Put agent-generated candidates in a sandbox and enforce accuracy and performance regression gates before canarying them in production.
4. Continuously monitor throughput, P99 latency, error rate, and output quality, with independent rollback paths for compiler artifacts and model versions.

For teams already using vLLM, TensorRT-LLM, SGLang, or custom operator libraries, Together Compiler may be most useful as a system that discovers better combinations across existing components rather than as a complete replacement for the inference stack. Compatibility with current caches, quantization formats, and serving protocols will determine the real integration cost.

Together Compiler moves agents from model-level tooling into inference-system optimization. Its notable proposition is that compilation results evolve with traffic instead of being produced only once at release time. The main risks are unsafe candidate code, locally optimized metrics, and tail-latency regressions. The 3.7-times throughput result has meaningful engineering value only if it is supported by end-to-end benchmarks, controlled canarying, and robust rollback mechanisms.

---

Source: Together AI Blog (2026-11-04)

— END —

Tags: #Together AI Follow #Inference Optimization Follow #Continual Compilation Follow #Agentic Systems Follow #LLM Infrastructure Follow