Flide Accelerates Agent Decision Inference with Exact State Forking

Summary:Flide decouples a decision request's logical tree from its physical execution on the GPU. Its ForkAttention mechanism shares prefix computation across query branches while preserving, in exact arithmetic, the same results as isolated forward passes. GPU-resident metadata describes branch topology, and checkpoints allow growing agent histories to resume without replaying the entire prefix. On public hybrid and decision models, the paper reports query-level speedups of up to 9.5× over vLLM and SG

---

Core Idea: Mapping a Decision Tree onto One Reusable GPU Execution

Flide targets categorical decision workloads rather than conventional token-by-token generation. In agent systems, a request branches from shared historical state into multiple candidate paths, and each branch only needs to continue from that common context. Generation-oriented engines often handle such workloads by repeatedly recomputing shared prefixes or by sharing computation in ways that distort output probabilities.

Flide separates the request's logical tree from its physical token execution stream. Each branch retains its own semantic state, while GPU-resident indices locate dependencies so that shared prefixes are computed only once. ForkAttention covers both standard attention and recurrent layers such as Gated DeltaNet. Its correctness target is explicit: in exact arithmetic, every branch must produce the same result as an isolated forward pass.

This distinction matters. The speedup does not come from approximate attention, truncated context, or modified sampling distributions. It comes from computational reuse.

How ForkAttention Executes Branches

A forked state can be represented as a shared prefix and several incremental nodes. Conceptually, each query contains a common sequence (p_0, p_1, \ldots, p_k) followed by a branch-specific segment (d_c). Rather than physically copying the prefix into every sequence, Flide stores the branch topology in GPU metadata and lets the kernel locate the required state directly.

For standard attention, Key and Value states for the shared prefix are computed once, after which each query reads the appropriate prefix and appends its own increment. Recurrent architectures are harder because state updates are sequential and cannot always be treated as simple token concatenation. Flide therefore incorporates branch-aware state isolation into the indexing and scheduling system, preventing a branch from inheriting incremental state produced by a sibling branch.

In memory terms, the method reduces duplicate activations and repeated prefix computation; it does not necessarily eliminate KV Cache. Cache size remains broadly governed by effective context length. The main benefit is lower repeated computation and memory-bandwidth pressure, which can be significant on bandwidth-bound hardware.

Why Exactness Matters for Decision Models

Decision models commonly select tools, actions, or candidate paths. A small shift in categorical probabilities can move a result across a decision boundary. Flide therefore treats equivalence with an isolated forward pass as a correctness condition rather than relying only on average latency.

The paper compares output distributions using total variation distance and reports that Flide remains closest to a float32 reference. Exact-arithmetic guarantees, however, should not be conflated with exact GPU behavior. BF16 or FP16 accumulation order, fused kernels, and reduction strategies can still introduce rounding differences. Production validation should include:

• comparison of class distributions, not only final Argmax outputs;
• probability stability as batch sizes and branch topologies change;
• state-isolation checks for mixed standard-attention and recurrent models;
• error accumulation under long contexts, checkpoint restoration, and dynamic branch expansion.

Engineering Trade-offs

Flide benefits most when requests genuinely share prefixes. If branches are mostly independent, little computation can be reused and metadata or scheduling overhead may outweigh the benefit. It is therefore most suitable for agents whose candidate actions share observations, tool results, or system state, rather than unrelated chat requests.

GPU-resident indices reduce CPU-to-GPU control overhead but consume memory and require consistent synchronization between branch metadata and execution graphs. The paper proposes compact sets of pre-recorded graphs to cover diverse request mixtures. In highly dynamic production workloads, however, graph construction, synchronization, and caching must be included in the cost model.

Checkpointing allows growing agent histories to resume from earlier state instead of replaying the full trajectory. Saving complete recurrent or KV state can be storage-intensive and slow to restore; partial checkpoints may require additional forward computation. Checkpoint granularity, restoration latency, and consistency guarantees therefore need explicit treatment.

Interpreting the Evaluation

The paper reports query speedups of up to 9.5× over vLLM and SGLang, together with the closest total variation distance to a float32 reference. The headline number is compelling, but a maximum result should not be treated as a universal expectation.

vLLM and SGLang primarily target continuous generation, whereas Flide exploits shared prefixes in decision trees. The comparison therefore involves different execution shapes. Readers should check whether the benchmark captures realistic prefix overlap, context lengths, batch sizes, GPU models, and kernel backends. Maximum gains are likely to appear when prefix sharing is high and a conventional engine repeatedly recomputes the same context.

The summary does not provide P50/P95/P99 latency, sustained throughput, peak memory, checkpoint restoration cost, or results across different hardware. Independent reproduction should report all of these alongside output-distribution error and end-to-end task success, rather than only a best-case speedup factor.

Deployment Guidance

Flide is most compelling for decision agents with dense context sharing. Before integration, teams should analyze production traces to measure the longest common prefix and its share of total tokens. Low overlap may limit the gains. If agents repeatedly branch while continuously expanding their histories, checkpointing and state restoration may become the dominant bottleneck.

A practical validation sequence is:

1. Replay production traces and measure branch depth, prefix overlap, and request-mixture structure.
2. Compare Flide, vLLM, SGLang, and isolated forward passes using the same model, precision, GPU, and batch settings.
3. Measure both distribution error and task success so that latency optimization does not silently alter decisions.
4. Benchmark standard-attention, Gated DeltaNet, and hybrid models separately.
5. Include checkpoint size, restoration latency, and index-construction cost in capacity planning.
6. Add state-isolation assertions and periodic reference validation for dynamic branches.

Flide's broader contribution is to optimize a workload that generic generation engines handle poorly: branching decision execution. The reported 9.5× ceiling still requires independent validation, but combining exact state forking with prefix reuse directly targets an important systems bottleneck in decision-oriented agents.

---

Source: Hugging Face Trending Papers (2026-10-30)

— END —

Tags: #Flide Follow #Decision Inference Follow #ForkAttention Follow #Agent Inference Follow #Gated DeltaNet Follow