Flide Accelerates Agent Decision Inference with Exact State Forking

Summary:Flide targets tree-structured queries in decision models and LLM agents. It decouples logical dependency trees from a flattened token stream and uses GPU-resident indices to implement ForkAttention across standard attention and recurrent Gated DeltaNet layers. In exact arithmetic, each branch is equivalent to an isolated forward pass, while checkpoints allow expanding agent histories to resume efficiently. On public hybrid and decision models, the paper reports up to 9.5× faster query execution

---

Core Idea: Separating Decision Trees from Token Streams

Flide addresses tree-structured queries in decision models and LLM agents rather than conventional token-by-token generation. In these workloads, a node depends on prior decisions and spawns multiple branches that may be evaluated in parallel. Generation-oriented engines such as vLLM and SGLang generally treat each branch as an independent request, repeatedly computing shared prefixes. Merging those requests without preserving branch boundaries can corrupt state or alter output probabilities.

Flide separates two concerns:

• The logical layer represents shared dependencies and isolated branches as a decision tree.
• The physical layer flattens branches into a token stream while maintaining their state mappings through GPU-resident indices and execution graphs.

This design allows different request mixtures to reuse a compact set of pre-recorded execution graphs instead of compiling a separate model path for every topology.

How ForkAttention Preserves Branch Exactness

The central primitive is ForkAttention. It covers both conventional self-attention and recurrent Gated DeltaNet layers. The latter is essential for hybrid models: Attention layers retain Key and Value history, while Gated DeltaNet layers maintain compressed recurrent state. A branching mechanism that only manipulates KV Cache cannot guarantee isolation or equivalence across the complete model.

ForkAttention must solve three problems:

1. Compute a shared prefix once while allowing each branch to read it.
2. Keep branch-specific writes isolated across both KV and recurrent states.
3. Preserve the mathematical semantics of each branch despite changes to token layout, operator tiling, and parallel scheduling.

GPU-resident indices map logical dependencies to physical state locations. In exact arithmetic, a branch executed by Flide is equivalent to an isolated forward pass. The reported speedup therefore comes from eliminating redundant computation rather than approximating attention, truncating context, or changing the sampling distribution. Flide can also resume expanding agent histories from checkpoints produced in earlier passes.

Engineering Gains and Constraints

On public hybrid and decision models, the paper reports up to 9.5× faster query execution than vLLM and SGLang. It also evaluates output-probability deviations through total variation distance against a float32 reference, where Flide remains closest to the reference distribution.

The 9.5× figure is a peak rather than a universal multiplier. Flide benefits most when branches share long prefixes. Independent queries may not amortize flattening, index construction, and scheduling overhead. Production evaluation should therefore measure:

• GPU memory used by indices, KV Cache, and Gated DeltaNet state;
• Kernel efficiency across GPU architectures and parallelism strategies;
• Effects of floating-point reduction order on end-to-end probabilities;
• Compilation and caching costs for dynamic topologies;
• Checkpoint serialization, storage, and restoration consistency.

Mathematical exactness does not automatically imply bitwise identity on every device. If the implementation changes reduction order or uses lower-precision operators, comparisons against the float32 reference must be repeated on the intended hardware, batch size, and data type.

Interpreting the vLLM and SGLang Comparison

Flide is optimized for a different workload from conventional autoregressive serving. vLLM and SGLang focus on continuous generation, continuous batching, and KV Cache reuse; Flide focuses on decision branches with explicit dependency relationships. The comparison is meaningful as a workload-specific evaluation, but not as a generic kernel-level superiority claim.

A robust deployment evaluation should document the models and tasks, tree width and depth, batch size, sequence lengths, GPU model, software stack, and whether prefix caching or chunked prefill is enabled. It should also distinguish per-request latency, aggregate throughput, and time to complete an entire query tree.

Deployment Guidance

Flide is best suited to agent systems with deep decision trees, long shared prefixes, and parallel branch evaluation, such as action enumeration, candidate scoring, and multi-tool routing.

A practical rollout should proceed in three stages:

1. Verify equivalence by comparing isolated forward passes with Flide branches at the KV-state, recurrent-state, and output-probability levels.
2. Benchmark representative topologies, including wide trees, deep chains, dynamic branches, and fully independent requests.
3. Optimize execution by compiling common topologies into reusable graphs, caching metadata, and incrementally checkpointing growing agent histories.

For conventional long-form generation, vLLM or SGLang may remain the better fit. For discrete decision queries dominated by shared context, Flide offers a principled exact-state-forking approach.

---

Source: Hugging Face Daily Papers (2026-10-30)

— END —

Tags: #Flide Follow #Decision Inference Follow #State Forking Follow #vLLM Follow #Gated DeltaNet Follow