PolyGraph: Introducing Plan-Buffered Graphs for Adaptive Speculative Decoding

Summary:PolyGraph is a novel plan-buffer mechanism for custom Triton kernels that addresses the challenges of adaptive speculative decoding, where batch size and draft length change dynamically. By capturing a single maximum-grid graph and writing the active shape to GPU memory before each replay, PolyGraph allows kernels to skip tiles beyond the active shape, thereby reducing graph build time and avoiding the overhead of padded payloads. Experiments on an NVIDIA H800 demonstrate that PolyGraph maintain

---

Technical Mechanism Analysis

The core innovation of PolyGraph lies in its plan-buffer mechanism, which optimizes adaptive speculative decoding through the following steps:

1. Maximum-Grid Graph Capture: PolyGraph captures a maximum-grid graph that includes all possible combinations of batch sizes and draft lengths.
2. Dynamic Shape Writing: Before each replay, PolyGraph writes the current active shape to GPU memory.
3. Tile Skipping: The kernel skips tiles beyond the active shape during execution, avoiding unnecessary computations.
4. Fixed Graph Topology: The graph topology, allocations, layouts, pointers, and launch grids remain fixed, with only 28 bytes of data updated.

This design allows PolyGraph to maintain efficiency while adapting to the dynamic demands of decoding.

Engineering Trade-offs and Performance

• Advantages:

  • Reduced Graph Build Time: The pre-service graph build time is reduced from 3915 ms to 93 ms, significantly improving efficiency.
  • Avoidance of Padded Payloads: By skipping tiles beyond the active shape, PolyGraph avoids the overhead of padded payloads.
  • Performance Matching Baseline: In terms of step latency, PolyGraph matches the baseline (17.7 vs. 17.8 ms P50) but avoids the performance degradation caused by padded payloads in practical applications.

• Disadvantages:

  • Single GPU Only: Currently limited to testing on a single NVIDIA H800 GPU, with no validation in multi-GPU or distributed environments.
  • Synthetic Traces Only: Experiments are based on synthetic traces, with no testing in real-world production environments.

Developer Implementation and Deployment Recommendations

PolyGraph is still in the research phase and has not yet become a production-grade serving system. However, its design philosophy and experimental results provide new ideas for AI inference optimization. Here are some recommendations:

1. Multi-GPU and Distributed Environment Testing: Future work should extend to multi-GPU and distributed environments to verify its applicability in more complex scenarios.
2. Production Environment Validation: Testing in real production environments is needed to evaluate its performance and stability in practical applications.
3. Integration into Existing Frameworks: Consider integrating PolyGraph into existing AI inference frameworks, such as Triton, to simplify the deployment process.

Conclusion

PolyGraph demonstrates the potential of using a plan-buffer mechanism to optimize AI inference performance in adaptive speculative decoding. Although still in the research phase, its innovative design and experimental results provide a new direction for AI inference optimization.

---

Source: ArXiv Machine Learning (cs.LG) (2026-10-20)

— END —

Tags: #PolyGraph Follow #AI Inference Optimization Follow #Adaptive Speculative Decoding Follow