SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding
Paper · https://arxiv.org/abs/2611.02501
Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46times to 10.87times with one request, and from 1.94times to 3.93times with batching.
First seen by Oriel on .
AI summary
SpecPilot is an autonomous agentic loop that tailors speculative-decoding algorithms and systems optimizations to each LLM deployment, substantially improving speedups without training or quality loss.
- A cost model identifies the largest performance gaps, while parallel agents explore optimization directions and integrate successful results.
- Token forcing compares every output token against stock to detect reward hacking, and structured memory transfers validated mechanisms across rounds and deployments.
- SpecPilot discovered a lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification, and three batching mechanisms.
- Across four target models, three drafters, two GPU platforms, and five benchmarks, speedups rose from 4.46× to 10.87× single-request and 1.94× to 3.93× batched.
#speculative decoding #llm inference #systems optimization #ai agents
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 20, latest observation ; +5 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 2, latest observation ; +1 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2611.02501 | https://arxiv.org/abs/2611.02501 Read discussion |