Fixing Tool-Call Accuracy in vLLM
Web page · https://ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm
First seen by Oriel on .
AI summary
Ubicloud explains how enabling vLLM constrained decoding raised tool-call schema accuracy for Kimi K2.6 and DeepSeek V4 Flash from 92% to 100%.
- K2VV ToolCall uses 2,000 requests to measure trigger similarity and the proportion of generated tool calls conforming to their schemas.
- XGrammar structural tags restrict token selection at each generation phase, enforcing valid tool names, indexes, arguments, and schema-constrained JSON.
- Precomputed token masks applied on the GPU provide runtime schema guarantees with negligible performance overhead.
- vLLM disabled constrained decoding by default because some models diverge under constraints, grammar translation can fail, and interactions with features such as speculative decoding cause bugs.
- vLLM 0.31 exposes the --tool-strict-level flag, which Ubicloud enabled after verifying 100% schema accuracy across all served model versions.
#vllm #constrained decoding #tool calling #llm inference
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 172, latest observation ; +166 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 72, latest observation ; +72 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50242870 | https://www.ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm Read discussion |