Oriel — AI & technology worth noticing, every day.

Fixing Tool-Call Accuracy in vLLM

Web page · https://ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm

First seen by Oriel on .

AI summary

Ubicloud explains how enabling vLLM constrained decoding raised tool-call schema accuracy for Kimi K2.6 and DeepSeek V4 Flash from 92% to 100%.

  • K2VV ToolCall uses 2,000 requests to measure trigger similarity and the proportion of generated tool calls conforming to their schemas.
  • XGrammar structural tags restrict token selection at each generation phase, enforcing valid tool names, indexes, arguments, and schema-constrained JSON.
  • Precomputed token masks applied on the GPU provide runtime schema guarantees with negligible performance overhead.
  • vLLM disabled constrained decoding by default because some models diverge under constraints, grammar translation can fail, and interactions with features such as speculative decoding cause bugs.
  • vLLM 0.31 exposes the --tool-strict-level flag, which Ubicloud enabled after verifying 100% schema accuracy across all served model versions.

#vllm #constrained decoding #tool calling #llm inference

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 172, latest observation ; +166 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 72, latest observation ; +72 over observed period since

Aliases

SourceExternal IDURL
Hacker News50242870https://www.ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm Read discussion