RC RANDOM CHAOS

vLLM Reaches 100% Tool-Call Accuracy with Constrained Decoding

· via Hacker News

Original source

Fixing Tool-Call Accuracy in vLLM

Hacker News →

Language models generating tool calls frequently produce outputs that violate the schema constraints required for downstream execution. When serving Kimi K2.6 with vLLM 0.28, only 92% of generated tool calls adhered to their schemas—forcing agents into retry loops that pollute context windows and inflate costs. These failures occur because language models generate tokens probabilistically without enforcing the strict validation that a compiler would apply. Models trained on diverse tool-calling datasets often emit field names or tool identifiers from their training data that don’t exist in production; for example, generating search_web when the available function is web_search.

vLLM addressed this by integrating constrained decoding, now available as a server-side flag in version 0.31. Using the XGrammar library, constrained decoding prevents invalid tokens from being sampled during generation. Instead of allowing the model to generate freely and validating the output afterward, the inference engine restricts choices at each step to only tokens that preserve syntactic validity according to the provided schema. With this approach enabled, newer models like Kimi and DeepSeek achieve 100% schema accuracy, completely eliminating malformed tool calls.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.