Oriel — AI & technology worth noticing, every day.

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Web page · https://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specialization

First seen by Oriel on .

AI summary

VocabX adapts a model’s vocabulary to domain-specific input and output costs, reducing measured inference cost by 77% in a Portuguese essay grader without changing its base model or training recipe.

  • VocabX combines domain-specific sub-word tokenization with macro tokens for repeated spans, selecting both according to estimated savings across prompts, user messages, and model responses.
  • The grader’s average response shrank from 350 to 101 tokens, while its bench score rose from 0.703 to 0.810 across 500 requests measured on an RTX 5090 using vLLM.
  • The grader’s output layer shrank from about 152,000 to about 65,000 rows, reducing the vocabulary scored at each generated token.
  • A statistical estimate for candidates missed in document samples raised token reduction on contracts from 44.7% to 52.8% using the same sample and budget.
  • Across five public domains, three model families, and fifteen combinations, VocabX compressed more than the strongest reference method; the paper’s results are preliminary and may change before publication.

#tokenization #language models #inference optimization #model compression

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specializationhttps://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specialization Read discussion