Oriel — AI & technology worth noticing, every day.

Artifact Distillation: Training-Free Transfer from Frontier to Small Language Models

Paper · https://arxiv.org/abs/2610.37680

arXiv:2610.37680v1 Announce Type: new Abstract: Closing the gap between frontier and smaller language models is usually approached by distillation: fine-tuning the small model (student) on a frontier model's (teacher) outputs, which binds transfer to training compute and leaves the transferred knowledge largely opaque. We propose Artifact Distillation: a teacher solves training instances and externalizes its problem-solving procedure into an artifact bank that a student queries at inference time, with no gradient updates. Across four LiveBench tasks, a bank written by Claude Opus 4.5 lets Claude Haiku 4.5 outperform unaided Claude Sonnet 4.5 on spatial reasoning (0.707 vs. 0.647) at 60% of its cost, and reach Sonnet 4.5's level on table joining at 89% of its cost. The same raw bank raises Qwen3.5-27B's pooled score from 0.730 to 0.800, showing that transfer is not confined to the teacher's model family.

First seen by Oriel on .

AI summary

Artifact Distillation transfers a frontier model’s problem-solving procedures to smaller models through a reusable artifact bank, without fine-tuning or gradient updates.

  • The teacher solves training instances and externalizes its procedures into artifacts that the student queries at inference time.
  • On spatial reasoning, Claude Haiku 4.5 with a bank written by Claude Opus 4.5 scored 0.707, exceeding unaided Claude Sonnet 4.5’s 0.647 at 60% of its cost.
  • On table joining, Claude Haiku 4.5 reached Sonnet 4.5’s level at 89% of its cost.
  • The same raw bank raised Qwen3.5-27B’s pooled score from 0.730 to 0.800, demonstrating transfer across model families. Results cover four LiveBench tasks.

#language models #knowledge distillation #inference-time compute

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.37680v1https://arxiv.org/abs/2610.37680 Read discussion