Oriel — AI & technology worth noticing, every day.

Transformers Model Integration Contracts

Web page · https://huggingface.co/blog/transformers-mic

First seen by Oriel on .

AI summary

Transformers introduces transformers-mic, a proposed standard for small, CPU-runnable model integration contracts that check real usage paths against human-reviewed baselines in CI.

  • Each contract packages a pinned, shrunk fixture, a runnable Markdown usage example, and reviewed expected outputs; compatible checkpoints can share a fixture while retaining separate baselines.
  • The UMT5 fixture caught an SDPA regression that altered label log-probabilities by 1.22 and training loss by about 0.20, from its introduction through its fix.
  • The selected T5 contract's CI runner step took 13.4 seconds on CPU, excluding setup; replaying frozen contracts across five months of commits reproduced the regression.
  • Fixtures preserve preprocessing files and original weight formats, retain architecture-specific layer types, and are built and validated once rather than rebuilt for every pull request.
  • The MIC job reports failures without blocking pull requests as of November 1, 2026; real-world detection rates, false alarms, review effort, and coverage remain open questions.

#transformers #model testing #continuous integration #machine learning

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/transformers-michttps://huggingface.co/blog/transformers-mic Read discussion