Transformers Model Integration Contracts
Web page · https://huggingface.co/blog/transformers-mic
First seen by Oriel on .
AI summary
Transformers introduces transformers-mic, a proposed standard for small, CPU-runnable model integration contracts that check real usage paths against human-reviewed baselines in CI.
- Each contract packages a pinned, shrunk fixture, a runnable Markdown usage example, and reviewed expected outputs; compatible checkpoints can share a fixture while retaining separate baselines.
- The UMT5 fixture caught an SDPA regression that altered label log-probabilities by 1.22 and training loss by about 0.20, from its introduction through its fix.
- The selected T5 contract's CI runner step took 13.4 seconds on CPU, excluding setup; replaying frozen contracts across five months of commits reproduced the regression.
- Fixtures preserve preprocessing files and original weight formats, retain architecture-specific layer types, and are built and validated once rather than rebuilt for every pull request.
- The MIC job reports failures without blocking pull requests as of November 1, 2026; real-world detection rates, false alarms, review effort, and coverage remain open questions.
#transformers #model testing #continuous integration #machine learning
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/transformers-mic | https://huggingface.co/blog/transformers-mic Read discussion |