On July 30, 2026, enabling PyTorch's scaled dot-product attention (SDPA) for T5 in Transformers introduced a bug that allowed UMT5's decoder to attend to future tokens during training. The issue went unnoticed in main for almost two months but was flagged on every affected commit when a small CPU UMT5 fixture was replayed. This article explains the fixture-driven testing approach and the Model Integration Contract (MIC) idea that helped surface the regression.
Why regressions reach main
Transformers supports over 750 model types and maintains a large test suite: more than 88,000 CPU tests on main and an additional 200,000+ GPU tests in daily runs. Tests cover pipelines, training, distributed execution, quantization and attention kernels. Running full integration checks on every PR would require many GPUs all day, so regressions sometimes reach main before scheduled runs detect them.
To address this, the project shrinks models into fixtures: small models derived from a real checkpoint's configuration that keep every type of layer but with fewer and narrower layers, while leaving tokenizer and processor files unchanged. A fixture is published once and reused for fast integration tests on inexpensive CPU runners. In an internal CPU study, an experimental UMT5 fixture used 66 MB of weights compared to the source checkpoint's 1,227 MB. This approach also enables coverage of very large models that don't fit on standard GPU runners by running their fixtures on CPU.
Fixtures are not a replacement for full integration tests, but they increase the chance of catching regressions earlier on PRs.
Model Integration Contracts (MIC)
transformers-mic is a proposed standard to package these tests. A Model Integration Contract contains three parts:
- fixture: the pinned, shrunk model
- usage path: a runnable Markdown example that demonstrates a common use of the model (text generation, a training step, etc.)
- baselines: human-reviewed recorded outputs when running the usage path on the fixture
The contract holds if running the usage path on the fixture reproduces the baselines. A contract covers an architecture and usage path rather than a single checkpoint: compatible checkpoints may share a fixture while each keeps its own expected outputs. MICs are published in huggingface-mic.
A simple SmolLM contract, for example, generates completions for prompts like "The capital of France is" and compares token IDs with a recorded answer. Usage tests can also check training loss, that gradients are finite, and that an optimizer step changes weights. A fleet dataset on the Hub pins each MIC to a frozen revision and records package versions used to generate baselines; CI runs log the fleet revision to allow later reproduction.
On a PR, MICs are selected based on which files the patch touches. The UMT5 regression was discovered by replaying frozen MICs across five months of commits: the small UMT5 fixture flagged the regression from its introduction on July 30, 2026 through its fix on September 28.
Details of the UMT5 regression
Replaying frozen MICs over the commits showed the small UMT5 fixture flagged the issue from 2026-07-30 until the fix on 2026-09-28. Label log-probabilities differed from the reviewed baseline by 1.22, and training loss differed by about 0.20. A key point: the contract used SDPA (the default attention path); running only the eager-attention variant did not catch the bug. A fixture only helps if it exercises the same code path as the real checkpoint.
When the fix was reverted on a proof branch in Transformers' CI, the MIC check failed with the same differences. The chosen T5 contract's runner step took 13.4 seconds on CPU, excluding setup. Replay results and a CI demo contain measurements and reproduction details.
How models are shrunk into fixtures
The initial recipe — simply reducing dimensions and layer counts — proved insufficient for mixed architectures (for example, keeping only two layers is useless if those two are of the same attention type and other types vanish). The current recipe has four steps:
- Shrink the configuration: reduce hidden sizes, attention heads and depth but retain the different layer kinds (including mixtures like sliding vs full attention or dense vs MoE); consult ModelTester sizes when the generic rule lacks a name.
- Preserve preprocessing unchanged: keep vocabulary, special tokens and input dimensions, and copy tokenizer/processor files byte-for-byte — loading and preprocessing are part of what's tested.
- Build and validate weights: start with seeded random weights and change inputs to confirm the model responds; if not, train on the contract's small batch, reload, and check outputs, loss and gradients are finite; reject failed builds.
- Freeze and record: save accepted weights and measure repeatable outputs on a known-good Transformers build, review baselines per platform/precision, then publish a pinned snapshot; CI reuses it rather than rebuilding for every PR.
The build keeps the original weights file format to ensure conversion code paths run when real checkpoints are loaded. The Llama 4 Scout shrink report shows an example reduction: 108.6 billion parameters -> 44.7 million; vision stack 34 -> 2 layers; language stack 48 -> 4 layers, preserving layer kinds and MoE routing style.
Building MICs with agents and the tmic CLI
The tmic CLI creates contracts and builds fixtures. For a new model, an LLM drafts usage code and the manifest; CPU and GPU runs measure candidate baselines. The LLM drafts but does not approve baselines — changes require human review. The CLI can run locally and will prompt for review at each step, stopping before upload.
Agents were used to build MICs across model families; when edge cases appeared, the team improved the general recipe rather than only fixing single fixtures, with help from model experts. Published models and remaining coverage gaps are listed in the coverage space.
Random versus overfit weights
Randomly initialized fixtures can be insensitive to some inputs: a Whisper fixture with random weights barely responded to audio changes, so it was a poor audio-path test. The team therefore trained (overfit) that fixture on a few tone-to-word pairs to make audio changes visible. In a weight-initialization study, random Whisper fixtures caught 18 of 24 injected faults, while overfit fixtures caught all 24. That yields two approaches:
- use random weights with a fixed seed when they pass checks
- overfit on a few examples when more sensitivity or readable answers are needed
The team hasn't yet found a real-world regression caught only by the overfit variant; both approaches missed a Whisper assisted-decoding bug that the real checkpoint exposed. Overfitting adds a training step that can fail to converge and increases maintenance cost, so both variants are kept where appropriate; the comparison study records the results.
Open questions and next steps
As of November 1, 2026, the MIC job in Transformers CI reports failures but does not block PRs. Whether it should block depends on unanswered questions:
- How many real bugs does it catch on incoming PRs (not just in replays)?
- How often does it produce false alarms?
- How much human effort do baseline reviews require?
It is also unclear how many regressions appear only with real weights, or whether overfitting catches real bugs that random weights miss. Coverage across model families and API usage remains partial. When a regression slips through, the missing usage path can be added and replayed on the broken and fixed versions. Existing tests will be kept until their coverage is accounted for.
Finally, the team wants to link original Hub model pages to their MICs so developers or agents using a checkpoint can start from an example already tested on that checkpoint, with its environment and expected outputs.
How to help
To contribute, pick an uncovered model from the coverage space, try the tmic workflow, and submit a usage path or review a generated contract.
