Your Experts Have Partners. There's HOPE for Keeping Them Together
Web page · https://huggingface.co/blog/amazon/higher-order-pruning-experts
First seen by Oriel on .
AI summary
HOPE is a second-order expert-pruning method for Mixture-of-Experts models that preserves pairwise cooperation, outperforming first-order baselines especially when removing 40–50% of experts.
- It builds an F-matrix from one calibration pass, then solves a relaxed binary quadratic program whose diagonal measures individual importance and off-diagonal entries capture co-activation.
- Across three models, two calibration datasets, and five methods, HOPE achieved a 2.07 average rank, finishing first in 39% and top-two in 67% of conditions.
- At 50% pruning on Qwen3.5-35B-A3B, HOPE scored 42.3 on SWE-Bench Pro versus 41.8 unpruned and 36.2 for REAP.
- The QP takes 1–2 seconds per layer; the package supports calibration, solving, and checkpoint pruning without changes to vLLM or SGLang model definitions.
- Benefits shrink at 10–20% pruning; HOPE treats layers independently and models only pairwise interactions because triplets and higher orders are combinatorially difficult to estimate.
#mixture of experts #model pruning #quadratic programming #llm inference
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/amazon/higher-order-pruning-experts | https://huggingface.co/blog/amazon/higher-order-pruning-experts Read discussion |