Oriel — AI & technology worth noticing, every day.

Your Experts Have Partners. There's HOPE for Keeping Them Together

Web page · https://huggingface.co/blog/amazon/higher-order-pruning-experts

First seen by Oriel on .

AI summary

HOPE is a second-order expert-pruning method for Mixture-of-Experts models that preserves pairwise cooperation, outperforming first-order baselines especially when removing 40–50% of experts.

  • It builds an F-matrix from one calibration pass, then solves a relaxed binary quadratic program whose diagonal measures individual importance and off-diagonal entries capture co-activation.
  • Across three models, two calibration datasets, and five methods, HOPE achieved a 2.07 average rank, finishing first in 39% and top-two in 67% of conditions.
  • At 50% pruning on Qwen3.5-35B-A3B, HOPE scored 42.3 on SWE-Bench Pro versus 41.8 unpruned and 36.2 for REAP.
  • The QP takes 1–2 seconds per layer; the package supports calibration, solving, and checkpoint pruning without changes to vLLM or SGLang model definitions.
  • Benefits shrink at 10–20% pruning; HOPE treats layers independently and models only pairwise interactions because triplets and higher orders are combinatorially difficult to estimate.

#mixture of experts #model pruning #quadratic programming #llm inference

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/amazon/higher-order-pruning-expertshttps://huggingface.co/blog/amazon/higher-order-pruning-experts Read discussion