Oriel — AI & technology worth noticing, every day.

FusionEval: Benchmarking Model Fusion for Large Language Models

Paper · https://arxiv.org/abs/2610.30162

arXiv:2610.30162v1 Announce Type: new Abstract: Model fusion aims to integrate the capabilities of multi-expert models into a single model. Existing evaluations offer limited coverage of reasoning with long chains of thought and tool use, while strong average performance can mask substantial losses in individual expert capabilities. We introduce FusionEval, a benchmark that provides controlled expert pools, integrates fusion methods with and without additional training, and offers a unified evaluation suite. The pools contain domain experts independently trained from Qwen3 and Llama base models across math, code, science, instruction following, and tool use. The suite evaluates capability retention and computational cost on eight public benchmarks under comparable data access rules, reporting normalized average and worst performance, runtime, and peak GPU memory usage. Extensive experiments show that training-free methods preserve reasoning performance but substantially degrade tool use at both model scales. In contrast, MOPD preserves tool use and achieves a more balanced capability profile, but requires additional data preparation, inference, and training. Jointly retaining reasoning, instruction following, and tool use at low computational cost remains a challenge. We hope FusionEval supports future research on efficient model fusion that jointly preserves expert capabilities. Our code is available at https://github.com/Baicaihaochi/FusionEval_Benchmark.

First seen by Oriel on .

AI summary

FusionEval benchmarks whether model fusion can combine LLM experts while retaining individual capabilities, and finds that preserving reasoning, instruction following, and tool use efficiently remains difficult.

  • The benchmark uses controlled math, code, science, instruction-following, and tool-use expert pools trained independently from Qwen3 and Llama base models.
  • Its unified suite evaluates eight public benchmarks, capability retention, runtime, and peak GPU memory under comparable data access rules.
  • Training-free fusion methods preserve reasoning but substantially degrade tool use at both model scales.
  • MOPD preserves tool use and produces a more balanced capability profile, but requires additional data preparation, inference, and training.

#llm fusion #llm benchmarks #tool use #model evaluation

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.30162v1https://arxiv.org/abs/2610.30162 Read discussion