LoRAWeaver: Directional Adapter Routing in LoRA Representation Space
Paper · https://arxiv.org/abs/2611.17340
arXiv:2611.17340v1 Announce Type: new Abstract: Large language models are expensive to update through full retraining. Parameter-efficient fine-tuning reduces the cost of adaptation, but repeated adaptation creates a growing pool of task-specific LoRA adapters. This pool complicates adapter selection during inference. Existing routing and encoder-based methods often require retraining when new adapters are added, which limits their use in dynamic settings. This paper treats adapter routing as a representation alignment problem in the adapter-output space. The proposed training-free method selects adapters by matching adapter-conditioned query representations against task prototypes using cosine similarity. Experiments with LLaMA-2-7B and LLaMA-2-13B on a 48-task FLAN-v2 benchmark and on out-of-distribution tasks show that the method outperforms training-free baselines in the cross-task setting and approaches a trained retrieval encoder without learning a router. An ablation confirms the benefit of directional alignment in the adapter-output space.
First seen by Oriel on .
AI summary
LoRAWeaver is a training-free adapter-routing method that selects task-specific LoRA adapters through directional alignment in adapter-output representation space.
- The method matches adapter-conditioned query representations against task prototypes using cosine similarity.
- It avoids retraining a router when new adapters are added, supporting dynamic adapter pools during inference.
- Experiments used LLaMA-2-7B and LLaMA-2-13B on a 48-task FLAN-v2 benchmark and out-of-distribution tasks.
- LoRAWeaver outperformed training-free baselines in cross-task settings and approached a trained retrieval encoder without learning a router.
- An ablation confirmed that directional alignment in adapter-output space improves routing.
#lora #parameter-efficient fine-tuning #adapter routing #large language models
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.17340v1 | https://arxiv.org/abs/2611.17340 Read discussion |