On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
Paper · https://arxiv.org/abs/2611.06976
arXiv:2611.06976v1 Announce Type: new Abstract: Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.
First seen by Oriel on .
AI summary
This paper introduces PLatour for training and evaluating Prefill-to-Autoregressive Cross-model Transfer, showing strong benchmark results while finding that some gains may not depend on sender information.
- PLatour connects frozen language models using lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures.
- The framework outperforms open-source baselines on generation tasks and exceeds Mostik's reported mathematics and coding results with a substantially smaller sender.
- Controlled experiments reproduce some improvements using uninformative sender signals, self-communication, or receiver-only training.
- The authors find trainable bridges can provide parameter-efficient adaptation without using useful sender information.
- PACT-Check measures how much a trained system's performance gain genuinely results from communication.
#language models #model transfer #parameter-efficient adaptation #reinforcement learning
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.06976v1 | https://arxiv.org/abs/2611.06976 Read discussion |