Oriel — AI & technology worth noticing, every day.

On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer

Paper · https://arxiv.org/abs/2611.06976

arXiv:2611.06976v1 Announce Type: new Abstract: Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.

First seen by Oriel on .

AI summary

This paper introduces PLatour for training and evaluating Prefill-to-Autoregressive Cross-model Transfer, showing strong benchmark results while finding that some gains may not depend on sender information.

  • PLatour connects frozen language models using lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures.
  • The framework outperforms open-source baselines on generation tasks and exceeds Mostik's reported mathematics and coding results with a substantially smaller sender.
  • Controlled experiments reproduce some improvements using uninformative sender signals, self-communication, or receiver-only training.
  • The authors find trainable bridges can provide parameter-efficient adaptation without using useful sender information.
  • PACT-Check measures how much a trained system's performance gain genuinely results from communication.

#language models #model transfer #parameter-efficient adaptation #reinforcement learning

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.06976v1https://arxiv.org/abs/2611.06976 Read discussion