Rank-Aware Differentially Private Release of Listwise Preferences for LLM Alignment
Paper · https://arxiv.org/abs/2610.37682
arXiv:2610.37682v1 Announce Type: new Abstract: Listwise preference feedback offers richer supervision for large language model (LLMs) alignment than pairwise comparisons, but a full ranking also reveals more sensitive user preferences. Existing privacy-preserving alignment methods focus mainly on pairwise feedback, while listwise alignment under local differential privacy (LDP) remains unexplored. Extending LDP to this listwise domain raises a critical challenge: the high sensitivity of the full ranking space demands excessive noise, neutralizing the benefits of listwise supervision. To address this challenge, we analyze the interaction between supervision granularity and privacy noise in downstream training, and introduce a rank-aware exponential mechanism that privatizes listwise preference data into a low-sensitivity granularity suitable for downstream alignment. The mechanism leverages ranking information to sample a fixed-size binary partition, concentrating the noise near borderline items instead of perturbing all items uniformly as randomized response (RR) does. Empirical evaluations on downstream LLM alignment tasks show that our mechanism consistently outperforms existing LDP baselines at matched privacy budgets. The code is available at https://github.com/tdsai-lab/RAEM.
First seen by Oriel on .
AI summary
The paper presents a rank-aware exponential mechanism for privately releasing listwise preferences, aiming to preserve useful ranking supervision for LLM alignment under local differential privacy.
- The method converts rankings into a fixed-size binary partition with lower sensitivity for downstream alignment.
- It uses ranking information to concentrate privacy noise near borderline items instead of perturbing all items uniformly as randomized response does.
- The authors report that the method consistently outperforms existing local differential privacy baselines at matched privacy budgets.
- The work addresses the trade-off between richer listwise supervision and the high privacy noise required for full rankings.
#llm alignment #differential privacy #preference learning
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2610.37682v1 | https://arxiv.org/abs/2610.37682 Read discussion |