Oriel — AI & technology worth noticing, every day.

Actionable Feedback: Improving Model Capabilities from User Edit Data on Clinical Documentation

Paper · https://arxiv.org/abs/2611.12904

arXiv:2611.12904v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being deployed in clinical documentation workflows, where clinicians review and edit their outputs before use. In principle, these clinician edits represent a scalable source of supervision for model training. In practice, not all edits represent actionable feedback: for instance, on data drawn from real-world deployment of an AI clinical documentation product, our review suggests that over 45% of edited outputs contain user-added information that is not directly supported by the inputs to the model. Naively fine-tuning on such edits risks teaching the model to hallucinate. In this paper, we develop a method to address this challenge. Given that real clinician edits on deployed documentation tools are not publicly available, we first develop a taxonomy of clinician-editing behaviors, informed by data from a deployed AI clinical documentation product. Then, we develop a method for generating semi-synthetic clinician-edit data, informed by our taxonomy, on a discharge-summary generation task based on MIMIC-III, a public dataset. Finally, we introduce simple filtering methods for identifying actionable edits, with accuracy evaluated by expert review. We show empirically, on our semi-synthetic dataset, that naive fine-tuning on raw edited output increases ungrounded content, while our approach of progressively filtering toward grounded edits enables improvement on clinician-alignment metrics while retaining faithfulness to the input context.

First seen by Oriel on .

AI summary

The paper presents filtering methods that turn clinician edits into safer training feedback, improving alignment on semi-synthetic discharge summaries without increasing unsupported content.

  • A review of deployed product data found that over 45% of edited outputs added information not directly supported by model inputs.
  • The authors created a taxonomy of clinician-editing behaviors based on data from a deployed AI clinical documentation product.
  • Because real clinician edits are unavailable publicly, they generated semi-synthetic edit data for a MIMIC-III discharge-summary task.
  • Expert-reviewed filtering progressively identifies grounded, actionable edits for fine-tuning.
  • Naive fine-tuning increased ungrounded content, while filtered edits improved clinician-alignment metrics and retained faithfulness to the input context.

#large language models #clinical documentation #fine-tuning #mimic-iii

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.12904v1https://arxiv.org/abs/2611.12904 Read discussion