Scaling Therapy Supervision: Assessing Psychotherapy Fidelity Using Large Language Models
Paper · https://arxiv.org/abs/2610.28772
arXiv:2610.28772v1 Announce Type: new Abstract: Psychotherapy fidelity assessment measures how faithfully a therapist adheres to a treatment protocol, a critical determinant of positive patient outcomes. However, fidelity assessment is bottlenecked by manual expert review, rendering the process labor-intensive and unscalable. To scale therapy supervision and support clinician self-learning, we present an integrated application of large language models (LLMs) that automates psychotherapy fidelity workflows and generates actionable supervisory feedback from authentic, long-form clinical transcripts. We evaluate three open-weight models across multiple task decomposition configurations and reasoning strategies, demonstrating that evidence-first reasoning improves fidelity estimation. Furthermore, we establish the framework's external construct validity on an independent randomized controlled trial (RCT), where it reliably distinguishes Cognitive Behavioral Therapy (CBT) from non-CBT sessions across core therapeutic skills. Through a user study that evaluates the generated feedback, mental health experts indicate that the pipeline synthesizes actionable clinical feedback. Ultimately, our work offers a practical, scalable solution toward bridging the community mental health supervision gap.
First seen by Oriel on .
No saved summary available.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2610.28772v1 | https://arxiv.org/abs/2610.28772 Read discussion |