Oriel — AI & technology worth noticing, every day.

The Schema Coherence Inventory: Measuring Whether an Assigned Belief Organizes a Language Model's Cognition

Paper · https://arxiv.org/abs/2610.31683

arXiv:2610.31683v1 Announce Type: new Abstract: Persona-conditioned language models are used as simulated patients and affective agents, but there is no validated way to ask whether a belief assigned to such a model organizes its behaviour or whether the model has merely adopted a mood. We present the Schema Coherence Inventory (SCI): 96 forced-choice items at eight levels of Beck's cognitive model, each offering six replies keyed to three negative self-schemas and their positive counterparts, scored by length-normalized log-likelihood so that nothing is generated or parsed. Affect alone can move a model onto a default belief, so the discriminant criterion is target specificity: a condition built for one belief must select that belief more often than conditions built for the other beliefs of its pole. Six validation studies over 107 conditions on five backbones (1.7B-14B parameters) support this reading. Nine clinically trained raters recover the keyed belief at 79.5% (chance 16.7%). Belief adapters are target-specific on every backbone (+17.7 points; 30/30 belief cells positive), and a condition's SCI profile identifies its own free generations in 24/30 cases against 5 expected by chance. A 48-item short form correlates r = 0.73 with the other 48 items. Applied to six published patient simulators, none carries its formulation's core belief better than a one-sentence statement of it. The validation route, which separates a construct from the affect that accompanies it and checks model judges against clinicians, offers a template for affective measurement in language models; we outline its extension across task types, raters, sites and languages. We release the items, scoring code and short form.

First seen by Oriel on .

AI summary

The Schema Coherence Inventory (SCI) measures whether a language model’s assigned belief consistently organizes its behavior, distinguishing belief-specific responses from affect alone.

  • SCI uses 96 forced-choice items across eight levels of Beck's cognitive model, scored with length-normalized log-likelihood without generating or parsing responses.
  • Across six validation studies, nine clinically trained raters identified the keyed belief with 79.5% accuracy, compared with 16.7% chance.
  • Belief adapters showed target-specific effects on all five model backbones, with a 17.7-point gain and positive results in all 30 belief cells.
  • SCI profiles matched the models’ own free generations in 24 of 30 cases; a 48-item short form correlated r = 0.73 with the remaining items.
  • Among six published patient simulators, none represented its formulation’s core belief better than a one-sentence statement; the authors release the items and scoring code.

#language models #psychometrics #cognitive models #patient simulation

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.31683v1https://arxiv.org/abs/2610.31683 Read discussion