Oriel — AI & technology worth noticing, every day.

Exposure Is Not Commitment: Double-Stratified Evaluation of Latent Monitors for Tool-Using Agents

Paper · https://arxiv.org/abs/2610.37679

arXiv:2610.37679v1 Announce Type: new Abstract: Tool-using language agents are vulnerable to indirect prompt injection, and a growing literature probes model internals to catch unsafe actions before they execute, with sharply conflicting results. We argue much of the conflict is an evaluation artifact, and propose a double-stratification protocol: separate injection exposure from behavioral commitment, in both the evaluation and the training population. On the primary Qwen AgentDojo grid, monitors trained on the naive mixed unsafe-versus-rest contrast mostly learn exposure: they flag exposed trajectories whether or not the agent complies, and barely predict which cases turn unsafe. Retrained on exposed cases only, the same probe families predict the model's own unsafe tool call up to two steps early, the maximum the three-step trajectories admit (logistic AUROC 0.853, matched by a trained TF-IDF text baseline at 0.855). Mistral-7B reaches 0.896 and OLMo-2-7B 0.912; their naive contrasts learn different things: inverted exposure ranking on one, both axes on the other. Representations separate on exposure (probe 1.000 versus trained text 0.908) and where visible context is sanitized or unavailable. Training contrast alone reproduces the shape of recent pre-action-probe negative results, one candidate explanation for the conflict: the signal exists, but the model's own zero-shot forecast and NLA verbalizations do not reliably recover it. An unstratified AUROC overstates what a latent monitor knows, and what a naive contrast learns is population-dependent: evaluations should stratify both.

First seen by Oriel on .

AI summary

The paper proposes stratifying both evaluation and training data by injection exposure and behavioral commitment, finding that naive contrasts often detect exposure rather than predict unsafe tool calls.

  • On the Qwen AgentDojo grid, probes trained on exposed cases predict unsafe tool calls up to two steps early, with logistic AUROC 0.853.
  • A trained TF-IDF text baseline matches Qwen’s 0.853 score; Mistral-7B reaches 0.896 and OLMo-2-7B reaches 0.912.
  • Representations distinguish exposure with AUROC 1.000, compared with 0.908 for trained text, including when visible context is sanitized or unavailable.
  • Naive training contrasts produce population-dependent signals, while models’ zero-shot forecasts and NLA verbalizations do not reliably recover the unsafe-action signal.

#prompt injection #agent safety #latent monitoring #model evaluation

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.37679v1https://arxiv.org/abs/2610.37679 Read discussion