IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction
Paper · https://arxiv.org/abs/2611.12918
arXiv:2611.12918v1 Announce Type: new Abstract: Offline web-behavior benchmarks frequently insert the demonstrated action into a finite candidate set and then measure reranking. We show that this protocol can create a candidate-support confound: when the target is outside training support while all distractors are sampled from that support, support membership identifies the gold action without modeling behavior. We formalize this effect with support-identifiability rate (SIR), construct support-balanced candidate sets, and evaluate a target-free dual-path proposer that combines support retrieval with open-vocabulary DOM/LM proposals. On filtered OPeRA (4,282/469/992 train/dev/test steps), the original protocol has SIR 1.0000 on target-OOV rows and a support-only rule reaches 1.0000 OOV Exact@1. Support balancing reduces SIR to 0.0000; the strongest conditional reranker obtains 0.7214 overall Exact@1, 0.7988 on IV rows, and 0.5668 on OOV rows. The target-free dual-path system reaches Proposal Recall@16 0.6915 and end-to-end Exact@1 0.5212, compared with oracle conditional Exact@1 0.7450. On Mind2Web, the same audit yields SIR 0.2380 and an oracle-to-proposal gap of 0.2125. A 200-row semantic audit labels 0.7150 of exact-key OOV targets as semantically novel, 0.2450 as locator/string variation, and 0.0400 as ambiguous (Cohen's $\kappa=0.8173$). These measurements show why support composition must be controlled alongside candidate accuracy when offline results are used to reason about deployment.
First seen by Oriel on .
AI summary
IntentTwin identifies candidate-support confounding in offline web-behavior benchmarks and quantifies the performance gap between oracle candidate reranking and deployable target-free proposal systems.
- On filtered OPeRA, the original protocol has SIR 1.0000 on target-OOV rows, allowing a support-only rule to achieve 1.0000 OOV Exact@1.
- Support balancing reduces SIR to 0.0000; the strongest conditional reranker scores 0.7214 overall Exact@1, including 0.7988 IV and 0.5668 OOV.
- The target-free dual-path proposer combines support retrieval with open-vocabulary DOM/LM proposals, reaching Proposal Recall@16 0.6915 and end-to-end Exact@1 0.5212.
- Oracle conditional Exact@1 is 0.7450 on OPeRA, while Mind2Web shows SIR 0.2380 and an oracle-to-proposal gap of 0.2125.
- A 200-row audit classifies 0.7150 of exact-key OOV targets as semantically novel, 0.2450 as locator/string variations, and 0.0400 as ambiguous.
#web agents #behavior prediction #offline evaluation #candidate reranking
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.12918v1 | https://arxiv.org/abs/2611.12918 Read discussion |