Oriel — AI & technology worth noticing, every day.

Causal Discovery Bench: Can LLMs Autonomously Conduct Effective Experiments?

Paper · https://arxiv.org/abs/2611.09934

arXiv:2611.09934v1 Announce Type: new Abstract: As large language model (LLM) agents get actively adopted for autonomous research, they must not only reason over existing evidence, but also design informative experiments, allocate resources, analyze sequential observations, and decide when to stop and make conclusions. Yet these capabilities remain poorly measured. We introduce Causal-Discovery-Bench (https://github.com/mshen1019/causal-discovery-bench), a verifiable sandbox with configurable structural equation models and resource constraints to evaluate sequential experimental design and causal discovery without domain-semantic cues. Across 3,802 episodes with 13 models from 5 vendors, we analyze agent behavior along 40 metrics in seven families. Agents' performance exhibits a four-level difficulty hierarchy spanning single-experiment design, sequential exploration of causal relationships, resource-aware experimentation balancing cost and statistical power, and autonomous resolution of conflicting evidence. We identify a substantial performance gap between causal inference from correctly curated evidence and navigating causal discovery through autonomous experimentation: Analyzing agents' action trajectories attributes 94.5% of their mistakes to experimental design rather than to inference, which is further consolidated with ablation studies. Relaxing resource constraints alone does not reliably resolve failures in sequential discovery. In contrast, basic statistical tools substantially improve performance across multiple model families, while advanced tools provide little additional benefit when models lack prerequisite lower-level capabilities.

First seen by Oriel on .

AI summary

Causal-Discovery-Bench evaluates whether LLM agents can autonomously discover causal relationships through sequential experiments, finding that experimental design—not inference—is the main source of failure.

  • The benchmark uses configurable structural equation models and resource constraints, with no domain-semantic cues.
  • Across 3,802 episodes and 13 models from 5 vendors, it measures agent behavior using 40 metrics in seven families.
  • Performance falls across four difficulty levels, from single-experiment design to resolving conflicting evidence autonomously.
  • Trajectory analysis attributes 94.5% of mistakes to experimental design; relaxing resource constraints alone does not reliably fix sequential discovery failures.
  • Basic statistical tools improve performance across multiple model families, while advanced tools add little when prerequisite capabilities are missing.

#causal discovery #llm agents #experimental design

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.09934v1https://arxiv.org/abs/2611.09934 Read discussion