Oriel — AI & technology worth noticing, every day.

The Benchmarking Crisis

Web page · https://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html

First seen by Oriel on .

AI summary

The article argues that AI-driven overfitting is making database benchmarks like ClickBench meaningless and proposes hidden, evolving workloads with updates and transactional behavior.

  • pgrust exploits ClickBench specifics by hashing strings without checking content, caching repeated predicates, and reading little data for an unordered limited query.
  • ClickBench permits overfitting because its queries are simple, underspecified, read-only, and share predicates, while lacking an update stream.
  • Meaningful benchmarks should forbid workload-specific knowledge and intermediate-result caching, include updates, and evaluate systems that predate the benchmark.
  • The article suggests hidden queries or freshly generated workloads, borrowing ideas from Kaggle and LiveBench to prevent benchmark-specific optimization.

#database benchmarking #clickbench #sqlstorm #ai code generation

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Lobsters: 11, latest observation ; +10 over observed period since

Comments

Comment count reported by the source.

  • Lobsters: 9, latest observation ; +9 over observed period since

Aliases

SourceExternal IDURL
Lobsterssmtpjehttp://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html Read discussion