The Benchmarking Crisis
Web page · https://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html
First seen by Oriel on .
AI summary
The article argues that AI-driven overfitting is making database benchmarks like ClickBench meaningless and proposes hidden, evolving workloads with updates and transactional behavior.
- pgrust exploits ClickBench specifics by hashing strings without checking content, caching repeated predicates, and reading little data for an unordered limited query.
- ClickBench permits overfitting because its queries are simple, underspecified, read-only, and share predicates, while lacking an update stream.
- Meaningful benchmarks should forbid workload-specific knowledge and intermediate-result caching, include updates, and evaluate systems that predate the benchmark.
- The article suggests hidden queries or freshly generated workloads, borrowing ideas from Kaggle and LiveBench to prevent benchmark-specific optimization.
#database benchmarking #clickbench #sqlstorm #ai code generation
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Lobsters: 11, latest observation ; +10 over observed period since
Comments
Comment count reported by the source.
- Lobsters: 9, latest observation ; +9 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Lobsters | smtpje | http://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html Read discussion |