Next-Level Benchmarking Overfitting – Thomas Neumann
Web page · https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html
First seen by Oriel on .
AI summary
The post argues GenDB-V’s reported 22× to 61× TPC-H speedups demonstrate benchmark overfitting through per-query precomputation and hard-coded dbgen assumptions unsuitable for production databases.
- At SF10, Umbra processes 15GB of compressed data in roughly 540ms, while memory bandwidth alone implies a lower bound near 280ms.
- GenDB-V builds query-specific auxiliary structures during loading, enabling near-constant SF100 runtimes including 0.7ms for Q2 and about 15ms for Q17.
- Its “plain-array” mode reports roughly 3× improvements but hard-codes dbgen row and key placement, making joins such as Q18 unusually cheap.
- The verified code checks its assumptions and falls back to a generic nested loop when they fail, but official TPC-H data never triggers that path.
- A single insertion or deletion could invalidate positional assumptions and force the slow fallback, making the approach fragile for production workloads.
#database benchmarking #tpc-h #ai agents #query optimization
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 64, latest observation ; +52 over observed period since
- Lobsters: 22, latest observation ; +19 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 13, latest observation ; +13 over observed period since
- Lobsters: 6, latest observation ; +6 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50452496 | http://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html Read discussion |
| Lobsters | nuoce5 | http://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html Read discussion |