Why we're moving our warehouse off DuckDB
Web page · https://posthog.com/blog/moving-off-duckdb
First seen by Oriel on .
AI summary
PostHog explains why it replaced DuckDB and DuckLake in its warehouse product with a system built around its own flavor of Trino and a new open table format, HogLake.
- Large, complex customer queries required oversized single-node workers, while sizing resources before a query arrived was difficult and disk spill made queries too slow or expensive.
- At roughly 2 PB, 1 trillion rows, and 15.3 million snapshots, DuckLake encountered write, catalog-statistics, and metadata-connection bottlenecks that were difficult to address within its compatibility constraints.
- Cleaning obsolete catalog statistics made writes 30 to 50 times faster, but the rows accumulated again; a single-table write could otherwise take more than three minutes.
- Direct Postgres connections over port 5432 proved harder to scale and support than HTTP because of connection management, load balancing, and client compatibility demands.
- PostHog says it tested Trino and HogLake at petabyte scale before inviting design partners to evaluate the warehouse.
#duckdb #trino #data warehouses #lakehouse
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 183, latest observation ; +139 over observed period since
- Lobsters: 7, latest observation ; +6 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 119, latest observation ; +101 over observed period since
- Lobsters: 9, latest observation ; +9 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50338747 | https://posthog.com/blog/moving-off-duckdb Read discussion |
| Lobsters | wpahfd | https://posthog.com/blog/moving-off-duckdb Read discussion |