Oriel — AI & technology worth noticing, every day.

Why we're moving our warehouse off DuckDB

Web page · https://posthog.com/blog/moving-off-duckdb

First seen by Oriel on .

AI summary

PostHog explains why it replaced DuckDB and DuckLake in its warehouse product with a system built around its own flavor of Trino and a new open table format, HogLake.

  • Large, complex customer queries required oversized single-node workers, while sizing resources before a query arrived was difficult and disk spill made queries too slow or expensive.
  • At roughly 2 PB, 1 trillion rows, and 15.3 million snapshots, DuckLake encountered write, catalog-statistics, and metadata-connection bottlenecks that were difficult to address within its compatibility constraints.
  • Cleaning obsolete catalog statistics made writes 30 to 50 times faster, but the rows accumulated again; a single-table write could otherwise take more than three minutes.
  • Direct Postgres connections over port 5432 proved harder to scale and support than HTTP because of connection management, load balancing, and client compatibility demands.
  • PostHog says it tested Trino and HogLake at petabyte scale before inviting design partners to evaluate the warehouse.

#duckdb #trino #data warehouses #lakehouse

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 183, latest observation ; +139 over observed period since
  • Lobsters: 7, latest observation ; +6 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 119, latest observation ; +101 over observed period since
  • Lobsters: 9, latest observation ; +9 over observed period since

Aliases

SourceExternal IDURL
Hacker News50338747https://posthog.com/blog/moving-off-duckdb Read discussion
Lobsterswpahfdhttps://posthog.com/blog/moving-off-duckdb Read discussion