PostHog Moves Data Warehouse Off DuckDB to Trino
PostHog is migrating its data warehouse away from DuckDB after discovering the single-node architecture couldn’t scale to its production demands. The company switched to DuckDB five months ago as a better alternative to ClickHouse, which struggled with complex queries and lacked cost-based optimization. While DuckDB excels for single-machine analytics, PostHog found that cloud deployment exposed fundamental limitations: resource prediction became difficult, customers needed larger instances than expected, and the Postgres connection layer proved harder to manage at scale than HTTP-based alternatives.
The lakehouse format DuckLake, which PostHog installed to make DuckDB warehouse-ready, introduced additional constraints. To maintain compatibility across multiple databases, the format prevented engine-specific optimizations critical at scale. PostHog encountered severe issues at petabyte scale, including catalog metadata bloat—where over 99% of statistics rows belonged to deleted tables—and write operations taking minutes per commit. The need to fork DuckLake with custom patches undermined a key benefit of open-source adoption.
PostHog is rebuilding its warehouse on a customized Trino deployment paired with HogLake, its own table format designed for distributed scale. The company is validating the architecture at its own petabyte-scale production workload before recruiting design partners, prioritizing reliability and predictable pricing over technology choices.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.