RC RANDOM CHAOS

PostHog Moves Data Warehouse Off DuckDB to Trino

· via Hacker News

Original source

Moving our warehouse off DuckDB

Hacker News →

PostHog is migrating its data warehouse away from DuckDB after discovering the single-node architecture couldn’t scale to its production demands. The company switched to DuckDB five months ago as a better alternative to ClickHouse, which struggled with complex queries and lacked cost-based optimization. While DuckDB excels for single-machine analytics, PostHog found that cloud deployment exposed fundamental limitations: resource prediction became difficult, customers needed larger instances than expected, and the Postgres connection layer proved harder to manage at scale than HTTP-based alternatives.

The lakehouse format DuckLake, which PostHog installed to make DuckDB warehouse-ready, introduced additional constraints. To maintain compatibility across multiple databases, the format prevented engine-specific optimizations critical at scale. PostHog encountered severe issues at petabyte scale, including catalog metadata bloat—where over 99% of statistics rows belonged to deleted tables—and write operations taking minutes per commit. The need to fork DuckLake with custom patches undermined a key benefit of open-source adoption.

PostHog is rebuilding its warehouse on a customized Trino deployment paired with HogLake, its own table format designed for distributed scale. The company is validating the architecture at its own petabyte-scale production workload before recruiting design partners, prioritizing reliability and predictable pricing over technology choices.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.