Managed Data Lake: A Guide for 2027
Managed Data Lake: A Guide for 2027 Apache Iceberg is the standard table format for production data lakes in 2027. Every major engine reads and writes it natively. The catalog ecosystem standardized on REST. You own your data on commodity storage with no lock-in. But Iceberg deliberately separates the table format from the system that keeps tables healthy. It gives you the primitives for maintenance — rewrite_data_files , expire_snapshots , remove_orphan_files , rewrite_manifests — but not the intelligence to decide when, how, and in what order to run them. Without that operational layer, every Iceberg table degrades over time: small files accumulate, snapshots bloat metadata, sort orders drift from query patterns, orphan files inflate storage costs, and query performance decays silently until something breaks visibly. This operational gap is the central challenge of running a data lake at production scale. Netflix built four internal services to address it — Autotune for compaction strategy selection, Polaris for catalog management, janitors for garbage collection, Metacat for cross-service observability — each staffed by dedicated teams over multiple years. Google engineered automatic compaction and garbage collection directly into BigLake , so their managed Iceberg tables stay healthy regardless of write volume or query pattern changes. In 2027, you do not need to replicate that investment. This guide covers what "managed" actually means for a data lake, the degradation mechanics that make it necessary, the control plane architecture that solves it, and the practical paths to getting there — whether you are running 50 tables or 5,000. Why Lakes Degrade — The Mechanics The degradation pattern is predictable and present in nearly every Iceberg lake running for more than three months without dedicated maintenance. Understanding these mechanics is necessary regardless of which management approach you choose. The Small-File Problem Every streaming writer — Flink, Spar