标签：#lakehouse

Managed Data Lake: A Guide for 2027 Apache Iceberg is the standard table format for production data lakes in 2027. Every major engine reads and writes it natively. The catalog ecosystem standardized on REST. You own your data on commodity storage with no lock-in. But Iceberg deliberately separates the table format from the system that keeps tables healthy. It gives you the primitives for maintenance — rewrite_data_files , expire_snapshots , remove_orphan_files , rewrite_manifests — but not the intelligence to decide when, how, and in what order to run them. Without that operational layer, every Iceberg table degrades over time: small files accumulate, snapshots bloat metadata, sort orders drift from query patterns, orphan files inflate storage costs, and query performance decays silently until something breaks visibly. This operational gap is the central challenge of running a data lake at production scale. Netflix built four internal services to address it — Autotune for compaction strategy selection, Polaris for catalog management, janitors for garbage collection, Metacat for cross-service observability — each staffed by dedicated teams over multiple years. Google engineered automatic compaction and garbage collection directly into BigLake , so their managed Iceberg tables stay healthy regardless of write volume or query pattern changes. In 2027, you do not need to replicate that investment. This guide covers what "managed" actually means for a data lake, the degradation mechanics that make it necessary, the control plane architecture that solves it, and the practical paths to getting there — whether you are running 50 tables or 5,000. Why Lakes Degrade — The Mechanics The degradation pattern is predictable and present in nearly every Iceberg lake running for more than three months without dedicated maintenance. Understanding these mechanics is necessary regardless of which management approach you choose. The Small-File Problem Every streaming writer — Flink, Spar

📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p