Databricks Lakehouse: where the data lake finally meets the warehouse

For a decade, every serious data platform ran two systems. A data lake to dump everything into cheap object storage, and a data warehouse to serve the clean, modelled tables that the business actually queried. Two copies of the data, two sets of pipelines, two bills, and a permanent argument about which one held the truth.

The lakehouse is the attempt to collapse that split into one. Databricks did not invent the storage or the file format, but it packaged the pattern into something a team can run in production. After several of these builds, here is what is real and what is marketing.

Two worlds, and why teams ran both

A data lake is object storage — Azure Data Lake Storage Gen2, S3, GCS — holding raw files in any format. It is cheap, infinitely scalable, and schema-on-read: you decide the structure when you query, not when you write. The cost is that a lake with no discipline becomes a swamp. No transactions, no schema enforcement, no easy way to update a single row.

A warehouse is the opposite. Structured tables, ACID guarantees, fast SQL, real governance. The cost is rigidity and price: you pay to load data in, you pay for proprietary storage, and machine-learning workloads that want raw files do not fit the model.

Running both means copying data between them and reconciling the copies forever. The lakehouse question is simple: what if the lake could behave like a warehouse without a second copy?

What a lakehouse actually is

The trick is an open table format layered over the files in object storage. On Databricks that is Delta Lake: Parquet data files plus a transaction log. That log is the whole game. It gives you:

• ACID transactions — concurrent writes no longer corrupt a table.
• Schema enforcement and evolution — bad data is rejected, good changes are versioned.
• Time travel — query the table as it was last Tuesday, or roll back a bad load.
• Upserts and deletes — MERGE works, which matters for GDPR erasure and change-data-capture.

Because the data stays as open Parquet in your own storage account, one copy serves everything: SQL dashboards, streaming jobs, and ML training all read the same tables. No lock-in on the bytes, no export step. That is the payoff.

The medallion architecture

The lakehouse gives you the primitives; the medallion architecture is how you keep them tidy. Data flows through three quality tiers:

1. Bronze — raw ingestion, exactly as it arrived. Append-only, auditable, never edited.
2. Silver — cleaned, de-duplicated, conformed. Joined and typed, but still granular.
3. Gold — business-level aggregates and star schemas that dashboards and reports consume.

The discipline matters more than the tooling. Bronze is your insurance policy — if a silver transformation is wrong, you reprocess from bronze instead of re-ingesting from source. Skip it and you lose the ability to fix history.

Unity Catalog: governance you cannot bolt on later

Unity Catalog is the piece most teams underestimate. It is a single governance layer across every workspace: fine-grained access control, column and row masking, and — the part that saves you in an audit — automatic lineage. You can see which report depends on which gold table, which depends on which silver job, all the way back to the raw file.

Governance is not a feature you retrofit once you have five hundred tables and no idea who can read what. Turn it on at table one. The migration cost later is brutal.

Performance: Photon and the SQL warehouse

The old objection to lakes was speed: querying files could not compete with a tuned warehouse. Two things closed the gap. Photon is a vectorised query engine written in C++ that accelerates SQL and DataFrame workloads without changing your code. And Databricks SQL warehouses give analysts a familiar endpoint — they connect a BI tool and query gold tables, unaware there is object storage underneath.

Storage and compute are separate, so you scale query clusters up for a heavy month-end and down again without touching the data. That elasticity is where the cloud economics actually land in your favour — the same discipline I argue for in calm AI platforms.

When the lakehouse is the wrong answer

It is not a universal upgrade. Be honest about scale:

• Small, purely relational data. If everything fits comfortably in Postgres and you only serve BI, a managed warehouse is simpler and cheaper. A lakehouse is machinery you do not need.
• No engineering capacity. Delta, clusters, and the medallion flow are real operational surface. A tiny team drowns.
• Pure, stable star schemas with no ML or streaming. The lakehouse shines when workloads are mixed. If yours never will be, the flexibility is dead weight.

---

The lakehouse earns its place when you genuinely have both worlds — raw and streaming data feeding ML and clean tables feeding the business — and you are tired of paying twice to keep them in sync. Start with Delta and the medallion tiers, turn on Unity Catalog immediately, and let Photon worry about speed. Next in this series: how Microsoft plays the same game with Fabric, OneLake and Copilot Studio.