Data engineering project

Lakehouse-Ledger

Governed medallion lakehouse on Databricks

See how it’s builtSource (opens in a new tab)
Illustration: bronze, silver and gold data layers with a quality check
  • Databricks
  • Delta Lake
  • Lakeflow Declarative Pipelines (DLT)
  • Unity Catalog
  • PySpark
  • Databricks SQL
  • GitHub Actions

The story

A Databricks pipeline that checks every record before it's used, keeps a history of every change, and has a dashboard showing how healthy the data is.

The hard part
Making re-runs safe: change data feed plus SCD Type 2 history means a repeated or backfilled batch could silently duplicate history, so every MERGE is guarded by a _batch_id and runs on the same code path as normal loads.

How it’s built

Inside Lakehouse-Ledger

System architecture

  1. STAGE 01

    • US EIA API · External
      • → Landing volume (API client)

  2. STAGE 02

    • Landing volume · Data
      • → Bronze: raw tables (Auto Loader)

  3. STAGE 03

    • Bronze: raw tables · Data
      • → DLT expectations

  4. STAGE 04

    • DLT expectations · Service
      • → Silver: clean tables (pass)
      • → Quarantine tables (fail)

  5. STAGE 05

    • Silver: clean tables · Data
      • → SCD2 dimension via MERGE (change feed)
      • → Gold KPI tables (aggregate)
    • Quarantine tables · Data
      • → gold.dq_metrics

  6. STAGE 06

    • SCD2 dimension via MERGE · Service
      • → Gold KPI tables (timeline join)

  7. STAGE 07

    • Gold KPI tables · Data
      • → gold.dq_metrics (observability log)
      • → Databricks SQL dashboard

  8. STAGE 08

    • gold.dq_metrics · Data
      • → Databricks SQL dashboard

  9. STAGE 09

    • Databricks SQL dashboard · Client

Engineering decisions

  1. Decision 01

    Rules as code, in one place

    Declarative DLT Expectations replace scattered if-checks. Every quality rule is visible, versioned and enforced on every row before it reaches Silver.

  2. Decision 02

    Quarantine, don't drop

    Rows that fail a rule go to quarantine tables with the reason attached. Bad data is kept for inspection instead of vanishing.

  3. Decision 03

    Change capture from the source of truth

    Delta Change Data Feed drives CDC and SCD Type 2 history instead of diffing full snapshots, so history is exact and cheap to compute.

  4. Decision 04

    Safe to run twice

    Every MERGE is guarded by a _batch_id, so any run can be repeated or backfilled on the same code path without duplicating history.

  5. Decision 05

    Quality you can see

    A dq_metrics Gold table feeds a live dashboard, and Unity Catalog grants are scoped per layer, so trust in the data is measured, not assumed.