Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering with Delta Lake + Unity Catalog

The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew , and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformation Diagnosing and fixing data skew with salting and adaptive query execution (AQE) Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files Governing access to the whole pipeline with Unity Catalog Architecture overview Pipeline shape — a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage — this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1 — Set up governed tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics ; CREATE SCHEMA IF NOT EXISTS retail_analytics . events ; CREATE TABLE IF NOT EXISTS retail_analytics . events . raw_orders ( order_id STRING , customer_id STRING , product_id STRING , quantity INT , event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders' ; CREATE TABLE IF NOT EXISTS retail_analytics . events . dim_products ( product_id STRING , category STRING , unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products' ; GRANT SELECT ON TABLE retail_analytics . events . raw_orders TO `analysts` ; Step 2 — Read and force a broadcast join for the small dimension table dim_products is s