95% of My PySpark Job Finished in 4 Minutes. The Last Task Took 40. Here's Why.

I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On paper, it looked like it should scale fine across the cluster. In practice, the job would race through most of its tasks and then stall. The Spark UI told the real story: almost every task finished in a few minutes, but one or two tasks ran for over 40 minutes while their executors sat at 80–90% CPU, and the rest of the cluster sat mostly idle waiting for them to finish. This post walks through what data skew actually is, how I confirmed it was the cause, and the fix that brought the job back under control. What data skew actually is Data skew happens when one or a few keys hold a disproportionate share of the data. When Spark distributes work across partitions — usually via hash partitioning on a join or group-by key — all the rows for a given key land in the same partition. If one key has millions of rows and most others have a few thousand, that one partition (and the single task processing it) ends up doing far more work than every other partition combined. The result is a job where 95% of tasks look completely healthy, and the remaining 5% become the actual bottleneck. Total job time is dictated by the slowest task, not the average one — so a single skewed key can dominate your entire runtime even if it represents a tiny fraction of your total row count. How I confirmed it The Spark UI's stage view was the first clue: a chart of task durations with almost every bar clustered together, and one or two bars stretching far beyond the rest. That pattern — uniform short tasks plus one long outlier — is close to a signature for skew. To confirm which key was responsible, I ran a simple aggregation on the join key before doing anything else: from pyspark.sql import functions as F df . groupBy ( " customer_id " ) \ . count () \ . orderBy ( F . desc ( " count " )) \ . show ( 20 ) The output made it obvious: a small number of cu