Apache Spark Partitions Explained

A partition is a piece of the work

Spark divides distributed collections into partitions. Within a stage, tasks process partitions, while available execution slots limit how many tasks run simultaneously. A partition is not a machine, and it is not always equivalent to one source file. File layout and execution partitioning are related but distinct concepts.

More partitions can expose more parallelism, but too many tiny tasks add scheduling overhead.

Why a shuffle exists

Suppose two input partitions both contain orders for Mumbai. A filter can often work independently on each partition. An aggregation by city needs to bring related records or partial aggregates together. A shuffle redistributes data across partitions to enable that work. Network transfer, serialization, and disk activity can make it expensive.

Read the plan before changing the cluster

Try this tiny demonstration in a Spark notebook. The numbers are for learning, not a production recommendation. The modulo expression creates a repeatable set of city identifiers. Compare a row-level filter with a grouped count, then inspect the plans and execution metrics.

from pyspark.sql import functions as F

df = spark.range(0, 100000, numPartitions=8)
orders = df.withColumn("city_id", F.col("id") % 10)
orders.filter("id > 500").explain()
orders.groupBy("city_id").count().explain()

Look for an exchange around the aggregation. Actual plans depend on the optimizer and configuration. The purpose is to connect your transformation to the work Spark performs.

Watch for uneven work

Now imagine that 80% of incoming orders share the same customer key. An operation partitioned by that key can leave one task with far more work than its peers. Before increasing capacity, compare task durations and input sizes. Depending on the operation, better key design, pre-aggregation, or a skew-aware strategy may help. Measure the result; there is no universally correct partition count.

Tune for the distribution of your data, not just its total size.