A large equi-join often needs both inputs partitioned by the join key. If one side is small, Spark can instead copy that relation to executors and join it locally with partitions of the large side. The optimization is powerful because it changes the shape of the data movement.
Move the small side once per executor
In a broadcast hash join, the smaller input is collected and distributed so executors can build an in-memory lookup structure. Partitions of the larger input are then processed without shuffling that large relation by the join key. This can remove a major exchange from the physical plan.
Small is an operational property
A table with few rows can still be large because it contains wide strings or nested payloads. Size estimates may also be wrong when statistics are stale. Consider serialized size, concurrent work and executor memory—not only row count.
A broadcast join is fast when the build side is predictably small. It is dangerous when “small” is an assumption nobody measures.
Let the plan explain the decision
Spark can automatically broadcast inputs below configured thresholds when statistics support the choice. Hints can influence strategy, but they should not replace measurement. Inspect the physical plan for BroadcastHashJoin and check runtime metrics to confirm the intended side was broadcast.
joined = orders.join(products, "product_id")
joined.explain(mode="formatted")If Spark chooses a sort-merge join instead, investigate size estimates, join conditions and configuration before adding a hint. The optimizer may know something your row-count query did not capture.
Design dimensions for the join
Frequently joined reference data benefits from stable keys, narrow schemas and predictable update behavior. Selecting only required dimension columns before the join reduces the broadcast payload. If the dimension grows beyond safe bounds, use a distributed strategy rather than repeatedly raising thresholds.
The best broadcast optimization is often a clean, narrow dimension table—not a larger configuration value.
Go to the source
Keep learning with the official documentation.
Apache Spark: SQL performance tuning ↗Apache Spark: join hints ↗