The Lab Journal.
Practical field notes across data engineering, analytics engineering, machine learning, AI and the systems that connect them.
07Apache Spark · 5 min
Spark waits because a plan is cheaper than a mistake.
Transformations build a logical plan; actions request results. Follow how that delay enables optimization and stage planning.
08Apache Spark · 6 min
A shuffle is data moving to meet its key.
Understand why joins and aggregations cross partition boundaries—and how to recognise expensive redistribution.
09Apache Spark · 5 min
Broadcast joins trade network shuffle for memory copies.
Learn when sending a small relation to every executor is faster—and when it becomes a stability risk.
02Apache Spark · 5 min
Partitions are the unit of parallelism.
Why more workers do not automatically make Spark faster, and how a shuffle redistributes your data.