The Lab Journal.
Practical field notes across data engineering, analytics engineering, machine learning, AI and the systems that connect them.
01What really happens when you hit Run?
Follow a Databricks job from orchestration to distributed execution—and learn where to look when it slows down.
04Unity Catalog is more than a permissions screen.
Understand the metastore, catalogs, schemas and securable objects as one governance hierarchy—not four unrelated settings.
05The Delta transaction log is the table's memory.
See how JSON commits, checkpoints and optimistic concurrency turn files in object storage into a reliable table.
06Declarative pipelines change what you operate.
Streaming tables, materialized views and expectations let you describe the desired data state while the pipeline manages execution.
07Spark waits because a plan is cheaper than a mistake.
Transformations build a logical plan; actions request results. Follow how that delay enables optimization and stage planning.
08A shuffle is data moving to meet its key.
Understand why joins and aggregations cross partition boundaries—and how to recognise expensive redistribution.
09Broadcast joins trade network shuffle for memory copies.
Learn when sending a small relation to every executor is faster—and when it becomes a stability risk.
02Partitions are the unit of parallelism.
Why more workers do not automatically make Spark faster, and how a shuffle redistributes your data.
03Bronze. Silver. Gold. But why?
Three layers are useful only when they represent three clear promises about your data.