Delta Lake Transaction Log Explained

A table is files plus decisions

A Delta table stores data files alongside a _delta_log directory. Each successful table version records actions such as adding a file, removing a previous file or updating table metadata. A remove action is initially a logical deletion; physical cleanup is handled later according to retention rules.

Readers choose one snapshot

A query begins from a specific table version and reads the files active in that snapshot. If another writer commits while the query is running, the existing reader does not suddenly mix old and new file sets. This snapshot model is the foundation for consistent reads and time travel.

Consistency comes from an agreed log of active files—not from renaming a folder at exactly the right millisecond.

Checkpoints reduce replay work

As versions accumulate, replaying every JSON commit from the beginning would become inefficient. Delta periodically writes checkpoints containing consolidated table state. A reader can begin from a suitable checkpoint and apply only the newer commits that follow it.

DESCRIBE HISTORY main.sales.orders;

SELECT *
FROM main.sales.orders VERSION AS OF 42;

History and time travel expose the versioned model, but availability depends on retained log and data files. VACUUM and retention choices therefore affect how far back a table can be reconstructed.

Writers coordinate optimistically

Writers prepare changes without placing a long exclusive lock on the entire table. Before committing, a writer checks whether a conflicting change was committed since it read the table. Compatible work can succeed; conflicting work is rejected and must be retried or redesigned. The exact conflict rules depend on the operation and isolation behavior.

A retry is safe only when the write itself is idempotent and the business rule still holds against the new table version.