From c9871ce290fd5989c82ff340af608f3d1b61109b Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:43:11 -0700 Subject: [PATCH] docs(parquet-design): scope position-delete bitmap cache key The previous text said the merged position-delete bitmap was "cached as a side index" without spelling out the cache key, leaving the implication that data-file identity was sufficient. It is not: the merged bitmap is a function of the *set* of position-delete files that currently target the data file, and that set changes as snapshots add new delete files or compaction removes them. Add a "cache key" subsection requiring (data_file_identity, sorted_set_of_input_delete_file_ids), with the v3 DV shorthand of (data_file_identity, puffin_blob_digest), and explain why snapshot_id alone is neither necessary nor sufficient as a key. --- PARQUET_PUSHDOWN_DESIGN.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index d7d650050..8063d9d29 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -630,6 +630,25 @@ Strategy: For tables that mix v2 position-delete files and v3 DVs (legal during migration), pushdown applies both: union the DV bitmap with the merged-position-delete bitmap before subtracting. +### Position-delete bitmap cache key + +The merged position-delete bitmap is a *function of the set of position-delete files that target the data file at the requested snapshot*, not of the data file alone. New snapshots add or compact away position-delete files, which silently changes the merged result. A cache keyed only by data-file identity will hand back stale bitmaps after the first new position-delete file lands. + +Cache key per data file: + +```text +key = ( + data_file_identity, // see Index Consistency + sorted_set_of_input_delete_file_ids, // each = (path, size, content_hash) +) +``` + +Equivalent shorthand for v3 DVs, which are intrinsically per-data-file: `(data_file_identity, puffin_blob_digest)` where `puffin_blob_digest` is the Puffin-recorded content hash of the DV blob. + +`snapshot_id` is *not* a sufficient key — different snapshots can produce the same bitmap, and pinning the cache to snapshot id wastes work after no-op snapshots. Conversely, snapshot_id alone is *insufficient* across tables because position-delete files are not uniquely identified by snapshot number. + +When pushdown receives a request, it computes the key from the request's `PositionDeletes` list (already supplied by the client's planner) and looks up or builds the bitmap. Cache eviction follows file identity: when a data file or any of its delete files goes out of scope (snapshot expiration, compaction), the corresponding cache entries are reclaimable. + ### Equality deletes (must evaluate at query time) Equality deletes carry a predicate (e.g. `id = 42`) and apply only to data files whose Iceberg sequence number is **strictly less than** the delete file's sequence number. Data files added at or after the delete file's sequence number are not affected — those rows simply never existed when the delete was written. This is why the per-data-file pushdown request must carry both the data file's sequence number and the delete files attached to it: the planner has already resolved this scoping using sequence numbers.