From eeef173dd0a12bf640dda76036a15a47d16f1c38 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:42:42 -0700 Subject: [PATCH] docs(parquet-design): cover Iceberg v3 deletion vectors MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous "Handling Iceberg Deletes" section knew only about v2 position-delete files and equality-delete files. Iceberg v3 introduces deletion vectors (DVs) — Puffin "deletion-vector-v1" blobs holding a roaring bitmap of file-absolute row positions for one data file — which are not optional for v3-spec tables. Add a comparison table for the three delete forms and a DV subsection covering: per-data-file scope, the (puffin_path, offset, length) pointer, cache-by-blob-content, and the v2/v3 mixed case where DVs and position-delete files can co-exist during migration. Forward-link to the cache-key section that the next commit tightens. --- PARQUET_PUSHDOWN_DESIGN.md | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index 0fd7c9e8c..d7d650050 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -593,18 +593,43 @@ If the identity of a Parquet file changes, its indexes are invalidated and rebui ## Handling Iceberg Deletes -Iceberg uses two delete mechanisms, with different implications for indexing: +Iceberg's delete model has evolved across spec versions. Pushdown must support all three forms: -### Position deletes (precomputable) +| Form | Iceberg spec | Storage | Per-data-file? | Precomputable? | +|---|---|---|---|---| +| Position delete files | v2 | Parquet/Avro/ORC of `(file_path, position)` rows | No (one delete file may target many) | Yes (merge to bitmap) | +| Equality delete files | v2 | Parquet/Avro/ORC of equality-key rows | No (predicate scope by sequence number) | No (must evaluate per query) | +| Deletion vectors | v3 | Puffin blob (`deletion-vector-v1`) holding a roaring bitmap | Yes (one DV per data file) | Yes (it *is* the bitmap) | -Position deletes name `(data_file, row_position)` pairs. They can be merged into a per-data-file roaring bitmap and cached as a side index: +### Position delete files (v2; precomputable) + +Position deletes name `(data_file, row_position)` pairs. The set of position-delete files that apply to a given data file is `{ pdf : pdf.data_sequence_number >= data_file.data_sequence_number AND pdf.referenced_data_file in {NULL, data_file.path} }`. Pushdown merges that set into a per-data-file roaring bitmap and caches it as a side index: ```text //...//deletes.position.bitmap ``` +The cached bitmap is only valid for the exact set of position-delete files that produced it — see [Cache key](#position-delete-bitmap-cache-key) below. + Pushdown subtracts this bitmap from candidate row sets before returning results. +### Deletion vectors (v3; precomputable) + +Iceberg v3 replaces position delete *files* with deletion *vectors*: a roaring bitmap of file-absolute row positions for one specific data file, stored as a Puffin blob of type `deletion-vector-v1`. Because each DV already targets a single data file and is already a bitmap, no merge is needed — the cached form is just the decoded bitmap, keyed by the DV blob's content (offset, length, hash) inside its Puffin file. + +Multiple DV blobs may live in one Puffin file, so the per-data-file pointer is `(puffin_file_path, blob_offset, blob_length)`, not just a file path. The pushdown request must carry that triple per data file (see [DeleteFileRef](#pushdown-request)). + +Strategy: + +```text +1. resolve referenced Puffin file + blob offset/length from the manifest +2. range-GET the Puffin blob +3. decode roaring bitmap (cached) +4. subtract from candidate row sets +``` + +For tables that mix v2 position-delete files and v3 DVs (legal during migration), pushdown applies both: union the DV bitmap with the merged-position-delete bitmap before subtracting. + ### Equality deletes (must evaluate at query time) Equality deletes carry a predicate (e.g. `id = 42`) and apply only to data files whose Iceberg sequence number is **strictly less than** the delete file's sequence number. Data files added at or after the delete file's sequence number are not affected — those rows simply never existed when the delete was written. This is why the per-data-file pushdown request must carry both the data file's sequence number and the delete files attached to it: the planner has already resolved this scoping using sequence numbers.