From c9c487282be5f39d8fd2522049fc2d5c9680839f Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:55:34 -0700 Subject: [PATCH] docs(parquet-design): include DVs in execution flow and Phase 4 rollout The end-to-end delete flow listed only position-delete bitmaps and equality-delete predicates, and Phase 4 only mentioned "position delete masks" and "equality delete masks". Iceberg v3 deletion vectors were covered in the dedicated subsection but never made it into the operational flow or the rollout checklist, so a reader could plausibly ship Phase 4 without DV support. Add DV bitmap application as a distinct step (b) in the end-to-end flow, and split Phase 4 into v2 position-delete ingestion, v3 DV ingestion, equality-delete evaluation, partition matching, and snapshot-aware identity. Phase 4 now lists the work that exists, not a v2-only subset. --- PARQUET_PUSHDOWN_DESIGN.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index a5ef5f7bf..5a718ca90 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -790,11 +790,16 @@ Strategy: evaluate equality-delete predicates at query time, accelerated by the ### End-to-end flow ```text -1. Iceberg catalog returns data files plus position+equality delete files +1. Iceberg catalog returns data files plus their attached delete files + (v2 position-delete files, v2 equality-delete files, v3 deletion + vectors), each tagged with PartitionSpecId / PartitionValues / + DataSequenceNumber so applicability is already resolved. 2. For each data file: - a. apply cached position-delete bitmap - b. evaluate live equality-delete predicates against side indexes -3. Pushdown execution returns candidate sets minus the union of (a) and (b) + a. apply cached merged position-delete bitmap (v2 position-delete files) + b. apply DV bitmap (v3 PositionDeletes + FileFormatPuffin entries) + c. evaluate live equality-delete predicates against side indexes +3. Pushdown execution returns candidate sets minus the union of (a), + (b), and (c). ``` ## Compaction and Maintenance @@ -879,9 +884,11 @@ For v1, side indexes inherit the same access controls as the underlying Parquet ### Phase 4: Iceberg Delete Integration -- position delete masks -- equality delete masks -- snapshot-aware index validity +- v2 position-delete file ingestion and merged-bitmap caching +- v3 deletion-vector (Puffin `deletion-vector-v1`) ingestion and bitmap caching +- v2 equality-delete evaluation accelerated by Phase 2 scalar indexes +- partition-spec / partition-value matching in the applicability rule +- snapshot-aware index validity (data-file identity + delete-file-set keying) ### Phase 5: Vector Indexes