From ac49cfd59b4b0c76316b6fcefcc31331291b2cb5 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 15:05:21 -0700 Subject: [PATCH] docs(parquet-design): equality deletes are key-row sets, not predicates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous "Equality deletes carry a predicate (e.g. id = 42)" framing is misleading. Iceberg equality-delete files are not serialized predicates — they are stored row sets keyed on a fixed set of columns named by the manifest's equality_ids list. A data row is deleted if its values on those columns match any delete-file row. Replace the predicate framing with an execution model that: - reads only the equality-key columns from the data file; - loads the equality-key rows from each applicable delete file; - joins on the key tuple using Iceberg's equality semantics (NULL matches NULL, Iceberg-defined NaN handling, canonical encoded comparison for decimals / timestamps / dates). Forward-link to a not-yet-added cache-reusability subsection that the next commit will make explicit. --- PARQUET_PUSHDOWN_DESIGN.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index ee3858a86..b6669c8cc 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -802,7 +802,16 @@ When pushdown receives a request, it computes the key from the position-delete e ### Equality deletes (must evaluate at query time) -Equality deletes carry a predicate (e.g. `id = 42`). The applicability rule is: +An equality-delete file is *not* a serialized SQL predicate. It is a list of rows, each carrying values for a fixed set of equality-key columns (the file's `equality_ids`). A data row in scope is deleted if its values for those columns match any row in the delete file. + +Execution model: + +1. Read the equality-key columns from the data file (using projection / page-level pruning over those columns only). +2. Read the equality-key rows from each applicable delete file. The set of these rows can be loaded once and reused across queries against the same data file (see [cache reusability](#equality-delete-bitmap-cache-reusability) below). +3. For each data row, look up its key tuple in the delete-file key set. Use Iceberg's equality semantics: `NULL` values match `NULL` (unlike SQL `NULL = NULL`), and floating-point comparisons follow Iceberg's documented rules (`NaN` is a single class for equality matching). Decimal, timestamp, and date types compare by canonical encoded value. +4. Mark matching data rows for deletion. + +The applicability rule for which equality-delete files apply is: ```text { edf :