From b114b318ae9bac338842c34e445248c1fffea3f4 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:20:21 -0700 Subject: [PATCH] docs(parquet-design): tighten index identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop modification time from the identity tuple — it is not stable across replication and metadata-only operations and is unnecessary alongside ETag or Iceberg manifest fields. Order remaining fields by strength, with Iceberg manifest identity preferred when available. --- PARQUET_PUSHDOWN_DESIGN.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index ca4c6cb56..5c1894e7a 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -479,20 +479,18 @@ Spark/Trino/DuckDB connector ## Index Consistency -Indexes must be tied to immutable file identity: +Indexes must be tied to a stable file identity. Preferred identity, in order of strength: ```text -file path -file size -etag/version -modification time -Iceberg snapshot id -content hash, optional +1. Iceberg manifest fields: file_path + file_size_in_bytes + record_count + (+ snapshot id for cross-snapshot lookups) +2. S3 ETag (canonical for raw S3 access without an Iceberg manifest) +3. Content hash (optional; expensive but fully unambiguous) ``` -If the Parquet file changes, indexes are invalidated and rebuilt. +Modification time is intentionally not used as identity: replication, re-upload, and metadata-only operations can change mtime without changing content, and content can change without mtime moving on some backends. -For Iceberg tables, this is easier because data files are generally immutable. New snapshots add or remove files rather than modifying files in place. +If the identity of a Parquet file changes, its indexes are invalidated and rebuilt. Iceberg makes this cheap because data files are immutable; new snapshots add or remove files rather than modifying files in place. ## Handling Iceberg Deletes