From 08889b0f7b57a4b726e499ef66cb8c27175062ae Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:46:47 -0700 Subject: [PATCH] docs(parquet-design): identify columns by Iceberg field ID String column refs are fragile under Iceberg schema evolution: a rename leaves indexes built under the old name unreachable, and a drop-and-re-add of the same name silently aliases two different columns onto the same index path. Replace string Column / Columns []string fields with a ColumnRef struct (FieldId int32 + Path string hint), used in ParquetPushdownRequest.Columns, VectorQuery.Column, and PageRef.Column. The server trusts FieldId when set and falls back to Path only for non-Iceberg-managed Parquet files where field IDs are absent. Side-index file names are also keyed by field ID (bitmap.fid_7, vector.fid_42.ivf) so that rename and re-add stay unambiguous on disk. --- PARQUET_PUSHDOWN_DESIGN.md | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index 71bbc8912..1a4804a1c 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -117,15 +117,15 @@ Logical layout per data file: ```text //data/ds=2026-01-01/part-00001.parquet// footer.cache - page_index.timestamp - bloom.user_id - bitmap.tenant_id - btree.timestamp - inverted.message - vector.embedding.ivf + page_index.fid_3 # column "timestamp", field id 3 + bloom.fid_5 # column "user_id", field id 5 + bitmap.fid_7 # column "tenant_id", field id 7 + btree.fid_3 # column "timestamp", field id 3 + inverted.fid_11 # column "message", field id 11 + vector.fid_42.ivf # column "embedding", field id 42 ``` -`` is derived from the index identity rules in [Index Consistency](#index-consistency). The original Parquet file is not modified. +Per-column index files are keyed by Iceberg field ID, not column name. A rename (e.g. `tenant_id` → `org_id`) does not invalidate the index, and a column dropped-and-re-added with the same name (a different field ID) does not silently reuse the wrong index. Side-index metadata may carry the human-readable name as a hint for logging. `` is derived from the index identity rules in [Index Consistency](#index-consistency). The original Parquet file is not modified. ## Logical View for Planning @@ -320,7 +320,7 @@ Possible index types: Index layout: ```text -//...//vector.embedding.ivf/ +//...//vector.fid_42.ivf/ centroids list_000001 list_000002 @@ -440,7 +440,7 @@ type ParquetPushdownRequest struct { // Iceberg planning. DataFiles []DataFileDescriptor - Columns []string + Columns []ColumnRef PredicateKind PredicateKind // SUBSTRAIT or ICEBERG_EXPRESSION Predicate []byte // serialized per PredicateKind VectorQuery *VectorQuery @@ -544,12 +544,25 @@ const ( ) type VectorQuery struct { - Column string + Column ColumnRef Vector []float32 Metric VectorMetric TopK int NProbe int } + +// ColumnRef identifies a column by Iceberg field ID, which is stable +// across rename and reordering. The Path is an optional hint (the +// dotted Iceberg name path, e.g. "user.address.zip") for logging and +// for engines that cannot resolve field IDs; the server must trust +// FieldId when it is set and only fall back to Path when FieldId is +// zero (unspecified) — for example, when querying tables that were +// not written through Iceberg and have no field IDs in the Parquet +// file metadata. +type ColumnRef struct { + FieldId int32 // Iceberg field ID; 0 means "use Path" + Path string // dotted Iceberg name path, optional hint +} ``` The client is responsible for Iceberg planning (resolving the snapshot to data files and delete files) and passes the resolved set in `DataFiles`. The server treats this list as authoritative and does not re-read the catalog. Each descriptor carries: @@ -588,7 +601,7 @@ type RowGroupRef struct { type PageRef struct { File string RowGroup int - Column string + Column ColumnRef Page int Offset int64 Length int64