From 6c826e710a5e8207e83405f8b098c8bb8327fa8b Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 15:07:55 -0700 Subject: [PATCH] docs(parquet-design): document page-pruning consumption contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A bare list of page byte ranges is not consumable by every Parquet reader — it requires a page-aware decoder (parquet-go's ColumnChunkReader.ReadPage, Arrow's PageReader, DuckDB's internal reader). Stock S3-aware clients only support row-group reads, and feeding them a page byte range as if it were a row-group range produces undefined output. Spell out the two supported consumption modes: - page-aware reader: consume Pages directly, fetching the dictionary page region alongside as already merged in; - row-group reader: ignore Pages, fall back to RowGroups, optionally request a row-selection vector to apply page-level filtering post-decode. Note that v1 returns Pages only and the row-selection-vector output is a documented future revision (post-M6) to keep the wire shape simple now. --- PARQUET_PUSHDOWN_DESIGN.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index 7754fcf0e..f1999cf9a 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -355,6 +355,17 @@ Use row group statistics, bloom filters, and side indexes. Use page-level indexes to read only matching pages. +#### Connector consumability + +Returning `[]PageRef` as `(file, row_group, column, page, offset, length)` is only useful to readers that can issue page-granular I/O on a single Parquet file. That includes parquet-go's `ColumnChunkReader.ReadPage`, Arrow C++/PyArrow with `parquet::PageReader`, and DuckDB's internal Parquet reader; it does *not* include a stock S3-aware Parquet client that only supports row-group-level reads. Two consumption modes are supported: + +- **Page-aware reader (preferred when available).** The connector issues range reads matching the returned page byte ranges plus the column chunk's dictionary-page region (already merged into the response per [Page-Level Index](#3-page-level-index)). It decodes only the surviving pages. +- **Row-group reader with row-selection.** Connectors without page-aware I/O ignore the `Pages` list and consume `RowGroups` instead. They then apply the same predicate locally, or call back for a row-selection vector (the planner can emit one — see below — keyed on row-group-local row position) to filter post-decode without re-evaluating the predicate. + +When the response carries `Pages` but the connector cannot consume them, it must drop to `RowGroups` granularity rather than reinterpret page byte ranges as row-group ranges. The page byte ranges do not align to row-group boundaries and are not safe to feed to a row-group-only reader. + +Future revision (post-M6): the response may add an optional row-selection bitmap per row group so a row-group reader can apply page-level pruning without a page-aware decoder. v1 returns `Pages` only; downstream consumers fall back as described above. + ### Column Pruning Read only requested columns and predicate columns.