From 317ee9d10c202a4c77e2b85663103640d4d681ab Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:31:39 -0700 Subject: [PATCH] docs(parquet-design): use planned scan size for pushdown threshold MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cost rule of thumb keyed off "total file size" of the table, which is the wrong input — a huge table that Iceberg has narrowed to a single 4 MiB file should not pay for a pushdown round-trip, while a medium table that planning leaves with thousands of small surviving files still benefits. Switch the threshold to planned scan bytes plus surviving file count, both observable after Iceberg planning, and update the configuration knob names accordingly. --- PARQUET_PUSHDOWN_DESIGN.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index 6f7049d99..0fd7c9e8c 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -665,13 +665,15 @@ In all cases, pushdown must degrade gracefully: if the requested side index is m Pushdown is not free. Each request adds a planning round-trip and forces the server to load and evaluate side indexes. For small or unselective queries this is a net loss versus a direct ranged GET. +The relevant cost input is the **planned scan after Iceberg pruning**, not the table's total size. A 100 TiB table where Iceberg planning has already narrowed the query to a single 4 MiB file is a poor candidate for pushdown; a 50 GiB table where Iceberg planning produces 5,000 surviving files is a great candidate even if each individual file is small. + Connectors should skip the pushdown API when: -- the table is small enough that a single Parquet scan is cheaper than the planning round-trip (rule of thumb: total file size below a few footer reads, ~10 MiB) +- the planned scan is small enough that a direct ranged GET is cheaper than the planning round-trip (rule of thumb: total planned scan bytes below ~10 MiB *and* surviving file count ≤ 1) - the predicate is non-selective (estimated selectivity > ~0.5) and no projection prunes meaningful columns - the query has no predicate and no vector clause — file pruning by Iceberg metadata alone already does the job -The pushdown API should expose its own selectivity and cost estimates in the response stats so connectors can learn when to use it; explicit configuration knobs (`min_table_size`, `min_selectivity`) can act as a fallback. +The pushdown API should expose its own selectivity and cost estimates in the response stats so connectors can learn when to use it; explicit configuration knobs (`min_planned_scan_bytes`, `min_planned_file_count`, `min_selectivity`) can act as a fallback. ## Security and Access Control