From 9ecd455310cd5a74371729b0939d3fd16a03b15b Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 25 Apr 2026 01:20:02 -0700 Subject: [PATCH] docs(parquet-design): reframe internal mapping as logical-only The previous wording suggested SeaweedFS would split a Parquet file into per-column-chunk needles, which contradicts the design's "do not modify the file" goal. Clarify that the per-column-chunk view is a planning abstraction over byte ranges in the unchanged object. --- PARQUET_PUSHDOWN_DESIGN.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/PARQUET_PUSHDOWN_DESIGN.md b/PARQUET_PUSHDOWN_DESIGN.md index 1edcfdd26..ca4c6cb56 100644 --- a/PARQUET_PUSHDOWN_DESIGN.md +++ b/PARQUET_PUSHDOWN_DESIGN.md @@ -122,9 +122,9 @@ SeaweedFS may store side indexes internally or in a hidden namespace: The original Parquet file is not modified. -## SeaweedFS-Aware Internal Mapping +## Logical View for Planning -A Parquet file can be represented internally as: +The Parquet object on disk is unchanged. For planning and pushdown purposes, SeaweedFS exposes a logical view of the file derived from the footer: ```text Parquet file @@ -141,19 +141,19 @@ Parquet file └── column chunk: payload ``` -SeaweedFS needle mapping may be: +Each leaf maps to a `(file, offset, length)` byte range inside the original object: ```text -footer -> cached metadata needle -row_group_0 / column_id -> data needle(s) -row_group_0 / column_timestamp -> data needle(s) -row_group_0 / column_tenant_id -> data needle(s) -row_group_0 / column_payload -> data needle(s) -row_group_1 / column_id -> data needle(s) +footer -> (file, footer_offset, footer_length) +row_group_0 / column_id -> (file, offset, length) +row_group_0 / column_timestamp -> (file, offset, length) +row_group_0 / column_tenant_id -> (file, offset, length) +row_group_0 / column_payload -> (file, offset, length) +row_group_1 / column_id -> (file, offset, length) ... ``` -This does not require physically splitting the Parquet file for compatibility. SeaweedFS can also map byte ranges inside one object to the underlying needles. +This is a planning abstraction only. The Parquet file is not split into per-column-chunk needles, and the on-disk object is not modified. Reads are still issued as ranged GETs against the original object. ## Side Index Types