mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-31 13:17:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(parquet-design): reframe internal mapping as logical-only

Browse Source 
The previous wording suggested SeaweedFS would split a Parquet file
into per-column-chunk needles, which contradicts the design's
"do not modify the file" goal. Clarify that the per-column-chunk view
is a planning abstraction over byte ranges in the unchanged object.
This commit is contained in:
Chris Lu
2026-04-25 01:20:02 -07:00
parent ca4c8d977d
commit 9ecd455310
1 changed files with 10 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PARQUET_PUSHDOWN_DESIGN.md
+10 -10
View File
@@ -122,9 +122,9 @@ SeaweedFS may store side indexes internally or in a hidden namespace:
The original Parquet file is not modified.
## SeaweedFS-Aware Internal Mapping
## Logical View for Planning
A Parquet file can be represented internally as:
The Parquet object on disk is unchanged. For planning and pushdown purposes, SeaweedFS exposes a logical view of the file derived from the footer:
```text
Parquet file
@@ -141,19 +141,19 @@ Parquet file
└── column chunk: payload
```
SeaweedFS needle mapping may be:
Each leaf maps to a `(file, offset, length)` byte range inside the original object:
```text
footer -> cached metadata needle
row_group_0 / column_id -> data needle(s)
row_group_0 / column_timestamp -> data needle(s)
row_group_0 / column_tenant_id -> data needle(s)
row_group_0 / column_payload -> data needle(s)
row_group_1 / column_id -> data needle(s)
footer -> (file, footer_offset, footer_length)
row_group_0 / column_id -> (file, offset, length)
row_group_0 / column_timestamp -> (file, offset, length)
row_group_0 / column_tenant_id -> (file, offset, length)
row_group_0 / column_payload -> (file, offset, length)
row_group_1 / column_id -> (file, offset, length)
...
```
This does not require physically splitting the Parquet file for compatibility. SeaweedFS can also map byte ranges inside one object to the underlying needles.
This is a planning abstraction only. The Parquet file is not split into per-column-chunk needles, and the on-disk object is not modified. Reads are still issued as ranged GETs against the original object.
## Side Index Types