mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-28 20:06:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(parquet-design): include DVs in execution flow and Phase 4 rollout

Browse Source 
The end-to-end delete flow listed only position-delete bitmaps and
equality-delete predicates, and Phase 4 only mentioned "position
delete masks" and "equality delete masks". Iceberg v3 deletion
vectors were covered in the dedicated subsection but never made it
into the operational flow or the rollout checklist, so a reader could
plausibly ship Phase 4 without DV support.

Add DV bitmap application as a distinct step (b) in the end-to-end
flow, and split Phase 4 into v2 position-delete ingestion, v3 DV
ingestion, equality-delete evaluation, partition matching, and
snapshot-aware identity. Phase 4 now lists the work that exists, not
a v2-only subset.
This commit is contained in:
Chris Lu
2026-04-25 01:55:34 -07:00
parent c3bfa48f25
commit c9c487282b
1 changed files with 14 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PARQUET_PUSHDOWN_DESIGN.md
+14 -7
View File
@@ -790,11 +790,16 @@ Strategy: evaluate equality-delete predicates at query time, accelerated by the
### End-to-end flow
```text
1. Iceberg catalog returns data files plus position+equality delete files
1. Iceberg catalog returns data files plus their attached delete files
(v2 position-delete files, v2 equality-delete files, v3 deletion
vectors), each tagged with PartitionSpecId / PartitionValues /
DataSequenceNumber so applicability is already resolved.
2. For each data file:
a. apply cached position-delete bitmap
b. evaluate live equality-delete predicates against side indexes
3. Pushdown execution returns candidate sets minus the union of (a) and (b)
a. apply cached merged position-delete bitmap (v2 position-delete files)
b. apply DV bitmap (v3 PositionDeletes + FileFormatPuffin entries)
c. evaluate live equality-delete predicates against side indexes
3. Pushdown execution returns candidate sets minus the union of (a),
(b), and (c).
```
## Compaction and Maintenance
@@ -879,9 +884,11 @@ For v1, side indexes inherit the same access controls as the underlying Parquet
### Phase 4: Iceberg Delete Integration
- position delete masks
- equality delete masks
- snapshot-aware index validity
- v2 position-delete file ingestion and merged-bitmap caching
- v3 deletion-vector (Puffin `deletion-vector-v1`) ingestion and bitmap caching
- v2 equality-delete evaluation accelerated by Phase 2 scalar indexes
- partition-spec / partition-value matching in the applicability rule
- snapshot-aware index validity (data-file identity + delete-file-set keying)
### Phase 5: Vector Indexes