mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-28 03:46:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(parquet-design): cover Iceberg v3 deletion vectors

Browse Source 
The previous "Handling Iceberg Deletes" section knew only about v2
position-delete files and equality-delete files. Iceberg v3 introduces
deletion vectors (DVs) — Puffin "deletion-vector-v1" blobs holding a
roaring bitmap of file-absolute row positions for one data file —
which are not optional for v3-spec tables.

Add a comparison table for the three delete forms and a DV subsection
covering: per-data-file scope, the (puffin_path, offset, length)
pointer, cache-by-blob-content, and the v2/v3 mixed case where DVs and
position-delete files can co-exist during migration. Forward-link to
the cache-key section that the next commit tightens.
This commit is contained in:
Chris Lu
2026-04-25 01:42:42 -07:00
parent 317ee9d10c
commit eeef173dd0
1 changed files with 28 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PARQUET_PUSHDOWN_DESIGN.md
+28 -3
View File
@@ -593,18 +593,43 @@ If the identity of a Parquet file changes, its indexes are invalidated and rebui
## Handling Iceberg Deletes
Iceberg uses two delete mechanisms, with different implications for indexing:
Iceberg's delete model has evolved across spec versions. Pushdown must support all three forms:
### Position deletes (precomputable)
| Form | Iceberg spec | Storage | Per-data-file? | Precomputable? |
|---|---|---|---|---|
| Position delete files | v2 | Parquet/Avro/ORC of `(file_path, position)` rows | No (one delete file may target many) | Yes (merge to bitmap) |
| Equality delete files | v2 | Parquet/Avro/ORC of equality-key rows | No (predicate scope by sequence number) | No (must evaluate per query) |
| Deletion vectors | v3 | Puffin blob (`deletion-vector-v1`) holding a roaring bitmap | Yes (one DV per data file) | Yes (it *is* the bitmap) |
Position deletes name `(data_file, row_position)` pairs. They can be merged into a per-data-file roaring bitmap and cached as a side index:
### Position delete files (v2; precomputable)
Position deletes name `(data_file, row_position)` pairs. The set of position-delete files that apply to a given data file is `{ pdf : pdf.data_sequence_number >= data_file.data_sequence_number AND pdf.referenced_data_file in {NULL, data_file.path} }`. Pushdown merges that set into a per-data-file roaring bitmap and caches it as a side index:
```text
<system-prefix>/<table_uuid>/.../<identity>/deletes.position.bitmap
```
The cached bitmap is only valid for the exact set of position-delete files that produced it — see [Cache key](#position-delete-bitmap-cache-key) below.
Pushdown subtracts this bitmap from candidate row sets before returning results.
### Deletion vectors (v3; precomputable)
Iceberg v3 replaces position delete *files* with deletion *vectors*: a roaring bitmap of file-absolute row positions for one specific data file, stored as a Puffin blob of type `deletion-vector-v1`. Because each DV already targets a single data file and is already a bitmap, no merge is needed — the cached form is just the decoded bitmap, keyed by the DV blob's content (offset, length, hash) inside its Puffin file.
Multiple DV blobs may live in one Puffin file, so the per-data-file pointer is `(puffin_file_path, blob_offset, blob_length)`, not just a file path. The pushdown request must carry that triple per data file (see [DeleteFileRef](#pushdown-request)).
Strategy:
```text
1. resolve referenced Puffin file + blob offset/length from the manifest
2. range-GET the Puffin blob
3. decode roaring bitmap (cached)
4. subtract from candidate row sets
```
For tables that mix v2 position-delete files and v3 DVs (legal during migration), pushdown applies both: union the DV bitmap with the merged-position-delete bitmap before subtracting.
### Equality deletes (must evaluate at query time)
Equality deletes carry a predicate (e.g. `id = 42`) and apply only to data files whose Iceberg sequence number is **strictly less than** the delete file's sequence number. Data files added at or after the delete file's sequence number are not affected — those rows simply never existed when the delete was written. This is why the per-data-file pushdown request must carry both the data file's sequence number and the delete files attached to it: the planner has already resolved this scoping using sequence numbers.