mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-03 06:37:09 +00:00
docs(parquet-design): scope position-delete bitmap cache key
The previous text said the merged position-delete bitmap was "cached as a side index" without spelling out the cache key, leaving the implication that data-file identity was sufficient. It is not: the merged bitmap is a function of the *set* of position-delete files that currently target the data file, and that set changes as snapshots add new delete files or compaction removes them. Add a "cache key" subsection requiring (data_file_identity, sorted_set_of_input_delete_file_ids), with the v3 DV shorthand of (data_file_identity, puffin_blob_digest), and explain why snapshot_id alone is neither necessary nor sufficient as a key.
This commit is contained in:
@@ -630,6 +630,25 @@ Strategy:
|
||||
|
||||
For tables that mix v2 position-delete files and v3 DVs (legal during migration), pushdown applies both: union the DV bitmap with the merged-position-delete bitmap before subtracting.
|
||||
|
||||
### Position-delete bitmap cache key
|
||||
|
||||
The merged position-delete bitmap is a *function of the set of position-delete files that target the data file at the requested snapshot*, not of the data file alone. New snapshots add or compact away position-delete files, which silently changes the merged result. A cache keyed only by data-file identity will hand back stale bitmaps after the first new position-delete file lands.
|
||||
|
||||
Cache key per data file:
|
||||
|
||||
```text
|
||||
key = (
|
||||
data_file_identity, // see Index Consistency
|
||||
sorted_set_of_input_delete_file_ids, // each = (path, size, content_hash)
|
||||
)
|
||||
```
|
||||
|
||||
Equivalent shorthand for v3 DVs, which are intrinsically per-data-file: `(data_file_identity, puffin_blob_digest)` where `puffin_blob_digest` is the Puffin-recorded content hash of the DV blob.
|
||||
|
||||
`snapshot_id` is *not* a sufficient key — different snapshots can produce the same bitmap, and pinning the cache to snapshot id wastes work after no-op snapshots. Conversely, snapshot_id alone is *insufficient* across tables because position-delete files are not uniquely identified by snapshot number.
|
||||
|
||||
When pushdown receives a request, it computes the key from the request's `PositionDeletes` list (already supplied by the client's planner) and looks up or builds the bitmap. Cache eviction follows file identity: when a data file or any of its delete files goes out of scope (snapshot expiration, compaction), the corresponding cache entries are reclaimable.
|
||||
|
||||
### Equality deletes (must evaluate at query time)
|
||||
|
||||
Equality deletes carry a predicate (e.g. `id = 42`) and apply only to data files whose Iceberg sequence number is **strictly less than** the delete file's sequence number. Data files added at or after the delete file's sequence number are not affected — those rows simply never existed when the delete was written. This is why the per-data-file pushdown request must carry both the data file's sequence number and the delete files attached to it: the planner has already resolved this scoping using sequence numbers.
|
||||
|
||||
Reference in New Issue
Block a user