mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-19 22:44:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(parquet-design): include partition matching in delete applicability

Browse Source 
Iceberg delete files (position and equality) carry a partition spec
id and partition values, and apply only to data files in the same
partition. Both prose applicability rules (position deletes,
equality deletes) previously omitted this constraint, leaving them
correct only for unpartitioned tables.

- Add PartitionSpecId and PartitionValues to DataFileDescriptor and
  DeleteFileRef so the planner can pass the partition triple through.
- Rewrite both applicability rules to include partition equality
  alongside the sequence-number rule, formatted as set-builder
  predicates so each clause is explicit.
- Note that v3 deletion vectors get partition matching implicitly via
  ReferencedDataFile, but populating the fields anyway keeps the
  applicability check uniform.
This commit is contained in:
Chris Lu
2026-04-25 01:55:08 -07:00
parent b47bcd26c0
commit c3bfa48f25
1 changed files with 45 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PARQUET_PUSHDOWN_DESIGN.md
+45 -2
View File
@@ -463,6 +463,17 @@ type DataFileDescriptor struct {
// manifest entry itself).
DataSequenceNumber int64
// PartitionSpecId and PartitionValues identify the partition this
// data file belongs to, in the partition spec used at write time.
// Required for delete applicability: a delete file with a
// different (PartitionSpecId, PartitionValues) does not apply,
// even if its sequence number rule matches. PartitionValues are
// serialized as a Substrait/Iceberg literal map keyed by partition
// field id, encoded the same way the planner sees them in the
// manifest.
PartitionSpecId int32
PartitionValues []byte
// Deletes is the union of position-delete files, equality-delete
// files, and deletion vectors that apply to this data file.
// Content type is discriminated by DeleteFileRef.Content.
@@ -473,6 +484,17 @@ type DeleteFileRef struct {
Path string
SizeBytes int64
// PartitionSpecId and PartitionValues match the partition this
// delete file was written for. A delete file applies to a data
// file only when (delete.PartitionSpecId, delete.PartitionValues)
// == (data.PartitionSpecId, data.PartitionValues). This holds for
// both position-delete files and equality-delete files; deletion
// vectors set ReferencedDataFile and so the partition match is
// implicit but the fields should still be populated for
// consistency.
PartitionSpecId int32
PartitionValues []byte
// DataSequenceNumber is the delete file's data_sequence_number.
// The scope rule is: this delete file applies to a data file iff
// data_file.DataSequenceNumber <= delete_file.DataSequenceNumber
@@ -687,7 +709,18 @@ Iceberg's delete model has evolved across spec versions. Pushdown must support a
### Position delete files (v2; precomputable)
Position deletes name `(data_file, row_position)` pairs. The set of position-delete files that apply to a given data file is `{ pdf : pdf.DataSequenceNumber >= data_file.DataSequenceNumber AND pdf.ReferencedDataFile in {empty, data_file.Path} }`. Pushdown merges that set into a per-data-file roaring bitmap and caches it as a side index:
Position deletes name `(data_file, row_position)` pairs. The set of position-delete files that apply to a given data file is
```text
{ pdf :
pdf.DataSequenceNumber >= data_file.DataSequenceNumber // sequence
AND pdf.PartitionSpecId == data_file.PartitionSpecId // same spec
AND pdf.PartitionValues == data_file.PartitionValues // same partition
AND pdf.ReferencedDataFile in { empty, data_file.Path } // file scope
}
```
Pushdown merges that set into a per-data-file roaring bitmap and caches it as a side index:
```text
<system-prefix>/<table_uuid>/.../<identity>/deletes.position.bitmap
@@ -735,7 +768,17 @@ When pushdown receives a request, it computes the key from the position-delete e
### Equality deletes (must evaluate at query time)
Equality deletes carry a predicate (e.g. `id = 42`) and apply only to data files whose `data_sequence_number` is **strictly less than** the delete file's `data_sequence_number`. Data files whose `data_sequence_number` is equal to or greater than the delete file's are not affected — those rows simply never existed when the delete was written. (The Iceberg manifest entry's `file_sequence_number` is the commit-time sequence number for the manifest entry itself and is not used for delete scoping.) This is why the per-data-file pushdown request must carry both the data file's `DataSequenceNumber` and the delete files attached to it: the planner has already resolved this scoping using `data_sequence_number`.
Equality deletes carry a predicate (e.g. `id = 42`). The applicability rule is:
```text
{ edf :
edf.DataSequenceNumber > data_file.DataSequenceNumber // strict; sequence
AND edf.PartitionSpecId == data_file.PartitionSpecId // same spec
AND edf.PartitionValues == data_file.PartitionValues // same partition
}
```
The sequence rule is strict: data files whose `data_sequence_number` is equal to or greater than the delete file's are not affected — those rows simply never existed when the delete was written. (The Iceberg manifest entry's `file_sequence_number` is the commit-time sequence number for the manifest entry itself and is not used for delete scoping.) Partition matching is required as well: equality-delete files are written per partition, and a delete in one partition does not delete from another, even if the sequence rule matches. This is why the per-data-file pushdown request must carry both the data file's `DataSequenceNumber` / `PartitionSpecId` / `PartitionValues` and the delete files attached to it: the planner has already resolved this scoping.
Equality deletes cannot be precomputed as a static bitmap per data file because: