mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-19 22:44:18 +00:00
docs(parquet-design): drop DeletionVector content value
Iceberg's manifest "content" enum has only two delete values: POSITION_DELETES (1) and EQUALITY_DELETES (2). Iceberg v3 deletion vectors are POSITION_DELETES stored in a Puffin file, not a third content value. The previous DeleteContent enum invented a DeletionVector = 3 that does not exist in the spec, which would have forced a translation step on the wire and confused any reader cross- checking against an Iceberg manifest. Drop DeletionVector from DeleteContent. A DV is now expressed as (Content == PositionDeletes, FileFormat == FileFormatPuffin), which matches what an Iceberg manifest actually carries. Update the DeleteFileRef field comments and the prose that listed the old three-way discriminator.
This commit is contained in:
@@ -483,7 +483,9 @@ type DeleteFileRef struct {
|
||||
// for equality deletes (predicate-based; only filters older data).
|
||||
DataSequenceNumber int64
|
||||
|
||||
// Content type matches Iceberg manifest "content" field.
|
||||
// Content matches Iceberg manifest "content" field. DVs are
|
||||
// POSITION_DELETES with FileFormat == FileFormatPuffin; there is
|
||||
// no separate "deletion vector" content value in Iceberg.
|
||||
Content DeleteContent
|
||||
|
||||
// FileFormat is the on-disk format of this delete file.
|
||||
@@ -491,13 +493,12 @@ type DeleteFileRef struct {
|
||||
|
||||
// EqualityFieldIds carries the Iceberg field IDs the equality
|
||||
// predicate is keyed on. Required for Content == EqualityDeletes;
|
||||
// empty for position deletes and deletion vectors.
|
||||
// empty for position deletes (file or DV).
|
||||
EqualityFieldIds []int32
|
||||
|
||||
// Puffin-only fields, used when Content == DeletionVector and
|
||||
// FileFormat == FileFormatPuffin. The DV blob lives at
|
||||
// (Path, BlobOffset, BlobLength) and BlobDigest is the Puffin
|
||||
// blob's recorded content hash, used to key the bitmap cache.
|
||||
// Puffin-only fields, used when Content == PositionDeletes and
|
||||
// FileFormat == FileFormatPuffin (deletion vectors). The DV blob
|
||||
// lives at (Path, BlobOffset, BlobLength).
|
||||
BlobOffset int64
|
||||
BlobLength int64
|
||||
BlobDigest []byte
|
||||
@@ -513,10 +514,11 @@ type DeleteContent int32
|
||||
|
||||
const (
|
||||
DeleteContentUnspecified DeleteContent = 0
|
||||
PositionDeletes DeleteContent = 1 // Iceberg manifest content=1
|
||||
PositionDeletes DeleteContent = 1 // Iceberg manifest content=1 (file or DV)
|
||||
EqualityDeletes DeleteContent = 2 // Iceberg manifest content=2
|
||||
DeletionVector DeleteContent = 3 // Iceberg v3 Puffin DV
|
||||
)
|
||||
// Iceberg v3 deletion vectors are POSITION_DELETES with
|
||||
// FileFormat == FileFormatPuffin; there is no separate enum value.
|
||||
|
||||
type FileFormat int32
|
||||
|
||||
@@ -570,7 +572,7 @@ The client is responsible for Iceberg planning (resolving the snapshot to data f
|
||||
|
||||
- enough identity (`SizeBytes`, `RecordCount`, optional `ETag`) for the server to verify a cached side index still matches the file,
|
||||
- the Iceberg `DataSequenceNumber` (the manifest entry's `data_sequence_number`, not `file_sequence_number`) so equality-delete and DV scope can be resolved correctly,
|
||||
- the `Deletes` list of position-delete files, equality-delete files, and deletion vectors that the client's planner has attached to this data file. Each entry carries the Iceberg `Content` discriminator (`PositionDeletes`, `EqualityDeletes`, `DeletionVector`), the on-disk `FileFormat`, and any content-specific fields (equality field IDs, Puffin blob offset/length/digest).
|
||||
- the `Deletes` list of position-delete files, equality-delete files, and deletion vectors that the client's planner has attached to this data file. Each entry carries the Iceberg `Content` discriminator (`PositionDeletes` or `EqualityDeletes`) plus the on-disk `FileFormat`, with the pair `(PositionDeletes, FileFormatPuffin)` denoting a v3 deletion vector. Content-specific fields cover equality field IDs and Puffin blob offset/length.
|
||||
|
||||
v1 implementations should accept Substrait as the canonical wire format. Iceberg Expression JSON is supported as a convenience for connectors that already produce it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user