mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-31 21:31:24 +00:00
docs(parquet-design): expand DeleteFileRef for v2 + v3 metadata
The previous DeleteFileRef carried only Path, SizeBytes, SequenceNumber. That is not enough to distinguish v2 position-delete files from equality-delete files from v3 deletion vectors, to know which file format to read, to evaluate equality predicates (which field IDs?), or to locate a DV inside a Puffin file. Add: - Content discriminator (PositionDeletes / EqualityDeletes / DeletionVector) matching the Iceberg manifest "content" enum - FileFormat (Parquet / Avro / ORC / Puffin) - EqualityFieldIds for equality-delete predicate columns - BlobOffset / BlobLength / BlobDigest for Puffin DV blobs - ReferencedDataFile, when the delete targets a single data file Also collapse DataFileDescriptor.PositionDeletes + DataFileDescriptor.EqualityDeletes into a single Deletes list now that DeleteFileRef.Content discriminates among the three forms, including DVs which fit neither old slot.
This commit is contained in:
@@ -455,16 +455,63 @@ type DataFileDescriptor struct {
|
||||
RecordCount int64 // Iceberg manifest record_count
|
||||
ETag string // optional, used when no Iceberg manifest is available
|
||||
SequenceNumber int64 // Iceberg sequence number, drives equality-delete scope
|
||||
PositionDeletes []DeleteFileRef // delete files that apply to this data file
|
||||
EqualityDeletes []DeleteFileRef
|
||||
|
||||
// Deletes is the union of position-delete files, equality-delete
|
||||
// files, and deletion vectors that apply to this data file.
|
||||
// Content type is discriminated by DeleteFileRef.Content.
|
||||
Deletes []DeleteFileRef
|
||||
}
|
||||
|
||||
type DeleteFileRef struct {
|
||||
Path string
|
||||
SizeBytes int64
|
||||
SequenceNumber int64
|
||||
|
||||
// Content type matches Iceberg manifest "content" field.
|
||||
Content DeleteContent
|
||||
|
||||
// FileFormat is the on-disk format of this delete file.
|
||||
FileFormat FileFormat
|
||||
|
||||
// EqualityFieldIds carries the Iceberg field IDs the equality
|
||||
// predicate is keyed on. Required for Content == EqualityDeletes;
|
||||
// empty for position deletes and deletion vectors.
|
||||
EqualityFieldIds []int32
|
||||
|
||||
// Puffin-only fields, used when Content == DeletionVector and
|
||||
// FileFormat == FileFormatPuffin. The DV blob lives at
|
||||
// (Path, BlobOffset, BlobLength) and BlobDigest is the Puffin
|
||||
// blob's recorded content hash, used to key the bitmap cache.
|
||||
BlobOffset int64
|
||||
BlobLength int64
|
||||
BlobDigest []byte
|
||||
|
||||
// ReferencedDataFile is set when this delete file targets a single
|
||||
// data file (mandatory for v3 deletion vectors; optional for v2
|
||||
// position-delete files). Empty when the delete file may target
|
||||
// many data files.
|
||||
ReferencedDataFile string
|
||||
}
|
||||
|
||||
type DeleteContent int32
|
||||
|
||||
const (
|
||||
DeleteContentUnspecified DeleteContent = 0
|
||||
PositionDeletes DeleteContent = 1 // Iceberg manifest content=1
|
||||
EqualityDeletes DeleteContent = 2 // Iceberg manifest content=2
|
||||
DeletionVector DeleteContent = 3 // Iceberg v3 Puffin DV
|
||||
)
|
||||
|
||||
type FileFormat int32
|
||||
|
||||
const (
|
||||
FileFormatUnspecified FileFormat = 0
|
||||
FileFormatParquet FileFormat = 1
|
||||
FileFormatAvro FileFormat = 2
|
||||
FileFormatOrc FileFormat = 3
|
||||
FileFormatPuffin FileFormat = 4
|
||||
)
|
||||
|
||||
type PredicateKind int32
|
||||
|
||||
const (
|
||||
@@ -494,7 +541,7 @@ The client is responsible for Iceberg planning (resolving the snapshot to data f
|
||||
|
||||
- enough identity (`SizeBytes`, `RecordCount`, optional `ETag`) for the server to verify a cached side index still matches the file,
|
||||
- the Iceberg `SequenceNumber` so equality-delete scope can be resolved correctly,
|
||||
- the position and equality delete files attached to this data file by the client's planner.
|
||||
- the `Deletes` list of position-delete files, equality-delete files, and deletion vectors that the client's planner has attached to this data file. Each entry carries the Iceberg `Content` discriminator (`PositionDeletes`, `EqualityDeletes`, `DeletionVector`), the on-disk `FileFormat`, and any content-specific fields (equality field IDs, Puffin blob offset/length/digest).
|
||||
|
||||
v1 implementations should accept Substrait as the canonical wire format. Iceberg Expression JSON is supported as a convenience for connectors that already produce it.
|
||||
|
||||
@@ -647,7 +694,7 @@ Equivalent shorthand for v3 DVs, which are intrinsically per-data-file: `(data_f
|
||||
|
||||
`snapshot_id` is *not* a sufficient key — different snapshots can produce the same bitmap, and pinning the cache to snapshot id wastes work after no-op snapshots. Conversely, snapshot_id alone is *insufficient* across tables because position-delete files are not uniquely identified by snapshot number.
|
||||
|
||||
When pushdown receives a request, it computes the key from the request's `PositionDeletes` list (already supplied by the client's planner) and looks up or builds the bitmap. Cache eviction follows file identity: when a data file or any of its delete files goes out of scope (snapshot expiration, compaction), the corresponding cache entries are reclaimable.
|
||||
When pushdown receives a request, it computes the key from the position-delete entries of the data file's `Deletes` list (already supplied by the client's planner) and looks up or builds the bitmap. Cache eviction follows file identity: when a data file or any of its delete files goes out of scope (snapshot expiration, compaction), the corresponding cache entries are reclaimable.
|
||||
|
||||
### Equality deletes (must evaluate at query time)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user