mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-28 20:06:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(parquet-design): document trust model and catalog validation modes

Browse Source 
The previous text said the server "treats this list as authoritative
and does not re-read the catalog" without flagging that this is only
safe when the caller is trusted. An untrusted client could:

- omit position/equality delete files to make deleted rows visible
- claim a snapshot id that does not match the data files listed
- point at data files outside the table's scope

Add a Trust Model section enumerating three deployment modes:
connector-trusted (v1 default; rely on a trust boundary around the
data plane), catalog-validated (server checks DataFiles + Deletes
against the manifest), and manifest-signed (future; requires catalog
support that doesn't exist today). Note the always-on guards (object
ACL, request-shape limits) and require the chosen mode to be
reported in PushdownStats so callers can verify which level handled
their request.
This commit is contained in:
Chris Lu
2026-04-25 01:56:07 -07:00
parent c9c487282b
commit 89ed588b34
1 changed files with 18 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PARQUET_PUSHDOWN_DESIGN.md
+18 -1
View File
@@ -596,7 +596,7 @@ type ColumnRef struct {
}
```
The client is responsible for Iceberg planning (resolving the snapshot to data files and delete files) and passes the resolved set in `DataFiles`. The server treats this list as authoritative and does not re-read the catalog. Each descriptor carries:
The client is responsible for Iceberg planning (resolving the snapshot to data files and delete files) and passes the resolved set in `DataFiles`. The server treats this list as authoritative and does not re-read the catalog. See [Trust Model](#trust-model-and-catalog-validation) below for the security implications and the validation modes a deployment can choose between. Each descriptor carries:
- enough identity (`SizeBytes`, `RecordCount`, optional `ETag`) for the server to verify a cached side index still matches the file,
- the Iceberg `DataSequenceNumber` (the manifest entry's `data_sequence_number`, not `file_sequence_number`) so equality-delete and DV scope can be resolved correctly,
@@ -682,6 +682,23 @@ Spark/Trino/DuckDB connector
-> read only needed Parquet ranges
```
## Trust Model and Catalog Validation
The pushdown server treats the request's `DataFiles` and `Deletes` as authoritative — it does not re-resolve the snapshot against the Iceberg catalog. That is fine when the caller is a trusted query engine that has just done its own planning, but it is unsafe when the caller is untrusted: a hostile client could omit delete files (returning rows that should be invisible), point at data files outside the table, or claim a snapshot id that does not match the request.
A deployment must pick one of the following modes per endpoint, and the choice should be visible in the pushdown response stats:
- **Connector-trusted (default for v1).** The pushdown API is reachable only by trusted connectors (Spark/Trino/DuckDB inside the same trust boundary as the catalog). The connector is responsible for honest planning; the server does not validate. Use mTLS, a signed service token, or a network boundary to enforce the trust assumption. Fastest path; appropriate for an internal data plane.
- **Catalog-validated.** The server validates the request against the Iceberg catalog before serving. For each `DataFileDescriptor` it confirms `(Path, SizeBytes, RecordCount, DataSequenceNumber, PartitionSpecId, PartitionValues)` against the manifest at `SnapshotId`, and confirms the `Deletes` list matches the manifest's attached delete files. Slower (one catalog read per request, cacheable per snapshot) but safe to expose to untrusted clients.
- **Manifest-signed (future).** The connector includes a catalog-signed token containing a hash of the resolved planning output; the server verifies the signature without re-reading the catalog. Lowest validation cost on the hot path; requires catalog support that does not exist today, so it is listed as a future option only.
Independent of mode, the server always enforces:
- the underlying object's ACL — a client that cannot read the Parquet file cannot query its indexes (see [Security and Access Control](#security-and-access-control));
- request-shape limits (`MaxRowIds`, predicate complexity caps, request size cap) so a malformed request cannot consume unbounded server resources.
The chosen mode is reported in `PushdownStats` so the connector and the human operator can verify which trust level actually serviced a request.
## Index Consistency
Indexes must be tied to a stable file identity. Preferred identity, in order of strength: