mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-25 09:24:21 +00:00
docs(parquet-design): plan residency switches to catalog + volume tiers
Decision #1 reverses again: pushdown is *not* a standalone weed pushdown daemon. Planning lives inside the existing catalog-server role (today colocated with the S3 gateway's Iceberg REST handlers in weed/s3api/iceberg/), so the planner can reuse the table / snapshot / file-list state the catalog already manages. Heavy execution lives on the volume servers next to the data, so one busy table cannot bottleneck cluster-wide planning. Update both the architectural-decisions bullet and the Decisions Recorded entry. Note that an optional query-worker tier remains available as the design's Phase 4 contingent on measurements. Code-layout, milestone, and existing-M0/M1-code revisions land in follow-up commits.
This commit is contained in:
@@ -30,7 +30,7 @@ The dependencies needed for v1 (`parquet-go`, `iceberg-go`, `RoaringBitmap/roari
|
||||
|
||||
These are commitments unless an open question below changes them.
|
||||
|
||||
- **Residency: separate `weed pushdown` daemon.** The pushdown service runs as its own process, scaled and operated independently of the filer. It talks to the filer over the existing filer gRPC API to read Parquet data and side-index blobs, and to the Iceberg catalog (via the existing internal package — see decision 6 below) for snapshot resolution and request validation. Rationale: pushdown availability and resource consumption (cache memory, predicate evaluation CPU, per-tenant index loads) decouple from the filer's hot path, and the daemon can scale horizontally without touching the storage layer. The daemon is stateless on disk: caches are in-memory; durable side indexes live in the filer per [Side-index path](#side-index-path).
|
||||
- **Residency: catalog server (planning) + volume servers (heavy execution).** Pushdown is *not* a separate daemon. Planning (snapshot resolution, file-level pruning, side-index registry, footer cache, request validation) runs inside the existing catalog-server role — today colocated with the S3 gateway's Iceberg REST handlers in `weed/s3api/iceberg/`. Heavy execution (local index lookup, scalar predicate evaluation against indexes, vector distance + local top-K) runs on the volume server next to the data. The S3 gateway remains the compatibility path for unmodified readers. Rationale: the catalog already knows tables, snapshots, and table-to-files mapping, which is exactly what the planner needs — adding a separate daemon would duplicate that state. Heavy work goes to the volume servers so a single hot table cannot bottleneck cluster-wide planning. A standalone query-worker tier remains an option (design's Phase 4) only if measurements demand it.
|
||||
- **Wire protocol: gRPC.** New service definition `parquet_pushdown_pb.ParquetPushdown` in `weed/pb/parquet_pushdown_pb/`. REST/HTTP shim only if a connector requires it. JSON variant deferred.
|
||||
- **Trust mode for v1: catalog-validated, with connector-trusted as a developer-only flag.** The default deployment validates every request's `DataFiles` and `Deletes` against the Iceberg catalog at the requested snapshot before serving (full rules in [Trust Model](./PARQUET_PUSHDOWN_DESIGN.md#trust-model-and-catalog-validation)). Connector-trusted is supported but only enabled by an explicit flag for local development and benchmarks; it is not the default and is rejected in production builds. Manifest-signed is not on the roadmap.
|
||||
- **Predicate engine for v1: built-in subset evaluator.** The wire format accepts Substrait `ExtendedExpression`, but v1 implements only: comparisons (`=, !=, <, <=, >, >=`), `IN`, `BETWEEN`, `IS NULL`, and boolean `AND/OR/NOT` over them. Anything outside this subset returns `unsupported predicate`, and the connector falls back to a standard scan. Full Substrait evaluation lands in Phase 2.
|
||||
@@ -175,7 +175,7 @@ Lands before M2 ships externally. Sequenced after M1 because catalog validation
|
||||
|
||||
These were the open questions before M0. All have been resolved; recording for traceability and to anchor future amendments.
|
||||
|
||||
1. **Residency:** standalone `weed pushdown` daemon. (See [Architectural decisions](#architectural-decisions-for-v1).)
|
||||
1. **Residency:** catalog server (planning, registry, footer cache) + volume servers (local index lookup, predicate eval, vector compute). Standalone daemons are *not* introduced; query workers as an extra tier remain a future option contingent on measurements. (See [Architectural decisions](#architectural-decisions-for-v1).)
|
||||
2. **Service surface:** new `weed/pb/parquet_pushdown_pb/` package; not extending `filer_pb`.
|
||||
3. **Trust mode:** catalog-validated is the default and required for Phase 1 shipping; connector-trusted is a dev-only flag rejected in production builds.
|
||||
4. **Side-index path:** co-located in a hidden `.index/` directory next to the Parquet data, under the same parent folder. File-scoped: `<parquet_parent>/.index/<file_name>/<identity>/...`. Folder-scoped: `<parquet_parent>/.index/<identity>/...`. Hidden-prefix name configurable for non-dot-aware readers.
|
||||
|
||||
Reference in New Issue
Block a user