diff --git a/design-lance-catalog.md b/design-lance-catalog.md new file mode 100644 index 000000000..ce657b98a --- /dev/null +++ b/design-lance-catalog.md @@ -0,0 +1,696 @@ +# Lance Catalog for SeaweedFS + +A second catalog surface next to the Iceberg REST catalog, speaking the Lance Namespace +REST spec, over the same table buckets and the same filer. + +## Why + +Gravitino 1.1 added a Lance REST service and 1.3 ships it as a standalone server; Lakekeeper +added Lance in the same window by a completely different route. That is the useful signal: +two unrelated catalogs decided independently that Lance had to be first-class, not a niche. +The client side is already there — `lance-spark` (`LanceNamespaceSparkCatalog` +with `impl=rest`), `lance-ray`, and the generated Python/Java/Rust clients all talk the same +OpenAPI. Implementing the spec means those engines work against SeaweedFS with no +SeaweedFS-specific code on the client. + +The second reason is that Gravitino's own documentation names the gap it cannot close: +DuckDB, pandas and DataFusion "do not support Lance REST natively yet" and have to fetch a +location from the catalog and then open the dataset directly. Gravitino cannot help there, +because it does not own the storage. SeaweedFS does. That is the whole design opportunity +below. + +## Prior art: three families + +Upstream lists twelve catalog implementations, and they fall into three shapes. Knowing +which one we are building matters more than any individual API decision. + +**1. Storage-native, no service.** The Lance Directory Catalog. V1 is a directory listing +where every `.lance/` child of a prefix is a table; V2 adds a `__manifest` table — +itself a Lance table — holding `object_id`/`object_type`/`location` rows, with nested +namespaces, hash-prefixed table directories, and optional managed versioning. No server, no +credentials, no governance. This is the floor every other implementation has to beat. + +**2. Protocol-native server.** Someone implements the Lance Namespace REST OpenAPI and +clients connect with `impl=rest`. Gravitino is the only one of the twelve that does this, +and it is what this design proposes. + +**3. Client-side adapters onto an existing catalog.** Nine of the twelve. The Lance client +translates namespace operations into whatever the backing catalog already speaks: Apache +Polaris, Unity Catalog, AWS Glue, Hive Metastore v2 and v3, Google BigLake, Dataproc, +Microsoft OneLake — and Apache Iceberg REST. Two flavors: + +- Catalogs with a real non-Iceberg table concept mark the format directly. Polaris uses its + Generic Table API with `format = lance`; Unity uses an `EXTERNAL` table with + `table_type=lance` in properties and the path in `storage_location`; Glue uses + `EXTERNAL_TABLE` plus `table_type=lance` in `Parameters`, path in + `StorageDescriptor.Location`. +- Catalogs with no such concept fake one. The Iceberg REST adapter registers **a regular + Iceberg table with a dummy schema — a single nullable string column named `dummy`** — + carrying the property `table_type=lance`, and treats the Iceberg table location as the + Lance dataset root. + +Every adapter in family 3 lands in the same place: `DeclareTable`/`ListTables`/ +`DescribeTable`/`DeregisterTable` only, `DropNamespace` in RESTRICT mode only, +`load_detailed_metadata=false` only, and `managed_versioning=false`. They are a name-to- +location map and nothing more. + +Lakekeeper is the instructive outlier. It has the same generic-table concept Polaris has, +but no upstream adapter exists for it — there is no `lance-namespace` reference anywhere in +its repository and no page for it in the supported-catalogs list. So Polaris's generic tables +are reachable from a stock Lance client and Lakekeeper's are not, despite being the same +idea. Shipping the concept is not the same as shipping the integration. + +## Gravitino and Lakekeeper: the two opposite bets + +Both shipped Lance support in the same window and did not build the same thing. + +**Gravitino implements the protocol.** Its `lance/` module serves the Lance Namespace REST +spec on its own port (`:9101/lance`), so stock `lance-spark` and `lance-ray` connect with +`impl=rest` and no vendor-specific client. The cost is governance: storage credentials are +static properties on the catalog (`lance.storage.access_key_id`, `secret_access_key`, +`endpoint`, `region`, `allow_http`), optionally overridden per table, handed to the engine +as-is. No STS, no expiry, no per-table scoping. + +**Lakekeeper refuses the protocol and governs the object instead.** There is no +`lance-namespace` anywhere in the repository; Lance arrived in 0.13.0 (2026-06-30, issue +#1673 `Generic Table API with Lance`) as one `format` string on a Lakekeeper-native Generic +Table API: + +``` +POST/GET/DELETE /lakekeeper/v1/{prefix}/namespaces/{ns}/generic-tables[/{table}] +GET /lakekeeper/v1/{prefix}/namespaces/{ns}/generic-tables/{table}/credentials +POST /lakekeeper/v1/{prefix}/generic-tables/rename +``` + +`format` is opaque, `schema` and `statistics` are stored but never validated, and the +catalog writes no format-specific metadata — engines go straight to the location. In +exchange Lance tables get everything Iceberg tables get: STS-vended prefix-scoped +credentials, OpenFGA per-action permissions (16 actions), soft-delete with undrop, a +protection flag, rename, pagination, and name uniqueness across Iceberg tables, views and +generic tables in one namespace. The price is that no stock Lance client can talk to it — +you need `pylakekeeper`, which exists mainly to translate vended credentials into +`lance_storage_options`. + +So: protocol fidelity and weak governance, or strong governance and client lock-in. Both +documented their limit honestly, and it is the same limit. Lakekeeper's capability table +says it outright — "Commit coordination: the catalog does not arbitrate writes — engines +write directly." Gravitino does not claim it either. Neither of them coordinates a Lance +commit, which is exactly the thing a store can do and a control plane cannot. + +We do not have to choose. Serve the Lance protocol natively the way Gravitino does, over +the `s3tables` entries that already carry ARNs, policies, tags and maintenance config, and +the governance comes from the layer underneath rather than from a proprietary API on top. +That is only available to us because we are the store, which is also what makes the third +option — arbitrating the commit — available. + +## We are probably already a Lance catalog, and that is a problem + +The Iceberg REST adapter does not care whose Iceberg catalog it is talking to. It needs +`/v1/config?warehouse=`, `/v1/{prefix}/namespaces`, `/v1/{prefix}/namespaces/{ns}/tables` +and unit-separator (`\x1F`) multi-level namespaces. We serve all of those, and +`parseNamespace` in `weed/s3api/iceberg/utils.go:22` already splits on `\x1F`. So a stock +Lance client pointed at our Iceberg catalog on :8181 with the Iceberg impl should already +create, list, describe and deregister Lance tables today, with no SeaweedFS change at all. + +That is worth testing before writing a line of the design above, for two reasons. It is a +free baseline — and possibly a free announcement. And it is a data-loss hazard. + +A Lance table registered this way is an Iceberg table whose metadata references no data +files, sitting on top of a Lance dataset that uses `data/` for its fragments — the same +subdirectory name Iceberg uses. The maintenance worker's orphan cleaner walks exactly +`/metadata` and `
/data`, and deletes every file not referenced by a snapshot +and older than `orphan_older_than_hours` +(`weed/worker/tasks/iceberg/operations.go:331`, default 72). Against an adapter-registered +Lance table, every fragment is unreferenced by construction. Run maintenance and the +dataset is deleted. + +Maintenance is disabled by default (`handler.go:334`), so this is a latent hazard rather +than a live one: it needs an operator to enable Iceberg maintenance on a bucket that also +holds adapter-registered Lance tables. But it costs nothing to close — detection should +skip any table carrying a non-Iceberg format marker (`table_type` property, or +`Format != "ICEBERG"` once the format field is honest), and that guard is worth landing on +its own regardless of whether the rest of this design ever gets built. It is the same +"catalog-only, no maintenance" marker the generic-format question needs. + +## Where we differ from Gravitino + +Gravitino is a metadata service in front of somebody else's object store: + +``` + Spark / Ray Spark / Ray / pandas / duckdb + | | + Lance REST Lance REST (direct S3) + | | | + Gravitino SeaweedFS S3 gateway ----+ + | | + S3 keys handed out SeaweedFS filer + volumes + | + somebody else's S3 +``` + +It resolves a name to a location plus `lance.storage.*` credentials, and steps out of the +way. Everything a Lance table actually is — `_versions/`, `data/`, `_indices/` — is opaque +to it. + +We are the store. Three things follow that Gravitino cannot do: + +1. The catalog and a plain directory listing can be made to agree, so a client with no + catalog at all still sees the right tables. +2. `_versions/` is a filer directory listing, not an object-store `LIST`. Version history + is cheap and can back the admin UI. +3. We can offer a genuinely atomic commit reservation. Lance's commit protocol needs + put-if-not-exists; our S3 layer does not currently provide one (see + [Commit safety](#commit-safety)). The filer does. + +## Placement + +The Iceberg catalog is a thin HTTP shell over `s3tables.Manager`; the storage work lives in +`weed/s3api/s3tables`. Table buckets live under `TablesPath = s3_constants.DefaultBucketsPath`, +i.e. the same filer tree the S3 gateway serves, so `s3://bucket/ns/table/` is simultaneously +a catalog entry and an S3 prefix. Catalog entries are filer directories carrying `s3tables.*` +extended attributes. `Table.Format` already exists and is hard-checked against `"ICEBERG"` +in `weed/s3api/s3tables/handler_table.go:48`. + +So: + +``` +weed/s3api/lance/ new: HTTP surface, id codec, error model +weed/s3api/s3tables/ extended: Format "LANCE", lance state xattr, version entries +weed/command/s3.go new: -port.lance (default 9101), startLanceServer +``` + +`Format: "LANCE"` on the table entry is the whole storage-model change for phase 1. +Everything else — namespaces, ARNs, policies, tags, ownership — is shared verbatim. + +``` + s3tables.Manager (filer) + | + +------------------------+------------------------+ + | | + weed/s3api/iceberg weed/s3api/lance + Iceberg REST :8181 Lance REST :9101 + | | + Iceberg tables Lance datasets + \ / + +-------------------- s3 :8333 -----------------+ + | + SeaweedFS volumes +``` + +## Identifier mapping + +Lance identifiers are `["ns", ..., "table"]`, encoded in the URL as a single string joined +by a delimiter that defaults to `$`. The delimiter alone means the root namespace, so +`/v1/namespace/$/list` lists the root's children. + +Iceberg had to invent a warehouse selector because its identifier is flat and every table +bucket is a separate catalog. Lance does not need that — its identifier is already +hierarchical, and Gravitino uses exactly three levels (`["lance_catalog", "sales", "orders"]`). +That maps onto us without inventing anything: + +``` + $ root -> list of table buckets + $analytics level 1 -> a table bucket + $analytics$sales level 2 -> a namespace in that bucket + $analytics$sales$orders table +``` + +`spark.sql.catalog.lance.parent = analytics` then makes `sales.orders` resolve, which is the +same shape Gravitino's Spark example uses. + +Levels 2..N join into one `s3tables` namespace with `.`, matching what the Iceberg catalog +already does with `flattenNamespacePath`. The flattened form is only the directory name — +`namespaceMetadata.Namespace []string` in the xattr keeps the authoritative parts, so the +mapping stays invertible even though `.` is a legal character inside a namespace part. +Reject `$` in any name part with `InvalidInput`; our charsets already exclude it, so no +escaping scheme is needed. + +Root-level `ListNamespaces` returning table buckets means an unauthenticated or +broadly-scoped caller can enumerate buckets. Filter it through the same +`s3tables/permissions.go` check `ListTableBuckets` uses, not a separate path. + +`CreateNamespace` on a one-part identifier creates a table bucket, and it does so only if +the caller is permitted to — the namespace never creates a bucket as a side effect of +creating something inside it. A table bucket is a tenant resource with its own policy, ARN +and lifecycle, and conjuring one because a client said `CREATE SCHEMA` is a privilege +escalation dressed as a convenience. Lakekeeper draws the same line explicitly: its client +creates tables, not warehouses. + +## Storage layout + +Lay tables out as: + +``` +s3:////
/ + data/ + _versions/ + _indices/ +``` + +**Built without the `.lance` suffix this design originally proposed.** The suffix would have +made every namespace prefix a valid Lance Directory Catalog V1 root, since V1 recognises a +table by exactly that naming. It does not survive contact with the storage layer: the +catalog entry *is* the dataset directory, `validateTableName` excludes `.` from the charset, +and a suffixed entry name would leak into ARNs, policy documents and the S3 Tables API, +where the same table would answer to two different names. Making `GetTablePath` format-aware +instead spreads an "unless it is Lance" branch through code that has no business knowing — +the exact cross-cutting cost this design rejects family 3 for. + +So one name, one directory. What survives is direct access by URI, which is the larger half +of the story and needs no naming convention at all: + +```python +# with the catalog +spark.sql("SELECT * FROM lance.sales.orders") + +# without it, same bytes +lance.dataset("s3://analytics/sales/orders") +``` + +DuckDB, pandas and DataFusion still reach the data with no catalog running, which is the gap +Gravitino's documentation admits to. What they no longer get for free is *enumeration* — a +directory-catalog client pointed at the namespace prefix will not list these as tables. If +that turns out to matter, the cheapest fix is a repair-style tool that materialises `.lance` +aliases, not a rename of the catalog entry. + +Note also that the directory catalog's own V2 mode puts child-namespace tables in +`_` directories at the root and creates no physical subdirectories for +namespaces, so full directory-catalog fidelity was never on offer anyway. We are a +server-backed catalog; the human-readable prefix layout is worth more than partial V1 +lookalike behaviour. + +## The table bucket was not a neutral container + +This design assumed a table bucket is a place to put a table's files. It is +not: `validateTableBucketObjectPath` runs on every S3 write into one and +validated the path against Iceberg's layout, so a Lance client got 403 on +`data/*.lance`, on `_versions/`, and on `_transactions/` — a directory Lance +writes that neither the spec documentation nor this design anticipated. Nothing +about the catalog worked end to end until that changed. + +The layout guard now admits the union of what the supported formats write, and +treats any underscore-prefixed top-level directory as belonging to the format, +checking only that the path stays inside the table. Enumerating Lance's +internal directories by name is exactly the mistake that missed +`_transactions`. Iceberg writes none of them, so it loses nothing. + +Found by pointing the real Python client at a running gateway, not by reading +the spec. Worth remembering for the next format: the premise to check first is +whether the storage layer will accept its files at all. + +## Table lifecycle + +Lance has three table states, and the spec pins them to marker files: + +| State | Marker | Created by | Visible in ListTables | +| --- | --- | --- | --- | +| declared | `.lance-reserved` | `DeclareTable` | yes, when `include_declared=true` | +| created | `_versions/` present | client writes, or `CreateTable` | yes | +| deregistered | `.lance-deregistered` | `DeregisterTable` | no; data preserved | + +Record the state in an xattr (`s3tables.lanceState`) on the catalog entry *and* write the +marker file into the table directory. The xattr is what the catalog reads; the marker is +what keeps a directory-catalog client honest. Dual-write is the price of the interop claim +above, and it is one extra filer write on three rarely-called operations. + +`DeclareTable` is the operation `lance-spark` actually calls on `CREATE TABLE` (it replaced +the legacy `create-empty`), so it is not optional in practice even though the spec marks +only a subset as required. + +`DeregisterTable` preserving data is the same shape as our Iceberg rename, where the catalog +entry moves and the data stays put — reuse `TableDataDirFromMetadataLocation`'s idea rather +than re-deriving the data path from the catalog name. + +## Commit safety + +This is the part I got wrong, and the correction removed a feature rather than adding one. + +Lance commits a version by writing `_versions/{v}.manifest` with put-if-not-exists: exactly +one writer is supposed to win, and the loser rebases. In lance 10 that path is not optional +and needs nothing bolted on — `commit_handler_from_url` hands every `s3://` dataset a +`ConditionalPutCommitHandler`, which calls `put_opts` with `PutMode::Create`, which +object_store's S3 backend sends as `If-None-Match: *`. + +I originally read our gateway as evaluating that header check-then-act, and designed around +it. That was already out of date. `buildWriteCondition` +(`weed/s3api/s3api_object_routed_write.go`) reduces `If-None-Match: *` to a filer +`WriteCondition{IF_NOT_EXISTS}`, and `putToFiler` routes the create to the object's owner +filer, which evaluates the precondition under its per-path lock; when routing is not +available it falls back to the object write lock, which evaluates it under the lock too. +Either way it is atomic. Sixteen concurrent writers of the same fresh key get one 200 and +fifteen 412s, repeatedly. + +So the store already has the primitive Lance needs, cluster-wide, for every conditional-PUT +client and not just this one. + +### What that removed + +An earlier draft of this design offered the catalog as an **external manifest store**: +`managed_versioning: true` plus `CreateTableVersion` and friends, with the reserve step as a +filer `CreateEntry` with `o_excl`. It was implemented, tested, and shipped behind a default-off +flag — and it should not exist. + +- It solves a problem this store does not have. The spec offers that path for stores that + cannot order commits themselves. +- It moves a table's version history out of the dataset and into the catalog, so a reader + that does not go through this namespace no longer sees the whole picture. That is a real + cost paid for nothing. +- lance 10 cannot even use it past the first commit: `NamespaceManifestStore::put_if_not_exists` + answers "put_if_not_exists is not supported for namespace-backed stores", which is exactly + what a second `append` needs. + +The version operations now answer `Unsupported` alongside the other operations the catalog +does not serve, and `managed_versioning` is answered `false`. The property they were +protecting is covered instead by a test that races eight writers at the manifest key through +S3 and asserts one wins — testing the path Lance actually takes. + +## Credential vending + +Iceberg needed a header (`X-Iceberg-Access-Delegation: vended-credentials`) and a bespoke +response shape. Lance has it in the spec: `vend_credentials: true` on the request, +`storage_options` on the response, with `expires_at_millis` as the well-known expiry key. + +Reuse the existing vendor interface unchanged — `iceberg.CredentialVendor` / +`STSService.AssumeRoleForPrincipal` scoped to the table prefix (#10777) — and map its output +to the storage options Lance passes through to `object_store`: + +``` +aws_access_key_id, aws_secret_access_key, aws_session_token, +aws_region, aws_endpoint, allow_http, expires_at_millis +``` + +Those are the names `pylakekeeper` emits as `lance_storage_options`, which is the shape +Lakekeeper's tested S3 path actually feeds to Lance. `object_store` also accepts the +un-prefixed aliases (`endpoint`, `region`) that the directory catalog's `storage.` prefix +strips down to and that Gravitino's `lance.storage.endpoint` resolves to, but the `aws_` +forms are the ones with a tested integration behind them, so emit those. `aws_endpoint` +should come from `deriveS3AdvertisedEndpoint()`, the same source the Iceberg `FileIO` config +uses, and `allow_http` must be set when that endpoint is plain HTTP or every read fails with +a TLS error that looks like a credential problem — Lakekeeper vends both automatically for +exactly this reason, and calls out that there is then no per-vendor branch in client code. + +We emit this server-side, in the `storage_options` field the Lance spec already defines, +which is strictly better than Lakekeeper's arrangement: no client library has to translate +anything, so vending works from any stock Lance client rather than only from theirs. + +Guard the same way #10777 had to after review: bucket-scoped list grants need an `s3:prefix` +condition, and a location containing `*` or `?` must be refused rather than widened into a +resource pattern. + +## Auth and authorization + +Authentication reuses `S3Authenticator` and `CredentialValidator` as-is. The Lance spec maps +identity to headers — `api_key` to `x-api-key`, `auth_token` to `Authorization: Bearer` — and +SigV4 keeps working because it is the same authenticator the Iceberg catalog already fronts. + +Authorization needs nothing new. A Lance table gets the same ARN shape, +`arn:aws:s3tables:...:bucket/B/table/NS/T`, so every existing table-bucket policy covers +Lance tables with no new policy language and no second permission model. Route it through +`s3tables/permissions.go` and inherit the `DefaultAllow` semantics the Iceberg server already +mirrors from the S3 port. + +One spec quirk worth honoring: request context entries prefixed `header.` become request +headers, and every response header comes back as a `header.`-prefixed context entry. Echoing +`x-request-id` through it costs nothing and makes tracing work. + +## What to take from Lakekeeper + +Rejecting Lakekeeper's API shape does not mean rejecting what it learned building it. + +**Deregister is soft-delete, so implement it as one.** Lakekeeper gives generic tables +soft-deletion with undrop and a `protected` flag that makes a drop require `force=true`. +Lance already has the concept — `DeregisterTable` preserves the data and hides the table — +so the `.lance-deregistered` marker is a soft-delete by another name, and a re-register is +an undrop. A protection flag on table-bucket entries is worth having regardless of Lance: +it is a few lines against the existing xattrs and it applies to Iceberg tables too. + +**Enforce one identifier space across entry kinds.** Lakekeeper rejects a generic table +whose name collides with an Iceberg table or view in the same namespace. Our catalog entries +already share one filer directory and already carry `s3tables.entryType`, so this is +structurally true — but it has to be enforced deliberately on every path, or a Lance handler +happily loads an Iceberg table's directory and vice versa. That is the same crossover bug +class as the view/table rename authorization fixed in #10776; the `catalogEntryKind` pattern +from that change is the thing to reuse rather than re-derive. + +**A re-vend path matters more than it looks.** Lakekeeper exposes `/credentials` separately +from load, because STS credentials expire in the middle of long jobs and re-loading the +whole table to refresh them is wasteful. In Lance the spec's answer is another +`DescribeTable` with `vend_credentials: true`, which is fine — but it means `DescribeTable` +must stay cheap when `load_detailed_metadata` is false, which is another reason not to open +the dataset on that path. + +**Generic tables are a cheap orthogonal win.** Lakekeeper's real insight is that Delta, +Parquet, CSV, Vortex and Paimon all get governance for free once the catalog stops caring +what the format is. Our `Table.Format` field already exists and the only thing stopping it +is the hard `"ICEBERG"` check in `handler_table.go:48`. Loosening that and letting the S3 +Tables API register a table with an arbitrary format and a location — no metadata, no +commits — is a small change that makes every format cataloguable. It is independent of this +design and probably worth doing first, since `Format: "LANCE"` is then just a value rather +than a special case. + +**Skip remote signing.** It is Lakekeeper's fallback for S3-compatible stores with no STS, +and their own documentation notes that Lance will not use it — format libraries with their +own S3 client expect static credentials and do not implement the Iceberg signer protocol. We +have STS, so vended credentials are the path, and the signer is not worth building for a +client that cannot consume it. + +## Errors + +Lance uses `{code, error, detail, instance}` with numeric codes, not Iceberg's exception-type +strings. The mapping is mechanical: + +| HTTP | code | when | +| --- | --- | --- | +| 400 | 13 InvalidInput | charset violations, malformed id, route/body mismatch | +| 401 | 16 Unauthenticated | | +| 403 | 15 PermissionDenied | | +| 404 | 1 NamespaceNotFound, 4 TableNotFound, 11 TableVersionNotFound | | +| 409 | 2/5 AlreadyExists, 3 NamespaceNotEmpty, 14 ConcurrentModification | | +| 501 | 0 Unsupported | every phase-3 data operation | + +Route/body mismatch is a spec requirement, not a nicety: when the identifier appears in both +the path and the body and they disagree, the server must return 400. Cheap to get right at +the decode step, annoying to retrofit. + +## Route surface + +Phase 0 is not in this table: point a stock Lance client at the existing Iceberg catalog +with the Iceberg impl, see how far it gets, and land the maintenance guard either way. That +tells us what the native server actually has to beat. + +Phase 1, the whole `lance-spark` and `lance-ray` contract: + +``` +POST /v1/namespace/{id}/create CreateNamespace mode: Create|ExistOk|Overwrite +GET /v1/namespace/{id}/list ListNamespaces +POST /v1/namespace/{id}/describe DescribeNamespace +POST /v1/namespace/{id}/drop DropNamespace mode: Fail|Skip, behavior: Restrict|Cascade +POST /v1/namespace/{id}/exists NamespaceExists +GET /v1/namespace/{id}/table/list ListTables ?include_declared, ?page_token, ?limit +GET /v1/table ListAllTables +POST /v1/table/{id}/declare DeclareTable +POST /v1/table/{id}/describe DescribeTable ?with_table_uri, ?load_detailed_metadata, ?check_declared +POST /v1/table/{id}/exists TableExists +POST /v1/table/{id}/register RegisterTable mode: Create|Overwrite +POST /v1/table/{id}/deregister DeregisterTable +POST /v1/table/{id}/drop DropTable +POST /v1/table/{id}/rename RenameTable +``` + +`DescribeTable` with `load_detailed_metadata=false` needs only `location`, which is the +common case and which we can answer from xattrs alone. With `load_detailed_metadata=true` +the spec wants `version`, `schema` and `stats`, which means reading the Lance manifest. For +phase 1, return the fields we can derive from the filer — `version` from the highest entry in +`_versions/`, given V2 naming is `{u64::MAX - version:020}.manifest` and V1 is +`{version}.manifest` — and omit `schema`/`stats` rather than fabricating them. The spec +tolerates a partial response here; it does not tolerate a wrong one. + +Phase 2 was the five version operations plus `managed_versioning`; it was built and then +removed, for the reasons under Commit safety. + +Phase 3 is the data plane: `CreateTable`, `InsertIntoTable`, `MergeInsertIntoTable`, +`UpdateTable`, `DeleteFromTable`, `QueryTable`, `CountTableRows`, and the index and tag +operations. These exchange Arrow IPC, and more to the point they require reading and writing +the Lance file format, for which no Go implementation exists. Return `Unsupported` (code 0) +and say so in the docs. `arrow-go/v18` is already an indirect dependency, so Arrow framing is +not the blocker — Lance is. + +## Does a Lance table need maintenance? + +Yes, and one part of it has no Iceberg equivalent. The client exposes three jobs: + +- `optimize.compact_files()` — Lance writes a fragment per write batch, so a table fed by + small appends accumulates small files exactly the way an Iceberg table does. +- `optimize.optimize_indices()` — **rows written after an index was built are not covered by + it.** A vector search against a stale index silently misses recent data. That is a + correctness-shaped failure, not a slow query, and it is specific to what people use Lance + for. +- `cleanup_old_versions()` — every version is retained until something removes it. Lance can + do this itself: `optimize.enable_auto_cleanup()` sets it on the dataset, so this one need + not be an external job at all. + +None of it can run in the Go worker. All three read and rewrite Lance files, which needs +Lance format code that does not exist in Go, and there is no useful subset either: deciding +which fragments an old version still references means parsing Lance manifests. + +So the maintenance worker must not touch a Lance table, and it declines by reading the format +the catalog recorded rather than by failing to parse Iceberg metadata. + +## The worker can be Rust, and it is not a sidecar + +The Go worker is not the only worker. `weed/pb/plugin.proto` defines `PluginControlService`, +a language-agnostic gRPC stream that external maintenance workers connect on: the worker +opens `WorkerStream`, sends `WorkerHello` with the job types it can `detect` and `execute`, +answers `RequestConfigSchema` with a `JobTypeDescriptor`, replies to `RunDetectionRequest` +with `JobProposal`s and to `ExecuteJobRequest` with `JobProgressUpdate`s and `JobCompleted`. +`weed worker -admin=host:23646` is the Go reference implementation of exactly that contract, +from outside the admin process. + +Nothing in it is Go-specific, and the Rust toolchain is already in the tree. +`seaweed-volume/build.rs` compiles protos straight out of `../weed/pb/` with `tonic_build`, +including `filer.proto`, on tonic 0.12 and prost 0.13. A Lance worker is that same build +with `plugin.proto` added and the `lance` crate as a dependency — the real one, no FFI and +no Python. + +Three job types, one per real maintenance operation: + +| Job type | Calls | Detected from | +| --- | --- | --- | +| `lance_compact` | `optimize.compact_files` | fragment count and sizes | +| `lance_optimize_indices` | `optimize.optimize_indices` | rows an index does not cover | +| `lance_cleanup_versions` | `cleanup_old_versions` | version count and age | + +What the existing machinery then supplies for free is the part worth noticing. Scheduling, +retries, dedupe by `dedupe_key`, progress reporting, per-job concurrency limits and the +admin settings page all come from the protocol: a worker that answers `RequestConfigSchema` +with a descriptor gets its configuration form rendered in the admin UI without a line of Go +or templ. A Rust worker is a first-class maintenance worker, not an appendage. + +The remaining wiring is small and mostly decided already. `RunDetectionRequest` carries a +`ClusterContext` with filer and S3 addresses plus a free-form `metadata` map, which is where +the Lance namespace URL goes; the worker lists Lance tables from the namespace, which is the +catalog of record and already filters by format. It gets at the data by asking +`DescribeTable` for `storage_options` with `vend_credentials`, so the worker is just another +client of the STS path rather than a component with its own credentials. And when it commits +a compaction it goes through `CreateTableVersion` like any other writer, which is what +managed versioning was for. + +## The worker is also the only thing that can describe the table + +Admin can render an Iceberg table because it can read Iceberg metadata. It cannot read +Lance: it knows the dataset's location and its format string, and that is the whole of it. +The details page showed a location and two empty panels, which is an honest answer and a +useless one. + +The worker already knows. Detection opens every dataset to decide whether it needs +compacting, so at that moment it holds the schema, the row count, the fragment count and +the version count. It just had no way to say so — every message on the stream was about +work. + +So `WorkerObservations` is a body on `WorkerToAdminMessage`: a repeated `ObjectObservation` +of `object_id`, `object_kind`, `format`, and a `ConfigValue` map the worker fills with +whatever it can cheaply say. Admin keeps the last observation per object and serves it back +with the time it was taken and the worker that took it. Nothing schedules from it, and it is +not authoritative — it is a cache with its staleness on the label, which is why the page +badges it rather than presenting it as metadata it read itself. + +The keys are the worker's to choose, which keeps the protocol out of the business of knowing +what a Lance table is. A worker for any other format admin cannot parse describes itself the +same way. + +## A bucket declares its format + +Format was recorded per table, which is enough for the storage layer and not enough for +anything that has to answer a question about a bucket. The admin UI printed one Iceberg +endpoint for every bucket, including the ones holding Lance datasets, where that endpoint +serves nothing; an empty bucket had no format at all. + +So `CreateTableBucket` takes an optional `format`, stored with the rest of the bucket +metadata. Empty means `ICEBERG` - what AWS S3 Tables serves, and therefore what an SDK +that has never heard of the field means. `CreateTable` refuses another format, and +`CreateView` refuses outright outside an Iceberg bucket, a view being Iceberg metadata. +The Lance namespace declares `LANCE` for the buckets it creates. + +**Enforced rather than defaulted**, because the point of showing a format at all is the +endpoint that follows from it, and that endpoint is only truthful if the bucket holds one +format. **Buckets that already exist stay undeclared** and keep taking anything: nothing is +migrated, and the UI shows "unset" as a fact about the bucket's age rather than a fault. +That state is also the only way to hold both formats at once, which is what the +Iceberg-REST adapter path produces. + +## Sample rows are fetched, not cached + +The same asymmetry has a second half. Admin renders an Iceberg table's rows by +reading its Parquet files directly; for Lance it has nothing to read with, so the data +page offered a Browse Data button that led to an empty grid. + +`RequestObjectPreview` / `ObjectPreviewResponse` mirror the config-schema round trip +already on the stream: admin asks, the worker scans the dataset and hands back rows it +has already rendered as text, because it is the only side that knows the types. Admin +picks the worker from the observation store, so the one that last described a table is +the one asked to read it. + +The rows are deliberately not cached, and that is the line between the two channels. An +observation describes an object, so a copy with a timestamp on it is useful. Rows are the +object's contents: a copy held in admin would be stale, larger, and nobody's business. +The page fetches on load, bounded, or says why it cannot. + +## The sidecar question + +The data plane is a different problem, and this design previously conflated the two. +Maintenance rides the worker protocol; `QueryTable` and `InsertIntoTable` do not, because +they are synchronous REST operations on the namespace's own surface. Serving those means a +Rust process that answers HTTP, either behind the Go namespace as a proxy target or in front +of it. It would make SeaweedFS a store you can run vector search *in* rather than one you +read vectors *out of*, which is the larger prize and the reason to keep the option open. + +Neither should gate phase 1. Phases 1 and 2 are pure Go over the filer and are worth +shipping on their own — they are what makes Spark and Ray work. + +## Testing + +Mirror the Iceberg package: `httptest` plus a fake filer client for the handler tests, in +`weed/s3api/lance`. Then an integration suite under `test/s3tables/catalog/` next to the +existing `pyiceberg_test.go`, driving the generated Python `lance-namespace` client against +a live gateway. Three things that suite must cover and unit tests cannot: + +- the storage-options key names actually work, i.e. a client that gets `storage_options` from + `DescribeTable` can open the dataset; +- a table created through the catalog is visible to `lance.dataset()` by URI and to a V1 + directory-catalog client rooted at the namespace prefix; +- concurrent writers do not lose a commit, which is the phase-2 acceptance test and the + thing that justifies the external manifest store. + +Phase 1 is validated: `lance_namespace` 0.11.1 with `impl=rest` drives the namespace, +`lance.write_dataset` writes to the vended location with the vended `storage_options`, and +the rows read back. Note that this client version drops `check_declared` and +`include_declared` on the wire, so `is_only_declared` reads null through it however the +server behaves. + +The commit path is validated at both levels. The mechanism: eight writers race the same +manifest key through S3 with `If-None-Match: *`, and exactly one wins. The property that +actually matters, which single-winner exclusivity does not by itself establish: eight +writers append to one dataset concurrently through lance, and afterwards every batch is +still there — the losers saw the conflict, rebased, and committed again. That second test +is also the sequence managed versioning could not complete at all, since its store answers +"put_if_not_exists is not supported" to the second commit. + +One more that belongs in the Iceberg suite, not this one: a Lance dataset registered through +the Iceberg adapter must survive a full maintenance pass. Reading the code, that test should +fail today; it has not been run. + +## Open questions + +- Root-level `ListNamespaces` enumerating table buckets is convenient and is a listing + surface we do not have on the Iceberg side. Decide whether it is gated behind a flag. +- Whether the `.lance` directory suffix is worth the divergence from the Iceberg layout. I + think yes — it is what makes the catalog optional — but it means the two catalogs' tables + do not look alike on disk, and the admin UI has to know that. +- Names: our charsets are lowercase-only and Lance identifiers are arbitrary strings. Reject + and document, as Iceberg does, or case-fold. Rejecting is right, but see #10734 for how + case handling bites when only one side normalizes. +- Whether to land generic-format registration first. Dropping the `"ICEBERG"` check and + letting a table carry an arbitrary format plus a location is smaller than this whole + design, gets Delta and Parquet catalogued as a side effect, and turns `Format: "LANCE"` + into an ordinary value. The argument against is that it invites tables the maintenance + worker cannot service, so it needs a "catalog-only, no maintenance" marker to be honest. diff --git a/seaweed-worker/.gitignore b/seaweed-worker/.gitignore new file mode 100644 index 000000000..2f7896d1d --- /dev/null +++ b/seaweed-worker/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/seaweed-worker/Cargo.lock b/seaweed-worker/Cargo.lock new file mode 100644 index 000000000..aaf222bb4 --- /dev/null +++ b/seaweed-worker/Cargo.lock @@ -0,0 +1,6754 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "const-random", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" +dependencies = [ + "memchr", +] + +[[package]] +name = "allocator-api2" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" + +[[package]] +name = "android_system_properties" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" +dependencies = [ + "libc", +] + +[[package]] +name = "anstream" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "anstyle-parse" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" +dependencies = [ + "anstyle", + "once_cell_polyfill", + "windows-sys 0.61.2", +] + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "arc-swap" +version = "1.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c049c0be4daef0b145cb3555416b3b8ef5b7888a38aea1a3a155801fe7b0810b" +dependencies = [ + "rustversion", +] + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.7.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" + +[[package]] +name = "arrow" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6cfdd0833e32a9874d2b55089333ad310c0be208aafa277385ce2461dec90be3" +dependencies = [ + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-csv", + "arrow-data", + "arrow-ipc", + "arrow-json", + "arrow-ord", + "arrow-row", + "arrow-schema", + "arrow-select", + "arrow-string", +] + +[[package]] +name = "arrow-arith" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0a41203398f0eaa6f7ec8e62c0da742a21abf282c148fc157f6c35c90e29981a" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-array" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae33dad492b7df00a217563a7b0ef2874df68a0deea1b1a3acf628152f7f7a69" +dependencies = [ + "ahash", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "chrono-tz", + "half", + "hashbrown 0.17.1", + "num-complex", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9552f96391c005e6ab449fa941420935e7e062489b12b8b1b08879b2163f5b5" +dependencies = [ + "bytes", + "half", + "num-bigint", + "num-traits", +] + +[[package]] +name = "arrow-cast" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a8a327c9649f30d8406995f27642b68df354713cca3baaaf100f076f18d5f34" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ord", + "arrow-schema", + "arrow-select", + "atoi", + "base64 0.22.1", + "chrono", + "comfy-table", + "half", + "lexical-core", + "num-traits", + "ryu", +] + +[[package]] +name = "arrow-csv" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af0dd6d90d1955e9f9a014c1e563ee8aeffc21909085d25623e1da44d96eca26" +dependencies = [ + "arrow-array", + "arrow-cast", + "arrow-schema", + "chrono", + "csv", + "csv-core", + "regex", +] + +[[package]] +name = "arrow-data" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b24852db04738907e06c04ea61e42fe7fda962a34513022dc0d0e754fb7976b" +dependencies = [ + "arrow-buffer", + "arrow-schema", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-ipc" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29a908a11fcfb3fb2f6730f4ac15e367bc644e419155e96238f68cf3adde572b" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "flatbuffers", + "lz4_flex", + "zstd", +] + +[[package]] +name = "arrow-json" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8a96aed3931c076adee39ec2a40d8219fc7f09e79bcdaca1df16272993e1e14" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-ord", + "arrow-schema", + "arrow-select", + "chrono", + "half", + "indexmap 2.14.0", + "itoa", + "lexical-core", + "memchr", + "num-traits", + "ryu", + "serde_core", + "serde_json", + "simdutf8", +] + +[[package]] +name = "arrow-ord" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63a083ec750f5c043f02946b4baf05fcdbb55f4560a3277055caca5cc99f3eb0" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", +] + +[[package]] +name = "arrow-row" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "514ba0ef0d4c5896202dae736251ce415abb43a950bed570fb7981b8716c0e4c" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "half", +] + +[[package]] +name = "arrow-schema" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "21ca356ad6425cecb6eb7b28e4f659f1ee7880fbb1a16127de7dd62901efee9e" +dependencies = [ + "bitflags 2.13.1", + "serde_core", + "serde_json", +] + +[[package]] +name = "arrow-select" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c58da39eb3d8350ad4a549e5c2bc49284dac554016c69829310350f1731b0aad" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "num-traits", +] + +[[package]] +name = "arrow-string" +version = "58.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6789b388467525e3271326b6b4915666ecfdf5142aef09779445c954b67543c" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + +[[package]] +name = "async-channel" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" +dependencies = [ + "concurrent-queue", + "event-listener-strategy", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-compression" +version = "0.4.43" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3976abdc8fe7d1133d43d304afd42abdf5bc3e1319d263d223bde07b5efc4be8" +dependencies = [ + "compression-codecs", + "compression-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "async-lock" +version = "3.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "290f7f2596bd5b78a9fec8088ccd89180d7f9f55b94b0576823bbbdc72ee8311" +dependencies = [ + "event-listener", + "event-listener-strategy", + "pin-project-lite", +] + +[[package]] +name = "async-recursion" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "async-stream" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" +dependencies = [ + "async-stream-impl", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-stream-impl" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "async-trait" +version = "0.1.92" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "async_cell" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "447ab28afbb345f5408b120702a44e5529ebf90b1796ec76e9528df8e288e6c2" +dependencies = [ + "loom", +] + +[[package]] +name = "atoi" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" +dependencies = [ + "num-traits", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "aws-config" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b180a3c8b55960db3426d8964b8745e652466a1a49fe1a2eda828046d30b5e4" +dependencies = [ + "aws-credential-types", + "aws-runtime", + "aws-sdk-sso", + "aws-sdk-ssooidc", + "aws-sdk-sts", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-types", + "bytes", + "fastrand", + "hex", + "http 1.5.0", + "sha1 0.10.7", + "time", + "tokio", + "tracing", + "url", + "zeroize", +] + +[[package]] +name = "aws-credential-types" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e93964ffdaf57857f544be3666a5f57570bb699e934700f11b49708f61bb556e" +dependencies = [ + "aws-smithy-async", + "aws-smithy-runtime-api", + "aws-smithy-types", + "zeroize", +] + +[[package]] +name = "aws-lc-rs" +version = "1.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e" +dependencies = [ + "aws-lc-sys", + "zeroize", +] + +[[package]] +name = "aws-lc-sys" +version = "0.44.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f09fae7be8bb3174e05c6afdb34199e6dc0c7c04ba9fa237b1967adfbde27483" +dependencies = [ + "cc", + "cmake", + "dunce", + "fs_extra", + "pkg-config", +] + +[[package]] +name = "aws-runtime" +version = "1.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9007227e10b5fed2f3e0a2beff489211e2b5604c400b7a9d5d81ca9d64c24bb" +dependencies = [ + "aws-credential-types", + "aws-sigv4", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-types", + "aws-types", + "bytes", + "bytes-utils", + "fastrand", + "http 1.5.0", + "http-body 1.1.0", + "percent-encoding", + "pin-project-lite", + "tracing", + "uuid", +] + +[[package]] +name = "aws-sdk-sso" +version = "1.106.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d0efcee834347b6705eca3eea2defd88242f43774f55d7326604222e3c86260" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-types", + "bytes", + "fastrand", + "http 0.2.12", + "http 1.5.0", + "regex-lite", + "tracing", +] + +[[package]] +name = "aws-sdk-ssooidc" +version = "1.108.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a59312a04cf19c962cfee32b64ecfee758f8786407ff6da5b30fff46ae96f201" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-types", + "bytes", + "fastrand", + "http 0.2.12", + "http 1.5.0", + "regex-lite", + "tracing", +] + +[[package]] +name = "aws-sdk-sts" +version = "1.111.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "120e7eb63457a9e547f9986fe3b273f77c43679da4d04f46359fa881c5e19b6e" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-query", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-smithy-xml", + "aws-types", + "fastrand", + "http 0.2.12", + "http 1.5.0", + "regex-lite", + "tracing", +] + +[[package]] +name = "aws-sigv4" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "723c2234ad7511ceef63eab016b7ba6ff7c55590fefb96fa8467af014a07309f" +dependencies = [ + "aws-credential-types", + "aws-smithy-http", + "aws-smithy-runtime-api", + "aws-smithy-types", + "bytes", + "form_urlencoded", + "hex", + "hmac", + "http 0.2.12", + "http 1.5.0", + "percent-encoding", + "sha2", + "time", + "tracing", +] + +[[package]] +name = "aws-smithy-async" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f02e407fb3b54891734224b9ffac8a71fdd35f542500fa1af95754a6b2beb316" +dependencies = [ + "futures-util", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "aws-smithy-http" +version = "0.64.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37843d9add67c3aff5856f409c6dc315d3cdff60f9c0cb5b670dab1e9920306d" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-types", + "bytes", + "bytes-utils", + "futures-core", + "futures-util", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "percent-encoding", + "pin-project-lite", + "pin-utils", + "tracing", +] + +[[package]] +name = "aws-smithy-http-client" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c1c8a04cb31ba74d0115af5a890bb8c0d48fba64b52812fa13929a6ef0cc83c" +dependencies = [ + "aws-smithy-async", + "aws-smithy-runtime-api", + "aws-smithy-types", + "h2", + "http 1.5.0", + "hyper", + "hyper-rustls", + "hyper-util", + "pin-project-lite", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "tokio", + "tokio-rustls", + "tower 0.5.3", + "tracing", +] + +[[package]] +name = "aws-smithy-json" +version = "0.63.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3dc65a121adb4b33729919fcfa14fa36fb33c1555a8f06bb0e2188dbfdc1d9ef" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", +] + +[[package]] +name = "aws-smithy-observability" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e86338c869539a581bf161247762a6e87f92c5c075060057b5ed6d06632ed0c" +dependencies = [ + "aws-smithy-runtime-api", +] + +[[package]] +name = "aws-smithy-query" +version = "0.62.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "512346c7212ab7436df2d77a16d976a468ae44a418835511d2a69269810aaf62" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-smithy-xml", + "urlencoding", +] + +[[package]] +name = "aws-smithy-runtime" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "483b858ff67522011c4786310c5cd8fd88d0be7ea3d5f1a48328446300c4269e" +dependencies = [ + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-http-client", + "aws-smithy-observability", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "bytes", + "fastrand", + "http 0.2.12", + "http 1.5.0", + "http-body 0.4.6", + "http-body 1.1.0", + "http-body-util", + "pin-project-lite", + "pin-utils", + "tokio", + "tracing", +] + +[[package]] +name = "aws-smithy-runtime-api" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b98f2e1fd67ec06618f9c291e5e495a468e60519e44c9c1979cd0521f3affdb" +dependencies = [ + "aws-smithy-async", + "aws-smithy-runtime-api-macros", + "aws-smithy-types", + "bytes", + "http 0.2.12", + "http 1.5.0", + "pin-project-lite", + "tokio", + "tracing", + "zeroize", +] + +[[package]] +name = "aws-smithy-runtime-api-macros" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "221eaa237ddf1ca79b60d1372aad77e47f9c0ea5b3ce5099da8c61d027dc77b3" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "aws-smithy-schema" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d56e0a4e53127a632224e43633b0fe045fa9e1e3cfc68b9830f1115e103f910" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-types", + "http 1.5.0", +] + +[[package]] +name = "aws-smithy-types" +version = "1.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fce83ce9abbb198d25bc7131e468d0f9fe1257125e58c39f3f9fc9f5098c9647" +dependencies = [ + "base64-simd", + "bytes", + "bytes-utils", + "http 0.2.12", + "http 1.5.0", + "http-body 0.4.6", + "http-body 1.1.0", + "http-body-util", + "itoa", + "num-integer", + "pin-project-lite", + "pin-utils", + "ryu", + "serde", + "time", +] + +[[package]] +name = "aws-smithy-xml" +version = "0.62.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce84f71c72fee2cbbadde6e7d082f5fb466e3a84733855295fa7aafd1b31b7d8" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "xmlparser", +] + +[[package]] +name = "aws-types" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eec1cd5469f328c782dc3e33d4153cf118a54e33cbb3356d60d16f89883e1f94" +dependencies = [ + "aws-credential-types", + "aws-smithy-async", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "rustc_version", + "tracing", +] + +[[package]] +name = "axum" +version = "0.7.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" +dependencies = [ + "async-trait", + "axum-core", + "bytes", + "futures-util", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "itoa", + "matchit", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "rustversion", + "serde", + "sync_wrapper", + "tower 0.5.3", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum-core" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199" +dependencies = [ + "async-trait", + "bytes", + "futures-util", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "mime", + "pin-project-lite", + "rustversion", + "sync_wrapper", + "tower-layer", + "tower-service", +] + +[[package]] +name = "backon" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef" +dependencies = [ + "fastrand", + "gloo-timers", + "tokio", +] + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + +[[package]] +name = "base64-simd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "339abbe78e73178762e23bea9dfd08e697eb3f3301cd4be981c0f78ba5859195" +dependencies = [ + "outref", + "vsimd", +] + +[[package]] +name = "bigdecimal" +version = "0.4.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" +dependencies = [ + "autocfg", + "libm", + "num-bigint", + "num-integer", + "num-traits", +] + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "bitvec" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddcec3d12c579d40898fe0a9a358a803c23e9c52ca3c425707f81c9436211837" +dependencies = [ + "funty", + "radium", + "tap", + "wyz", +] + +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest 0.10.7", +] + +[[package]] +name = "blake3" +version = "1.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76ae7bad254120e9e4c63bafc385310756f90c484eac0e36b8317cf09cb92a77" +dependencies = [ + "arrayref", + "arrayvec", + "cc", + "cfg-if", + "constant_time_eq", + "cpufeatures 0.3.0", +] + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bs58" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf88ba1141d185c399bee5288d850d63b8369520c1eafc32a0430b5b6c287bf4" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytemuck" +version = "1.25.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95832e849adfb21180ccb6826a99da14e5d266ae5c2e668e1602cf234f153797" +dependencies = [ + "bytemuck_derive", +] + +[[package]] +name = "bytemuck_derive" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc0e56a716f1e132ff6bf4bdac1c944a3fcdc1cae65f70a4a2a1ac3b401d2d1f" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "bytes-utils" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7dafe3a8757b027e2be6e4e5601ed563c55989fcf1546e933c66c8eb3a058d35" +dependencies = [ + "bytes", + "either", +] + +[[package]] +name = "cc" +version = "1.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "509591b7bcd67f4ef775afad7662703b4935daaa6ec0e5605cfb1090b32a2b6d" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + +[[package]] +name = "clap" +version = "4.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" +dependencies = [ + "anstream", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "cmake" +version = "0.1.58" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" +dependencies = [ + "cc", +] + +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + +[[package]] +name = "colorchoice" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + +[[package]] +name = "comfy-table" +version = "7.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "958c5d6ecf1f214b4c2bbbbf6ab9523a864bd136dcf71a7e8904799acfe1ad47" +dependencies = [ + "unicode-segmentation", + "unicode-width", +] + +[[package]] +name = "compression-codecs" +version = "0.4.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf" +dependencies = [ + "compression-core", + "flate2", + "memchr", +] + +[[package]] +name = "compression-core" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789" + +[[package]] +name = "concurrent-queue" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "core-foundation" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc32c" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" +dependencies = [ + "rustc_version", +] + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-channel" +version = "0.5.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-deque" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-epoch" +version = "0.9.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-queue" +version = "0.3.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "803d13fb3b09d88be9f4dbc29062c66b19bf7170867ceb746d2a8689bf6c7a26" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-skiplist" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df29de440c58ca2cc6e587ec3d22347551a32435fbde9d2bff64e78a9ffa151b" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "ctor" +version = "1.0.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "914a755b7c2d4af2bdcff7ce1739e2db9a1b81a9b07123d8015786ae03c0980d" +dependencies = [ + "link-section", + "linktime-proc-macro", +] + +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + +[[package]] +name = "darling" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" +dependencies = [ + "darling_core", + "darling_macro", +] + +[[package]] +name = "darling_core" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0" +dependencies = [ + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 2.0.119", +] + +[[package]] +name = "darling_macro" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" +dependencies = [ + "darling_core", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "dashmap" +version = "6.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c" +dependencies = [ + "cfg-if", + "crossbeam-utils", + "hashbrown 0.14.5", + "lock_api", + "once_cell", + "parking_lot_core", +] + +[[package]] +name = "datafusion" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "754ef4e8f073922a26f5b23133b9db4829342362b09be0bc94309cf261c2f098" +dependencies = [ + "arrow", + "arrow-schema", + "async-trait", + "chrono", + "datafusion-catalog", + "datafusion-catalog-listing", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-nested", + "datafusion-functions-table", + "datafusion-functions-window", + "datafusion-optimizer", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-optimizer", + "datafusion-physical-plan", + "datafusion-session", + "datafusion-sql", + "futures", + "indexmap 2.14.0", + "itertools", + "log", + "object_store", + "parking_lot", + "sqlparser", + "tempfile", + "tokio", + "url", + "uuid", +] + +[[package]] +name = "datafusion-catalog" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06afd1e38dd27bbb1258685a1fc6524df6aff4e07b25b393a47de59635178d99" +dependencies = [ + "arrow", + "async-trait", + "dashmap", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools", + "log", + "object_store", + "parking_lot", + "tokio", +] + +[[package]] +name = "datafusion-catalog-listing" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0668fb32c12065ec242be0e5b4bc62bd7a06a0be3ecd83791ef877e4be67e02" +dependencies = [ + "arrow", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "futures", + "itertools", + "log", + "object_store", +] + +[[package]] +name = "datafusion-common" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca43b263cdff57042cfa8fb817fb3469f4878933380dccff25f5e793580abbf9" +dependencies = [ + "arrow", + "arrow-ipc", + "arrow-schema", + "chrono", + "foldhash 0.2.0", + "half", + "hashbrown 0.17.1", + "indexmap 2.14.0", + "itertools", + "libc", + "log", + "object_store", + "sqlparser", + "tokio", + "uuid", + "web-time", +] + +[[package]] +name = "datafusion-common-runtime" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05f0ba2b864792bdca4d76c59a1de0ab6e1b61946596b9936888dbd6360035f2" +dependencies = [ + "futures", + "log", + "tokio", +] + +[[package]] +name = "datafusion-datasource" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b840a8bce0bcbf5afad02946d438591e7c373f7afccaf3d874c04485772514dd" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "chrono", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "glob", + "itertools", + "log", + "object_store", + "parking_lot", + "rand 0.9.5", + "tokio", + "url", +] + +[[package]] +name = "datafusion-datasource-arrow" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a24cc0b9cf6e367f27f27406eff13abf48a11b72446aaa40b3105c0ded5c17d9" +dependencies = [ + "arrow", + "arrow-ipc", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools", + "object_store", + "tokio", +] + +[[package]] +name = "datafusion-datasource-csv" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1abe56b2a7a2d1d6de5117dd1a203181e267f28529faa5da546947621b697d7" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "object_store", + "regex", + "tokio", +] + +[[package]] +name = "datafusion-datasource-json" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c3e467f0611ad7bdd5aad17c63c9bb6182d04e5282e5496d897ea2b49c024ba" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "object_store", + "tokio", + "tokio-stream", +] + +[[package]] +name = "datafusion-doc" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d69bb69d8769e34f76839c960dbde24c1ac0c885a79b6c3c2287bdc56ec67891" + +[[package]] +name = "datafusion-execution" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8eac0a09bc8d263f52025cad9e001da4d8138d633fa288edda4d06b1772eae6" +dependencies = [ + "arrow", + "arrow-buffer", + "async-trait", + "dashmap", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr-common", + "futures", + "log", + "object_store", + "parking_lot", + "rand 0.9.5", + "tempfile", + "url", +] + +[[package]] +name = "datafusion-expr" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eeb14d374767ee0fc62dc79a5ba8bcf8a63c14e993c7d992d0e63adfa23d77d3" +dependencies = [ + "arrow", + "arrow-schema", + "async-trait", + "chrono", + "datafusion-common", + "datafusion-doc", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr-common", + "indexmap 2.14.0", + "itertools", + "serde_json", + "sqlparser", +] + +[[package]] +name = "datafusion-expr-common" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7b19a8c95522bee8cbb313d74263b85e355d2b52f42e67ef5694bf5de9e9356" +dependencies = [ + "arrow", + "datafusion-common", + "indexmap 2.14.0", + "itertools", +] + +[[package]] +name = "datafusion-functions" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5f64c983bbbdcb729d921a2b2ac3375598719b5cc0c30345ad664936f3176fc7" +dependencies = [ + "arrow", + "arrow-buffer", + "base64 0.22.1", + "blake2", + "blake3", + "chrono", + "chrono-tz", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-macros", + "datafusion-physical-expr-common", + "hex", + "itertools", + "log", + "md-5 0.11.0", + "memchr", + "num-traits", + "rand 0.9.5", + "regex", + "sha2", + "uuid", +] + +[[package]] +name = "datafusion-functions-aggregate" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89bc17041e424a47ed062f43df24d84aab8b57c4c3221e5c1a5eef46d6c5718b" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "foldhash 0.2.0", + "half", + "log", + "num-traits", +] + +[[package]] +name = "datafusion-functions-aggregate-common" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97dd2a9e865c6108059f5b37b77934f84b50bfb108f837bd0e5c9536e03f0545" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-functions-nested" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75f0bdfeef16d96417b9632ef855645376b242e9006a126dfd0bedfc54a93f5f" +dependencies = [ + "arrow", + "arrow-ord", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr-common", + "hashbrown 0.17.1", + "itertools", + "itoa", + "log", + "memchr", +] + +[[package]] +name = "datafusion-functions-table" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f4e4941673c917819616877e9993da4503e4f4739812be0bc32c5356184c6383" +dependencies = [ + "arrow", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "parking_lot", +] + +[[package]] +name = "datafusion-functions-window" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12dd2e16c12b84b6f6b41b19f55b366dd1c46876bb35b86896c6349067379e8d" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-doc", + "datafusion-expr", + "datafusion-functions-window-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "log", +] + +[[package]] +name = "datafusion-functions-window-common" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cdc5e4b6f8b6ef823cc1c761f85088ad4c884fe8df64df3cbcc6b2b84698441" +dependencies = [ + "datafusion-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-macros" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a3614234dd93578c92428cb4f408e020874f0d2b7e6c90c928d9d28b5df2ceb" +dependencies = [ + "datafusion-doc", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "datafusion-optimizer" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0635620b050b81bb92764e99250868f654e2cd5ad1bece413283b3f73c83179" +dependencies = [ + "arrow", + "chrono", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "indexmap 2.14.0", + "itertools", + "log", + "regex", + "regex-syntax", +] + +[[package]] +name = "datafusion-physical-expr" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8cabf7a86eb70b816729e33c81bf7767c936ee1226f607a114f5dac2decac8d0" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr-common", + "half", + "hashbrown 0.17.1", + "indexmap 2.14.0", + "itertools", + "parking_lot", + "petgraph 0.8.3", + "tokio", +] + +[[package]] +name = "datafusion-physical-expr-adapter" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de222e04f7e6744501555a54ab0abe26bfdfebee380af79a9bdc175704246859" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "itertools", +] + +[[package]] +name = "datafusion-physical-expr-common" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72d0d0057fc5a502d45c870cb6d47c66eb7bdd5edb1bd71ad6f3f724975ac2a8" +dependencies = [ + "arrow", + "chrono", + "datafusion-common", + "datafusion-expr-common", + "hashbrown 0.17.1", + "indexmap 2.14.0", + "itertools", + "parking_lot", + "pin-project", +] + +[[package]] +name = "datafusion-physical-optimizer" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "86046eed10950c5f9aaed9acfd148e9bd2e1dfdfe4f9aef607d1447b271e4183" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "itertools", +] + +[[package]] +name = "datafusion-physical-plan" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9bc84da934c903407ba297971ebcc020c4c1a38aafd765d6c144c76eff3fa6a1" +dependencies = [ + "arrow", + "arrow-data", + "arrow-ipc", + "arrow-ord", + "arrow-schema", + "async-trait", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "futures", + "half", + "hashbrown 0.17.1", + "indexmap 2.14.0", + "itertools", + "log", + "num-traits", + "parking_lot", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "datafusion-pruning" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb63eeac6de19be40f487b65dd84e546195f783c5a9928618e0c4f2a3569b0d7" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "log", +] + +[[package]] +name = "datafusion-session" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5f961d209177f91bd014db5cbb2c33b7d28a2597b9003e77f17aeb712964315a" +dependencies = [ + "async-trait", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-plan", + "parking_lot", +] + +[[package]] +name = "datafusion-sql" +version = "54.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1d71cb454da682b2af7488e1fc1ddd72ee1b28f19297b8ccad73f0a21ee9a69" +dependencies = [ + "arrow", + "bigdecimal", + "chrono", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-nested", + "indexmap 2.14.0", + "log", + "regex", + "sqlparser", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +dependencies = [ + "serde_core", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer 0.10.4", + "crypto-common 0.1.7", + "subtle", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer 0.12.1", + "const-oid", + "crypto-common 0.2.2", + "ctutils", +] + +[[package]] +name = "dirs" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e" +dependencies = [ + "dirs-sys", +] + +[[package]] +name = "dirs-sys" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab" +dependencies = [ + "libc", + "option-ext", + "redox_users", + "windows-sys 0.61.2", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "dlv-list" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f" +dependencies = [ + "const-random", +] + +[[package]] +name = "dunce" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" + +[[package]] +name = "dyn-clone" +version = "1.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" + +[[package]] +name = "either" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e5e8f6c15a24b9a3ee5efec809ccd006d3b30e8b3bb63c39af737c7f87daa1d" + +[[package]] +name = "encoding_rs" +version = "0.8.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "ethnum" +version = "1.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40404c3f5f511ec4da6fe866ddf6a717c309fdbb69fbbad7b0f3edab8f2e835f" + +[[package]] +name = "event-listener" +version = "5.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a23add41df1562121a9393cb065eab5146a1242410f23a644851e90cfd669d2" +dependencies = [ + "parking", + "pin-project-lite", +] + +[[package]] +name = "event-listener-strategy" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" +dependencies = [ + "event-listener", + "pin-project-lite", +] + +[[package]] +name = "fast-float2" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6e8948ce679d00a02a94739ea185595dca7118ed04feb991127e443bd3d761f" + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "find-msvc-tools" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "flatbuffers" +version = "25.12.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" +dependencies = [ + "bitflags 2.13.1", + "rustc_version", +] + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "fs_extra" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" + +[[package]] +name = "fsst" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0981ce90521824089f3cd68a48b1c4c89e9ad96d0eb74f58d6e550a3d604545" +dependencies = [ + "arrow-array", + "rand 0.9.5", +] + +[[package]] +name = "fst" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ab85b9b05e3978cc9a9cf8fea7f01b494e1a09ed3037e16ba39edc7a29eb61a" +dependencies = [ + "utf8-ranges", +] + +[[package]] +name = "funty" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" + +[[package]] +name = "futures" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a31d2a3fbaaeb2af2368bbdd904aa8e812d3c04a1ee10d3171f52d556e5d0a3" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" + +[[package]] +name = "futures-executor" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "031b47cf1a3c6cc8bc2fc76cd437f521619387907d469316e7c0bc278f1f5432" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed" + +[[package]] +name = "futures-macro" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "futures-sink" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" + +[[package]] +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generator" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3b854b0e584ead1a33f18b2fcad7cf7be18b3875c78816b753639aa501513ae" +dependencies = [ + "cc", + "cfg-if", + "libc", + "log", + "rustversion", + "windows-link", + "windows-result", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "wasi", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", + "wasm-bindgen", +] + +[[package]] +name = "glob" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b" + +[[package]] +name = "gloo-timers" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbb143cf96099802033e0d4f4963b19fd2e0b728bcf076cd9cf7f6634f092994" +dependencies = [ + "futures-channel", + "futures-core", + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "h2" +version = "0.4.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http 1.5.0", + "indexmap 2.14.0", + "slab", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "bytemuck", + "cfg-if", + "crunchy", + "num-traits", + "zerocopy", +] + +[[package]] +name = "hashbrown" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash 0.1.5", +] + +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hermit-abi" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + +[[package]] +name = "http" +version = "0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "601cbb57e577e2f5ef5be8e7b83f0f63994f25aa94d673e54a92d5c516d101f1" +dependencies = [ + "bytes", + "fnv", + "itoa", +] + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ceab25649e9960c0311ea418d17bee82c0dcec1bd053b5f9a66e265a693bed2" +dependencies = [ + "bytes", + "http 0.2.12", + "pin-project-lite", +] + +[[package]] +name = "http-body" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" +dependencies = [ + "bytes", + "http 1.5.0", +] + +[[package]] +name = "http-body-util" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23169fe34a5fbcdd3f3862e78fb9b6fccd5f02a6dc6f732547005d45631ce71c" +dependencies = [ + "bytes", + "futures-core", + "http 1.5.0", + "http-body 1.1.0", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + +[[package]] +name = "humantime" +version = "2.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15cdd26707701c53297e2fa6afb323d55fbc1d0810c3aec078ae3ef0424c3c15" + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "hyper" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d22053281f852e11534f5198498373cbb59295120a20771d90f7ed1897490a72" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "h2", + "http 1.5.0", + "http-body 1.1.0", + "httparse", + "httpdate", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-rustls" +version = "0.27.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" +dependencies = [ + "http 1.5.0", + "hyper", + "hyper-util", + "rustls", + "rustls-native-certs", + "tokio", + "tokio-rustls", + "tower-service", + "webpki-roots", +] + +[[package]] +name = "hyper-timeout" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" +dependencies = [ + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64 0.22.1", + "bytes", + "futures-channel", + "futures-util", + "http 1.5.0", + "http-body 1.1.0", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2 0.6.5", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "hyperloglogplus" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "621debdf94dcac33e50475fdd76d34d5ea9c0362a834b9db08c3024696c1fbe3" +dependencies = [ + "serde", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "icu_collections" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb" +dependencies = [ + "displaydoc", + "litemap", + "serde", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_locale_fallback" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "251af8e57c9400e3eb58242fe5b8b1152b2a64fdf4cf632f923c38ccee6f2fa9" +dependencies = [ + "icu_locale_core", + "icu_locale_fallback_data", + "icu_provider", + "potential_utf", + "tinystr", + "zerovec", +] + +[[package]] +name = "icu_locale_fallback_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "decf2a22ec8fa68f1a0c1129a3f8583f8f8bc24e8b9ccbe98ead99f62a4dc3a8" + +[[package]] +name = "icu_normalizer" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0" + +[[package]] +name = "icu_properties" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148" +dependencies = [ + "displaydoc", + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa" + +[[package]] +name = "icu_provider" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92a7ed671a6aad807a8651a2e1782a6598fda9ce5185dd8158549e95a91c6428" +dependencies = [ + "displaydoc", + "icu_locale_core", + "serde", + "stable_deref_trait", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_segmenter" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82d07aafccd67af15d02512a6adf5896fbc5ed00f2e99b471d2efa14016db3db" +dependencies = [ + "icu_collections", + "icu_locale_fallback", + "icu_provider", + "icu_segmenter_data", + "potential_utf", + "smallvec", + "utf8_iter", + "zerovec", +] + +[[package]] +name = "icu_segmenter_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae293c039020f9ec10710af98d29ce6aa2051486638b49c9a6409f3b4a9e98ad" + +[[package]] +name = "ident_case" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "indexmap" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99" +dependencies = [ + "autocfg", + "hashbrown 0.12.3", + "serde", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", + "serde", + "serde_core", +] + +[[package]] +name = "io-uring" +version = "0.7.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d64d8ca234d152948ceaede1f419b6a83983a5ecccaac05fb337a809c96d3aa6" +dependencies = [ + "bitflags 2.13.1", + "cfg-if", + "libc", +] + +[[package]] +name = "ipnet" +version = "2.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78" + +[[package]] +name = "is_terminal_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.119", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.3", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e0c1080212aad755ea003d18543e8768dd432c48819efd73a7bf1e39b7a5a3a" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "jsonb" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb98fb29636087c40ad0d1274d9a30c0c1e83e03ae93f6e7e89247b37fcc6953" +dependencies = [ + "byteorder", + "ethnum", + "fast-float2", + "itoa", + "jiff", + "nom", + "num-traits", + "ordered-float", + "rand 0.9.5", + "serde", + "serde_json", + "zmij", +] + +[[package]] +name = "lance" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "830548f9fc92ae74b848a504282d4da06f016f2ee94b663773c2738b9cb61c42" +dependencies = [ + "arc-swap", + "arrow", + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-ipc", + "arrow-ord", + "arrow-row", + "arrow-schema", + "arrow-select", + "async-recursion", + "async-trait", + "async_cell", + "aws-credential-types", + "byteorder", + "bytes", + "chrono", + "crossbeam-queue", + "crossbeam-skiplist", + "dashmap", + "datafusion", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-plan", + "either", + "fst", + "futures", + "half", + "humantime", + "itertools", + "lance-arrow", + "lance-bitpacking", + "lance-core", + "lance-datafusion", + "lance-encoding", + "lance-file", + "lance-index", + "lance-io", + "lance-linalg", + "lance-namespace", + "lance-select", + "lance-table", + "lance-tokenizer", + "log", + "moka", + "object_store", + "permutation", + "pin-project", + "prost 0.14.4", + "prost-build 0.14.4", + "prost-types 0.14.4", + "rand 0.9.5", + "rayon", + "roaring", + "rustc-hash", + "semver", + "serde", + "serde_json", + "snafu", + "tokio", + "tokio-stream", + "tokio-util", + "tracing", + "url", + "uuid", +] + +[[package]] +name = "lance-arrow" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24187f972374567bb3573cffd8728f2f4f7f06d644c3c5d4430b79d6b0b67300" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ipc", + "arrow-ord", + "arrow-schema", + "arrow-select", + "bytemuck", + "bytes", + "futures", + "getrandom 0.2.17", + "half", + "jsonb", + "num-traits", + "rand 0.9.5", +] + +[[package]] +name = "lance-arrow-scalar" +version = "58.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "771f68b04b47f3addf781116f65061808de94b05e1e9411c23c18f32d14ebe79" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-data", + "arrow-row", + "arrow-schema", + "half", +] + +[[package]] +name = "lance-arrow-stats" +version = "58.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dd47ec33c90bf29f688fd02118e37d3a5ad5c339caa3163f89e417dc0867001f" +dependencies = [ + "arrow-array", + "arrow-schema", + "half", + "lance-arrow-scalar", +] + +[[package]] +name = "lance-bitpacking" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55fa50ad941e25298afb54eec107c3584f80c16db4adf7a6e4e9c4b6066f4281" +dependencies = [ + "arrayref", + "crunchy", + "paste", + "seq-macro", +] + +[[package]] +name = "lance-core" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "21f4fd872bfe948150a6878d983327c4f150dc1b629e94ae7677310fa6a3f35f" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "async-trait", + "blake3", + "byteorder", + "bytes", + "datafusion-common", + "datafusion-sql", + "futures", + "itertools", + "lance-arrow", + "lance-derive", + "libc", + "libm", + "log", + "moka", + "num_cpus", + "object_store", + "pin-project", + "prost 0.14.4", + "quick_cache", + "rand 0.9.5", + "roaring", + "serde_json", + "snafu", + "tempfile", + "tokio", + "tokio-stream", + "tokio-util", + "tracing", + "twox-hash", + "url", +] + +[[package]] +name = "lance-datafusion" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "230770734cd5f6fe1f1cb4a66750acad5c5a862c23c7f587d7def57c9f2c5f6b" +dependencies = [ + "arrow", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-ord", + "arrow-schema", + "arrow-select", + "async-trait", + "chrono", + "datafusion", + "datafusion-common", + "datafusion-functions", + "datafusion-physical-expr", + "futures", + "jsonb", + "lance-arrow", + "lance-core", + "lance-datagen", + "log", + "pin-project", + "prost 0.14.4", + "prost-build 0.14.4", + "tokio", + "tracing", +] + +[[package]] +name = "lance-datagen" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0c9edcea9dbf154a3baf470595bdcdb7ef6b44dcf2b5a0c28e8815c2d4837c5" +dependencies = [ + "arrow", + "arrow-array", + "arrow-cast", + "arrow-schema", + "chrono", + "futures", + "half", + "hex", + "rand 0.9.5", + "rand_distr", + "rand_xoshiro", +] + +[[package]] +name = "lance-derive" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03ac280ef94d66c2e7a4b0104e60d548f45156f114f02cd0ac57423721ec96ec" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "lance-encoding" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65b345fecf1792d4147982e9ded1cf211992a442bcf73dd17a598b1c8c9b53a5" +dependencies = [ + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-data", + "arrow-schema", + "arrow-select", + "bytemuck", + "byteorder", + "bytes", + "fsst", + "futures", + "hex", + "hyperloglogplus", + "itertools", + "lance-arrow", + "lance-bitpacking", + "lance-core", + "log", + "lz4", + "num-traits", + "prost 0.14.4", + "prost-build 0.14.4", + "rand 0.9.5", + "strum", + "tokio", + "tracing", + "xxhash-rust", + "zstd", +] + +[[package]] +name = "lance-file" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "203400160ecd4ac6f67f6bfdb3c186f94758ff70a6ca76d4f300c327f5c14a11" +dependencies = [ + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-data", + "arrow-schema", + "arrow-select", + "async-recursion", + "async-trait", + "byteorder", + "bytes", + "datafusion-common", + "futures", + "lance-arrow", + "lance-core", + "lance-encoding", + "lance-io", + "log", + "num-traits", + "object_store", + "prost 0.14.4", + "prost-build 0.14.4", + "prost-types 0.14.4", + "tokio", + "tracing", +] + +[[package]] +name = "lance-index" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db67792e22332a7be08d35e60ce9e4c6fb7f75f70ab1566b67b462d1aac21b13" +dependencies = [ + "arc-swap", + "arrow", + "arrow-arith", + "arrow-array", + "arrow-ipc", + "arrow-ord", + "arrow-schema", + "arrow-select", + "async-channel", + "async-recursion", + "async-trait", + "bitvec", + "bytes", + "chrono", + "crossbeam-queue", + "datafusion", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", + "dirs", + "fst", + "futures", + "half", + "itertools", + "jsonb", + "lance-arrow", + "lance-arrow-stats", + "lance-bitpacking", + "lance-core", + "lance-datafusion", + "lance-datagen", + "lance-encoding", + "lance-file", + "lance-index-core", + "lance-io", + "lance-linalg", + "lance-select", + "lance-table", + "lance-tokenizer", + "libsais-rs", + "log", + "ndarray", + "num-traits", + "object_store", + "prost 0.14.4", + "prost-build 0.14.4", + "prost-types 0.14.4", + "rand 0.9.5", + "rand_distr", + "rangemap", + "rayon", + "regex-syntax", + "roaring", + "serde", + "serde_json", + "smallvec", + "tempfile", + "tokio", + "tracing", + "uuid", +] + +[[package]] +name = "lance-index-core" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49b5700d71f43e5da2057908f3243a6bc01247b5dd12247313f789c280155aae" +dependencies = [ + "arrow-array", + "arrow-schema", + "arrow-select", + "async-trait", + "bytes", + "datafusion", + "datafusion-common", + "datafusion-expr", + "futures", + "lance-core", + "lance-io", + "lance-select", + "prost-types 0.14.4", + "roaring", + "serde", + "serde_json", +] + +[[package]] +name = "lance-io" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b8a6872b9bff95e75e223e04011feebd990d8784878eaf5df73c86d6acebb47" +dependencies = [ + "arrow", + "arrow-array", + "arrow-schema", + "async-trait", + "aws-config", + "aws-credential-types", + "byteorder", + "bytes", + "chrono", + "futures", + "http 1.5.0", + "io-uring", + "lance-arrow", + "lance-core", + "lance-namespace", + "log", + "moka", + "object_store", + "object_store_opendal", + "opendal", + "path_abs", + "pin-project", + "prost 0.14.4", + "rand 0.9.5", + "serde", + "tempfile", + "tokio", + "tracing", + "url", +] + +[[package]] +name = "lance-linalg" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a1dd49fdf61da86650dc335fbe1f239803f9d73bc2bc5ffbc0c905dd8809892" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-schema", + "cc", + "half", + "lance-arrow", + "lance-core", + "num-traits", + "rand 0.9.5", + "rayon", +] + +[[package]] +name = "lance-namespace" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1449221f599147e570bec6e09eaea6ce6b9286f632e00a27f945dd0c6b70ccd8" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "lance-core", + "lance-namespace-reqwest-client", + "snafu", +] + +[[package]] +name = "lance-namespace-reqwest-client" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba3f0a235e3ed5f8805205649ccc7d7d0f3df23ce1294242c9265ad488d7f19d" +dependencies = [ + "reqwest 0.12.28", + "serde", + "serde_json", + "serde_repr", + "serde_with", + "url", +] + +[[package]] +name = "lance-select" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c66c5cb12ed194c7c8e427a0e90e990cdbc7159edba2bfc7b7e8db2c90c747af" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-schema", + "byteorder", + "bytes", + "itertools", + "lance-core", + "roaring", + "tracing", +] + +[[package]] +name = "lance-table" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6cd2a39f2e288d1acecf887cb9cc655711fb67bcf3b24a2034158dbac7a6f60e" +dependencies = [ + "arrow", + "arrow-array", + "arrow-buffer", + "arrow-ipc", + "arrow-schema", + "async-trait", + "byteorder", + "bytes", + "chrono", + "futures", + "lance-arrow", + "lance-core", + "lance-file", + "lance-io", + "lance-select", + "log", + "object_store", + "prost 0.14.4", + "prost-build 0.14.4", + "prost-types 0.14.4", + "rand 0.9.5", + "rangemap", + "roaring", + "semver", + "serde", + "serde_json", + "snafu", + "tokio", + "tracing", + "url", + "uuid", +] + +[[package]] +name = "lance-tokenizer" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "254c1e9287789c8d744f93f2010a8389a5ea3fcedc0dddf7649c4117ec830c57" +dependencies = [ + "icu_segmenter", + "rust-stemmers", + "serde", + "stop-words", + "unicode-normalization", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "lexical-core" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" +dependencies = [ + "lexical-parse-float", + "lexical-parse-integer", + "lexical-util", + "lexical-write-float", + "lexical-write-integer", +] + +[[package]] +name = "lexical-parse-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" +dependencies = [ + "lexical-parse-integer", + "lexical-util", +] + +[[package]] +name = "lexical-parse-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "lexical-util" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" + +[[package]] +name = "lexical-write-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" +dependencies = [ + "lexical-util", + "lexical-write-integer", +] + +[[package]] +name = "lexical-write-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "libredox" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28d0a00925a9f930d679b6789b721e3a7f9ed110f41b86d2497caa780c3a070a" +dependencies = [ + "libc", +] + +[[package]] +name = "libsais-rs" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "00777f770949ecbe5524eb6630699a501c9c27e4d18493705a7dad49c9bdfbd1" +dependencies = [ + "rayon", +] + +[[package]] +name = "link-section" +version = "0.19.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39c29a617ce3df32c08497bdc1ab6e2376e0b17948ac166a2fbe5977c5954cd9" + +[[package]] +name = "linktime-proc-macro" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e57c38c1e860fd37c604281cdfb1dd2216977fd76a50f85ba2f388ef3219616" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae" + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "loom" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca" +dependencies = [ + "cfg-if", + "generator", + "scoped-tls", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "lru-slab" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" + +[[package]] +name = "lz4" +version = "1.28.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a20b523e860d03443e98350ceaac5e71c6ba89aea7d960769ec3ce37f4de5af4" +dependencies = [ + "lz4-sys", +] + +[[package]] +name = "lz4-sys" +version = "1.11.1+lz4-1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6bd8c0d6c6ed0cd30b3652886bb8711dc4bb01d637a68105a3d5158039b418e6" +dependencies = [ + "cc", + "libc", +] + +[[package]] +name = "lz4_flex" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "matchers" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9" +dependencies = [ + "regex-automata", +] + +[[package]] +name = "matchit" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94" + +[[package]] +name = "matrixmultiply" +version = "0.3.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7" +dependencies = [ + "autocfg", + "num_cpus", + "once_cell", + "rawpointer", + "thread-tree", +] + +[[package]] +name = "md-5" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" +dependencies = [ + "cfg-if", + "digest 0.10.7", +] + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "mea" +version = "0.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c709842c4ce65cb91e2666ad5319dfc1efc3af0d34f02075eddca9000d9f8afb" +dependencies = [ + "hashbrown 0.17.1", + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mime_guess" +version = "2.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" +dependencies = [ + "mime", + "unicase", +] + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "mio" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" +dependencies = [ + "libc", + "wasi", + "windows-sys 0.61.2", +] + +[[package]] +name = "moka" +version = "0.12.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4293f18e7567a1caf3c584855554377025c65e0aa445344d04171f5ad63d19b9" +dependencies = [ + "async-lock", + "crossbeam-channel", + "crossbeam-epoch", + "crossbeam-utils", + "equivalent", + "event-listener", + "futures-util", + "parking_lot", + "portable-atomic", + "smallvec", + "tagptr", + "uuid", +] + +[[package]] +name = "multimap" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" + +[[package]] +name = "ndarray" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "882ed72dce9365842bf196bdeedf5055305f11fc8c03dee7bb0194a6cad34841" +dependencies = [ + "matrixmultiply", + "num-complex", + "num-integer", + "num-traits", + "portable-atomic", + "portable-atomic-util", + "rawpointer", +] + +[[package]] +name = "nom" +version = "8.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405" +dependencies = [ + "memchr", +] + +[[package]] +name = "nu-ansi-term" +version = "0.50.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-integer" +version = "0.1.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", + "libm", +] + +[[package]] +name = "num_cpus" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" +dependencies = [ + "hermit-abi", + "libc", +] + +[[package]] +name = "object_store" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" +dependencies = [ + "async-trait", + "base64 0.22.1", + "bytes", + "chrono", + "form_urlencoded", + "futures-channel", + "futures-core", + "futures-util", + "http 1.5.0", + "http-body-util", + "humantime", + "hyper", + "itertools", + "md-5 0.10.6", + "parking_lot", + "percent-encoding", + "quick-xml 0.39.4", + "rand 0.10.2", + "reqwest 0.12.28", + "ring", + "serde", + "serde_json", + "serde_urlencoded", + "thiserror", + "tokio", + "tracing", + "url", + "walkdir", + "wasm-bindgen-futures", + "web-time", +] + +[[package]] +name = "object_store_opendal" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eb12a624a41fce745838d0ef3701ff6c47797c13cd18ad3612fd2a3134fdbd8" +dependencies = [ + "async-trait", + "bytes", + "chrono", + "futures", + "mea", + "object_store", + "opendal", + "pin-project", + "tokio", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "once_cell_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" + +[[package]] +name = "opendal" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96c9c85ce253ff87225e7669979d877a20c98a06604ec9d6dd5f4473e08f1ae1" +dependencies = [ + "ctor", + "opendal-core", + "opendal-layer-concurrent-limit", + "opendal-layer-logging", + "opendal-layer-retry", + "opendal-layer-timeout", + "opendal-service-s3", +] + +[[package]] +name = "opendal-core" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4f8607c90e2c963a91467f50fb49fbc7fb3d573f88cea219ca59ccd3740b309" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bytes", + "futures", + "http 1.5.0", + "http-body 1.1.0", + "jiff", + "log", + "md-5 0.11.0", + "mea", + "percent-encoding", + "quick-xml 0.39.4", + "reqsign-core", + "reqwest 0.13.4", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-layer-concurrent-limit" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d6f81ba6960e3fae1882f253b114b21d7e444e1534f209c7737a79f6243eb6f" +dependencies = [ + "futures", + "http 1.5.0", + "mea", + "opendal-core", +] + +[[package]] +name = "opendal-layer-logging" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58ada45c6d81d1aa4c9305d0c7d4bc317c59c85866a0908a2d75a7a978aa5ee2" +dependencies = [ + "log", + "opendal-core", +] + +[[package]] +name = "opendal-layer-retry" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b2a25a718afb81fad81cb9a0580a1cb989221fa2317f888c6a37f8dad408eb7" +dependencies = [ + "backon", + "log", + "opendal-core", +] + +[[package]] +name = "opendal-layer-timeout" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e91f731724c213af81e9d03517859c8fc47b4578e64ad61ae4f099f10fe36e3" +dependencies = [ + "opendal-core", + "tokio", +] + +[[package]] +name = "opendal-service-s3" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "313d46c9f5ae70bca26b7c3e3fbb9b639292625f28af73aa016f47e788af9deb" +dependencies = [ + "base64 0.22.1", + "bytes", + "crc32c", + "http 1.5.0", + "log", + "md-5 0.11.0", + "opendal-core", + "quick-xml 0.39.4", + "reqsign-aws-v4", + "reqsign-core", + "reqsign-file-read-tokio", + "serde", + "url", +] + +[[package]] +name = "openssl-probe" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" + +[[package]] +name = "option-ext" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" + +[[package]] +name = "ordered-float" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7d950ca161dc355eaf28f82b11345ed76c6e1f6eb1f4f4479e0323b9e2fbd0e" +dependencies = [ + "num-traits", +] + +[[package]] +name = "ordered-multimap" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79" +dependencies = [ + "dlv-list", + "hashbrown 0.14.5", +] + +[[package]] +name = "outref" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" + +[[package]] +name = "parking" +version = "2.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-link", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "path_abs" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05ef02f6342ac01d8a93b65f96db53fe68a92a15f41144f97fb00a9e669633c3" +dependencies = [ + "serde", + "serde_derive", + "std_prelude", + "stfu8", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "permutation" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df202b0b0f5b8e389955afd5f27b007b00fb948162953f1db9c70d2c7e3157d7" + +[[package]] +name = "petgraph" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +dependencies = [ + "fixedbitset", + "indexmap 2.14.0", +] + +[[package]] +name = "petgraph" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" +dependencies = [ + "fixedbitset", + "hashbrown 0.15.5", + "indexmap 2.14.0", + "serde", +] + +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_shared", +] + +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "pkg-config" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548" + +[[package]] +name = "portable-atomic" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661" +dependencies = [ + "serde_core", + "writeable", + "zerovec", +] + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn 2.0.119", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "prost" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" +dependencies = [ + "bytes", + "prost-derive 0.13.5", +] + +[[package]] +name = "prost" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1" +dependencies = [ + "bytes", + "prost-derive 0.14.4", +] + +[[package]] +name = "prost-build" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" +dependencies = [ + "heck", + "itertools", + "log", + "multimap", + "once_cell", + "petgraph 0.7.1", + "prettyplease", + "prost 0.13.5", + "prost-types 0.13.5", + "regex", + "syn 2.0.119", + "tempfile", +] + +[[package]] +name = "prost-build" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" +dependencies = [ + "heck", + "itertools", + "log", + "multimap", + "petgraph 0.8.3", + "prettyplease", + "prost 0.14.4", + "prost-types 0.14.4", + "regex", + "syn 2.0.119", + "tempfile", +] + +[[package]] +name = "prost-derive" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "prost-derive" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "prost-types" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" +dependencies = [ + "prost 0.13.5", +] + +[[package]] +name = "prost-types" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f94967dc7688f3054c7fac87473ffae4cc4c3904800e2d9f5b857246d8963b0a" +dependencies = [ + "prost 0.14.4", +] + +[[package]] +name = "quick-xml" +version = "0.39.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdcc8dd4e2f670d309a5f0e83fe36dfdc05af317008fea29144da1a2ac858e5e" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quick_cache" +version = "0.6.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9c6658afe513a3b484e3abfdaa0d03ef3c0bbf017542c178dd55f94eb3051f9" +dependencies = [ + "ahash", + "equivalent", + "hashbrown 0.16.1", + "parking_lot", +] + +[[package]] +name = "quinn" +version = "0.11.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash", + "rustls", + "socket2 0.6.5", + "thiserror", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "04759210543be93709136e28212294a659ef5001836ff4eab4d663e4529bba83" +dependencies = [ + "aws-lc-rs", + "bytes", + "getrandom 0.4.3", + "lru-slab", + "rand 0.10.2", + "rand_pcg", + "ring", + "rustc-hash", + "rustls", + "rustls-pki-types", + "slab", + "thiserror", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2 0.6.5", + "tracing", + "windows-sys 0.61.2", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "radium" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" + +[[package]] +name = "rand" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22f6172bdec972074665ed81ed53b71da00bfc44b65a753cfde883ec4c702a1a" +dependencies = [ + "libc", + "rand_chacha 0.3.1", + "rand_core 0.6.4", +] + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha 0.9.0", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core 0.6.4", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rand_distr" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8615d50dcf34fa31f7ab52692afec947c4dd0ab803cc87cb3b0b4570ff7463" +dependencies = [ + "num-traits", + "rand 0.9.5", +] + +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + +[[package]] +name = "rand_xoshiro" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f703f4665700daf5512dcca5f43afa6af89f09db47fb56be587f80636bda2d41" +dependencies = [ + "rand_core 0.9.5", +] + +[[package]] +name = "rangemap" +version = "1.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a611d15b50743feb4c76b7d03edcb0e64f399c26961e4efe6975bc398be6aa3d" + +[[package]] +name = "rawpointer" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" + +[[package]] +name = "rayon" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" +dependencies = [ + "either", + "rayon-core", +] + +[[package]] +name = "rayon-core" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" +dependencies = [ + "crossbeam-deque", + "crossbeam-utils", +] + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags 2.13.1", +] + +[[package]] +name = "redox_users" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac" +dependencies = [ + "getrandom 0.2.17", + "libredox", + "thiserror", +] + +[[package]] +name = "ref-cast" +version = "1.0.26" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "216e8f773d7923bcba9ceb86a86c93cabb3903a11872fc3f138c49630e50b96d" +dependencies = [ + "ref-cast-impl", +] + +[[package]] +name = "ref-cast-impl" +version = "1.0.26" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9283685feec7d69af75fb0e858d5e7378f33fe4fc699383b2916ab9273e03c" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "regex" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-lite" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "reqsign-aws-core" +version = "3.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d63b56638bb3cc7bd376a7cdce1ba3089777a08f47e4097888f2d784cc3f46c" +dependencies = [ + "bytes", + "form_urlencoded", + "hex", + "http 1.5.0", + "log", + "percent-encoding", + "quick-xml 0.41.0", + "reqsign-core", + "rust-ini", + "serde", + "serde_json", + "serde_urlencoded", + "sha1 0.11.0", +] + +[[package]] +name = "reqsign-aws-v4" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4a0c499f4ed12d04c3d4c78fe4cb01aee22c9dae22848c14db2c6313d9df9f43" +dependencies = [ + "bytes", + "http 1.5.0", + "log", + "quick-xml 0.41.0", + "reqsign-aws-core", + "reqsign-core", + "serde", +] + +[[package]] +name = "reqsign-core" +version = "3.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f4ac1510872d9481205975d264deb39c109797e5068cc882ed9064270eaae5fa" +dependencies = [ + "anyhow", + "base64 0.23.1", + "bytes", + "futures", + "hex", + "hmac", + "http 1.5.0", + "jiff", + "log", + "percent-encoding", + "sha1 0.11.0", + "sha2", + "windows-sys 0.61.2", +] + +[[package]] +name = "reqsign-file-read-tokio" +version = "3.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95c3371bfc7e5c7f9627a04133af3583fd6c28715e7c83f79db38f3b384f535f" +dependencies = [ + "anyhow", + "reqsign-core", + "tokio", +] + +[[package]] +name = "reqwest" +version = "0.12.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" +dependencies = [ + "base64 0.22.1", + "bytes", + "encoding_rs", + "futures-core", + "futures-util", + "h2", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "mime", + "mime_guess", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower 0.5.3", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams 0.4.2", + "web-sys", + "webpki-roots", +] + +[[package]] +name = "reqwest" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" +dependencies = [ + "base64 0.22.1", + "bytes", + "futures-core", + "futures-util", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "rustls-platform-verifier", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower 0.5.3", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams 0.5.0", + "web-sys", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "roaring" +version = "0.11.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18bd8a37d17a58532776dcdf6041ce64929adca78e8489d5cacbafe99229d3e1" +dependencies = [ + "bytemuck", + "byteorder", +] + +[[package]] +name = "rust-ini" +version = "0.21.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7" +dependencies = [ + "cfg-if", + "ordered-multimap", +] + +[[package]] +name = "rust-stemmers" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e46a2036019fdb888131db7a4c847a1063a7493f971ed94ea82c67eada63ca54" +dependencies = [ + "serde", + "serde_derive", +] + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags 2.13.1", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls" +version = "0.23.43" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06" +dependencies = [ + "aws-lc-rs", + "log", + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-native-certs" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" +dependencies = [ + "openssl-probe", + "rustls-pki-types", + "schannel", + "security-framework", +] + +[[package]] +name = "rustls-pemfile" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "rustls-pki-types" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96" +dependencies = [ + "web-time", + "zeroize", +] + +[[package]] +name = "rustls-platform-verifier" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0" +dependencies = [ + "core-foundation", + "core-foundation-sys", + "jni", + "log", + "once_cell", + "rustls", + "rustls-native-certs", + "rustls-platform-verifier-android", + "rustls-webpki", + "security-framework", + "security-framework-sys", + "webpki-root-certs", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls-platform-verifier-android" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" + +[[package]] +name = "rustls-webpki" +version = "0.103.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0527518605e68109d875e248ea259b6758801cf165e4b2c2733ae3b51f12535a" +dependencies = [ + "aws-lc-rs", + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "schannel" +version = "0.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "schemars" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cd191f9397d57d581cddd31014772520aa448f65ef991055d7f61582c65165f" +dependencies = [ + "dyn-clone", + "ref-cast", + "serde", + "serde_json", +] + +[[package]] +name = "schemars" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a" +dependencies = [ + "dyn-clone", + "ref-cast", + "serde", + "serde_json", +] + +[[package]] +name = "scoped-tls" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "seaweed-worker-core" +version = "0.1.0" +dependencies = [ + "anyhow", + "async-trait", + "prost 0.13.5", + "prost-types 0.13.5", + "tokio", + "tokio-stream", + "tonic", + "tonic-build", + "tracing", +] + +[[package]] +name = "security-framework" +version = "3.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" +dependencies = [ + "bitflags 2.13.1", + "core-foundation", + "core-foundation-sys", + "libc", + "security-framework-sys", +] + +[[package]] +name = "security-framework-sys" +version = "2.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_repr" +version = "0.1.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8d3b1629de253c70a0508c3899572da79ca359fdab27c7920ff00406df418906" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "serde_with" +version = "3.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee78f1fbe43ac4a0e47aadb3dbd357b69eb0d3793e948624cd03dd2750ab1c0a" +dependencies = [ + "base64 0.22.1", + "bs58", + "chrono", + "hex", + "indexmap 1.9.3", + "indexmap 2.14.0", + "jiff", + "schemars 0.9.0", + "schemars 1.2.2", + "serde_core", + "serde_json", + "serde_with_macros", + "time", +] + +[[package]] +name = "serde_with_macros" +version = "3.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8705578779c2b6bd90d84d66eb2e206b708b1a4d7b9f17641b293545bf1c7e46" +dependencies = [ + "darling", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "sha1" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.3", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.3", +] + +[[package]] +name = "sharded-slab" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +dependencies = [ + "lazy_static", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "simd-adler32" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea" + +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "snafu" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e45cb604038abb7b926b679887b3226d8d0f23874b66623625a0454be425a4b7" +dependencies = [ + "snafu-derive", +] + +[[package]] +name = "snafu-derive" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "287f59010008f0d7cf5e3b03196d666c1acc46c8d3e9cf34c28a1a7157601e72" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "socket2" +version = "0.5.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678" +dependencies = [ + "libc", + "windows-sys 0.52.0", +] + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "sqlparser" +version = "0.62.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c6d1b651dc4edf07eead2a0c6c78016ce971bc2c10da5266861b13f25e7cec" +dependencies = [ + "log", + "sqlparser_derive", +] + +[[package]] +name = "sqlparser_derive" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6dd45d8fc1c79299bfbb7190e42ccbbdf6a5f52e4a6ad98d92357ea965bd289" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "std_prelude" +version = "0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8207e78455ffdf55661170876f88daf85356e4edd54e0a3dbc79586ca1e50cbe" + +[[package]] +name = "stfu8" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e51f1e89f093f99e7432c491c382b88a6860a5adbe6bf02574bf0a08efff1978" + +[[package]] +name = "stop-words" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d68df56303396bcfb639455b3c166804aeb7994005010aab5e9e8a1277b8871d" +dependencies = [ + "serde_json", +] + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" +dependencies = [ + "strum_macros", +] + +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "rustversion", + "syn 2.0.119", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tagptr" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" + +[[package]] +name = "tap" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.3", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "thiserror" +version = "2.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "thread-tree" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffbd370cb847953a25954d9f63e14824a36113f8c72eecf6eccef5dc4b45d630" +dependencies = [ + "crossbeam-channel", +] + +[[package]] +name = "thread_local" +version = "1.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "time" +version = "0.3.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643" +dependencies = [ + "displaydoc", + "serde_core", + "zerovec", +] + +[[package]] +name = "tinyvec" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb4ebadaa0af04fab11ae01eb5f9fdb5f9c5b875506e210e71c07873528baa7f" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "libc", + "mio", + "parking_lot", + "pin-project-lite", + "signal-hook-registry", + "socket2 0.6.5", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3d06f0b082ba57c26b79407372e57cf2a1e28124f78e9479fe80322cf53420b" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", + "tokio-util", +] + +[[package]] +name = "tokio-util" +version = "0.7.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "libc", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tonic" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52" +dependencies = [ + "async-stream", + "async-trait", + "axum", + "base64 0.22.1", + "bytes", + "h2", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "hyper", + "hyper-timeout", + "hyper-util", + "percent-encoding", + "pin-project", + "prost 0.13.5", + "rustls-pemfile", + "socket2 0.5.10", + "tokio", + "tokio-rustls", + "tokio-stream", + "tower 0.4.13", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tonic-build" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9557ce109ea773b399c9b9e5dca39294110b74f1f342cb347a80d1fce8c26a11" +dependencies = [ + "prettyplease", + "proc-macro2", + "prost-build 0.13.5", + "prost-types 0.13.5", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tower" +version = "0.4.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c" +dependencies = [ + "futures-core", + "futures-util", + "indexmap 1.9.3", + "pin-project", + "pin-project-lite", + "rand 0.8.7", + "slab", + "tokio", + "tokio-util", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "async-compression", + "bitflags 2.13.1", + "bytes", + "futures-core", + "futures-util", + "http 1.5.0", + "http-body 1.1.0", + "http-body-util", + "pin-project-lite", + "tokio", + "tokio-util", + "tower 0.5.3", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", + "valuable", +] + +[[package]] +name = "tracing-log" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" +dependencies = [ + "log", + "once_cell", + "tracing-core", +] + +[[package]] +name = "tracing-subscriber" +version = "0.3.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" +dependencies = [ + "matchers", + "nu-ansi-term", + "once_cell", + "regex-automata", + "sharded-slab", + "smallvec", + "thread_local", + "tracing", + "tracing-core", + "tracing-log", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "twox-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9" +dependencies = [ + "rand 0.10.2", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicase" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-normalization" +version = "0.1.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" +dependencies = [ + "tinyvec", +] + +[[package]] +name = "unicode-segmentation" +version = "1.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" + +[[package]] +name = "unicode-width" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "urlencoding" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" + +[[package]] +name = "utf8-ranges" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fcfc827f90e53a02eaef5e535ee14266c1d569214c6aa70133a624d8a3164ba" + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + +[[package]] +name = "uuid" +version = "1.24.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cefc03fd367c0c6d4305de1b312cf00248c4114f4a0418ce6a6af769e3b0bd9" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "valuable" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "vsimd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.127" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b70935747edd64d89de3efa29d73789b806c15798f8e7dca4d8ac356b50ce70" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.77" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b7777d5cc23d0e91404e53ce2d5e8ec7acae3026b16233dba62cd3246457950" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.127" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77775f8f3f7217702089053b94958f8f54061a3f663417df76e19cbdcca29bc1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.127" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e11d33f857dc2fb11b8bc75aee111aa9cbeb12cd9f25efd3d4c2a3dd4e235284" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.127" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ef64dbcc55df09c7e5a46182d181c2cfa3e925f3da937ea764728b4bbb9dcbf" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-streams" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "wasm-streams" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "web-sys" +version = "0.3.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c435338968042f4f59a557f690a253676d47ce13ceb55d70100e7facf6620a30" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "webpki-root-certs" +version = "1.0.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b96554aa2acc8ccdb7e1c9a58a7a68dd5d13bccc69cd124cb09406db612a1c9b" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "webpki-roots" +version = "1.0.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7dcd9d09a39985f5344844e66b0c530a33843579125f23e21e9f0f220850f22a" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "weed-lance-worker" +version = "0.1.0" +dependencies = [ + "anyhow", + "arrow-array", + "arrow-cast", + "arrow-schema", + "async-trait", + "chrono", + "clap", + "futures", + "lance", + "lance-index", + "lance-linalg", + "reqwest 0.12.28", + "seaweed-worker-core", + "serde", + "serde_json", + "tokio", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "writeable" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc" + +[[package]] +name = "wyz" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" +dependencies = [ + "tap", +] + +[[package]] +name = "xmlparser" +version = "0.13.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" + +[[package]] +name = "xxhash-rust" +version = "0.8.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6" + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerocopy" +version = "0.8.56" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "556764e583adb45a9f8d413c2a147fa7e8d821e48e12b14fd560b607998b75eb" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.56" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2ab42fc20575779bd240faa45f94a74256f755c0fa9e89f0ede20d91d0cdfc1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" + +[[package]] +name = "zerotrie" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "zerovec" +version = "0.11.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94b5c6b5976d66c1d703c4fd17d3f5e43c8cedaacf604961b171adc7130896d8" +dependencies = [ + "serde", + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f212a141d820099d57ffafb9569be9617a6f27d3dc881fbee8fb56642f917a9" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/seaweed-worker/Cargo.toml b/seaweed-worker/Cargo.toml new file mode 100644 index 000000000..ef33f34e7 --- /dev/null +++ b/seaweed-worker/Cargo.toml @@ -0,0 +1,24 @@ +# Rust plugin workers for SeaweedFS. +# +# `core` is the plugin.proto contract and nothing else; a worker crate beside it +# supplies job handlers and a binary. Adding a worker means adding a member here, +# not touching the protocol. +[workspace] +resolver = "2" +members = ["crates/core", "crates/lance"] + +[workspace.package] +version = "0.1.0" +edition = "2021" + +[workspace.dependencies] +anyhow = "1" +async-trait = "0.1" +prost = "0.13" +prost-types = "0.13" +tokio = { version = "1", features = ["full"] } +tokio-stream = "0.1" +tonic = { version = "0.12", features = ["tls"] } +tonic-build = "0.12" +tracing = "0.1" +tracing-subscriber = { version = "0.3", features = ["env-filter"] } diff --git a/seaweed-worker/README.md b/seaweed-worker/README.md new file mode 100644 index 000000000..7ef6fbe47 --- /dev/null +++ b/seaweed-worker/README.md @@ -0,0 +1,47 @@ +# SeaweedFS Rust workers + +`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker +connects out to admin, announces the job types it can detect and execute, and +answers requests on that one stream. `weed worker -admin=host:23646` is the Go +implementation of it from outside the admin process. This workspace is the Rust +one. + + crates/core the contract: stream, handshake, heartbeat, registry, config forms + crates/lance maintenance jobs for Lance tables, and a binary + +`core` knows nothing about any job. A second worker is a new crate beside +`lance` that depends on it, not a fork of the protocol. + +## Running + + cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 + +The admin's *HTTP* address is what an operator has; the gRPC port is derived +from it the way the Go side does. Dialling the HTTP port fails as "frame with +invalid size", which reads like a protocol bug rather than a wrong port. + +## Credentials + +The worker holds none. It asks the namespace to describe a table with +`vend_credentials` and hands the `storage_options` that come back to lance. A +gateway without STS configured vends no credentials at all, so `--access-key` +and `--secret-key` supply a fallback; anything the namespace does vend wins over +them. + +## State + +All three jobs are implemented and tested end to end against a live gateway: + + compaction result: 12 fragments became 1 + reindex result: 512 uncovered rows became 0 + cleanup result: removed 14 versions and 24272 bytes + +`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a +live namespace and skips otherwise, the way the Go integration tests skip +without Docker. Each test seeds the table it needs, including building a vector +index and then appending rows outside it, so a run does not depend on what the +previous one left behind — the first version of these did, and quietly stopped +testing anything once it had done its job. + +The handshake, descriptor exchange and heartbeat work against a live admin, +which logs the worker connecting and prefetches all three descriptors. diff --git a/seaweed-worker/crates/core/Cargo.toml b/seaweed-worker/crates/core/Cargo.toml new file mode 100644 index 000000000..8627e8a95 --- /dev/null +++ b/seaweed-worker/crates/core/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "seaweed-worker-core" +version.workspace = true +edition.workspace = true +description = "SeaweedFS plugin.proto worker contract" + +[lib] +name = "seaweed_worker_core" + +[dependencies] +anyhow.workspace = true +async-trait.workspace = true +prost.workspace = true +prost-types.workspace = true +tokio.workspace = true +tokio-stream.workspace = true +tonic.workspace = true +tracing.workspace = true + +[build-dependencies] +tonic-build.workspace = true diff --git a/seaweed-worker/crates/core/build.rs b/seaweed-worker/crates/core/build.rs new file mode 100644 index 000000000..80bd222d2 --- /dev/null +++ b/seaweed-worker/crates/core/build.rs @@ -0,0 +1,12 @@ +fn main() -> Result<(), Box> { + // Compiled straight out of the Go tree, the way seaweed-volume already reads + // filer.proto, so the contract cannot drift from a vendored copy. + tonic_build::configure() + // The server half is only for tests, which stand up a fake admin. + .build_server(true) + .build_client(true) + .protoc_arg("--experimental_allow_proto3_optional") + .compile_protos(&["../../../weed/pb/plugin.proto"], &["../../../weed/pb/"])?; + println!("cargo:rerun-if-changed=../../../weed/pb/plugin.proto"); + Ok(()) +} diff --git a/seaweed-worker/crates/core/src/address.rs b/seaweed-worker/crates/core/src/address.rs new file mode 100644 index 000000000..a96fd72b7 --- /dev/null +++ b/seaweed-worker/crates/core/src/address.rs @@ -0,0 +1,73 @@ +//! SeaweedFS addresses the way the Go tree does. +//! +//! An operator gives a worker the admin's HTTP address, and the gRPC port is +//! derived from it rather than asked for separately. Dialling the HTTP port +//! instead fails as "frame with invalid size", which reads like a protocol bug +//! rather than a wrong port, so getting this right is worth its own module. +//! Mirrors pb.ServerToGrpcAddress in weed/pb/grpc_client_server.go. + +const GRPC_PORT_OFFSET: u16 = 10000; + +/// Converts `host:port` to the gRPC address, and accepts the explicit +/// `host:port.grpcPort` form the Go side also understands. +pub fn server_to_grpc_address(server: &str) -> Option { + let (host, port_part) = server.rsplit_once(':')?; + + // "port.grpcPort" states the gRPC port outright. + if let Some((_, grpc_port)) = port_part.split_once('.') { + if let Ok(port) = grpc_port.parse::() { + return Some(join_host_port(host, port)); + } + } + + let port: u16 = port_part.parse().ok()?; + Some(join_host_port(host, port.checked_add(GRPC_PORT_OFFSET)?)) +} + +fn join_host_port(host: &str, port: u16) -> String { + // An IPv6 literal has to keep its brackets or the port reads as part of it. + if host.contains(':') && !host.starts_with('[') { + format!("[{host}]:{port}") + } else { + format!("{host}:{port}") + } +} + +#[cfg(test)] +mod tests { + use super::server_to_grpc_address; + + #[test] + fn derives_the_grpc_port() { + assert_eq!( + server_to_grpc_address("localhost:23646").as_deref(), + Some("localhost:33646") + ); + assert_eq!( + server_to_grpc_address("127.0.0.1:9333").as_deref(), + Some("127.0.0.1:19333") + ); + } + + #[test] + fn honours_an_explicit_grpc_port() { + assert_eq!( + server_to_grpc_address("localhost:23646.33999").as_deref(), + Some("localhost:33999") + ); + } + + #[test] + fn brackets_ipv6_literals() { + assert_eq!( + server_to_grpc_address("::1:23646").as_deref(), + Some("[::1]:33646") + ); + } + + #[test] + fn rejects_what_it_cannot_parse() { + assert!(server_to_grpc_address("localhost").is_none()); + assert!(server_to_grpc_address("localhost:notaport").is_none()); + } +} diff --git a/seaweed-worker/crates/core/src/config.rs b/seaweed-worker/crates/core/src/config.rs new file mode 100644 index 000000000..c8fcb726d --- /dev/null +++ b/seaweed-worker/crates/core/src/config.rs @@ -0,0 +1,49 @@ +use std::time::Duration; + +/// How one worker process connects and how much work it will take on. +#[derive(Clone, Debug)] +pub struct WorkerOptions { + /// Admin gRPC address, e.g. "localhost:23646". + pub admin_address: String, + pub worker_id: String, + pub worker_version: String, + /// Advertised address; empty when the worker takes no inbound connections. + pub worker_address: String, + pub heartbeat_interval: Duration, + pub reconnect_delay: Duration, + pub max_detection_concurrency: i32, + pub max_execution_concurrency: i32, + /// mTLS for the control stream, mirroring the Go worker's `[grpc.worker]` + /// section of security.toml. None means plaintext, which is the default the + /// Go worker also takes when no certificates are configured. + pub tls: Option, +} + +/// Certificates for the control stream. All three are required together: the +/// cluster's gRPC TLS is mutual, so a CA without a client identity gets refused +/// by admin rather than falling back to one-way TLS. +#[derive(Clone, Debug)] +pub struct TlsOptions { + pub ca_path: String, + pub client_cert_path: String, + pub client_key_path: String, + /// Name to verify the server certificate against, when the address a worker + /// dials is not the name the certificate carries. + pub server_name: Option, +} + +impl Default for WorkerOptions { + fn default() -> Self { + Self { + admin_address: "localhost:23646".to_string(), + worker_id: String::new(), + worker_version: env!("CARGO_PKG_VERSION").to_string(), + worker_address: String::new(), + heartbeat_interval: Duration::from_secs(10), + reconnect_delay: Duration::from_secs(5), + max_detection_concurrency: 1, + max_execution_concurrency: 1, + tls: None, + } + } +} diff --git a/seaweed-worker/crates/core/src/config_form.rs b/seaweed-worker/crates/core/src/config_form.rs new file mode 100644 index 000000000..09106f679 --- /dev/null +++ b/seaweed-worker/crates/core/src/config_form.rs @@ -0,0 +1,95 @@ +//! Builders for the config forms a worker returns in its JobTypeDescriptor. +//! +//! Admin renders these into the job's settings page, so a worker written in any +//! language gets a UI without touching Go or templ. That only holds if the +//! field types and defaults are right, which is why they are built here rather +//! than spelled out at each call site. + +use std::collections::HashMap; + +use crate::pb::{ + config_value::Kind, ConfigField, ConfigFieldType, ConfigForm, ConfigSection, ConfigValue, +}; + +pub fn int_value(value: i64) -> ConfigValue { + ConfigValue { + kind: Some(Kind::Int64Value(value)), + } +} + +pub fn bool_value(value: bool) -> ConfigValue { + ConfigValue { + kind: Some(Kind::BoolValue(value)), + } +} + +pub fn string_value(value: impl Into) -> ConfigValue { + ConfigValue { + kind: Some(Kind::StringValue(value.into())), + } +} + +/// Reads an integer a request carried, falling back when admin sent nothing. +pub fn int_or(values: &HashMap, name: &str, fallback: i64) -> i64 { + match values.get(name).and_then(|v| v.kind.as_ref()) { + Some(Kind::Int64Value(value)) => *value, + Some(Kind::DoubleValue(value)) => *value as i64, + _ => fallback, + } +} + +pub fn bool_or(values: &HashMap, name: &str, fallback: bool) -> bool { + match values.get(name).and_then(|v| v.kind.as_ref()) { + Some(Kind::BoolValue(value)) => *value, + _ => fallback, + } +} + +pub fn string_or(values: &HashMap, name: &str, fallback: &str) -> String { + match values.get(name).and_then(|v| v.kind.as_ref()) { + Some(Kind::StringValue(value)) if !value.is_empty() => value.clone(), + _ => fallback.to_string(), + } +} + +pub fn number_field(name: &str, label: &str, description: &str, min: i64, max: i64) -> ConfigField { + ConfigField { + name: name.to_string(), + label: label.to_string(), + description: description.to_string(), + field_type: ConfigFieldType::Int64 as i32, + min_value: Some(int_value(min)), + max_value: Some(int_value(max)), + ..Default::default() + } +} + +pub fn bool_field(name: &str, label: &str, description: &str) -> ConfigField { + ConfigField { + name: name.to_string(), + label: label.to_string(), + description: description.to_string(), + field_type: ConfigFieldType::Bool as i32, + ..Default::default() + } +} + +pub fn form( + form_id: &str, + title: &str, + fields: Vec, + defaults: HashMap, +) -> ConfigForm { + ConfigForm { + form_id: form_id.to_string(), + title: title.to_string(), + description: String::new(), + sections: vec![ConfigSection { + section_id: format!("{form_id}-main"), + title: title.to_string(), + description: String::new(), + fields, + }], + default_values: defaults, + } +} diff --git a/seaweed-worker/crates/core/src/lib.rs b/seaweed-worker/crates/core/src/lib.rs new file mode 100644 index 000000000..315d87c59 --- /dev/null +++ b/seaweed-worker/crates/core/src/lib.rs @@ -0,0 +1,24 @@ +//! The SeaweedFS plugin worker contract, in Rust. +//! +//! `weed/pb/plugin.proto` is a language-agnostic gRPC stream: a worker connects +//! out to admin, announces the job types it can detect and execute, and then +//! answers requests on that one stream. `weed worker -admin=host:23646` is the +//! Go implementation of the same contract from outside the admin process; this +//! is the Rust one, and it knows nothing about any particular job. + +pub mod address; +pub mod config; +pub mod config_form; +pub mod registry; +pub mod senders; +pub mod stream; + +/// Generated plugin.proto types. +pub mod pb { + tonic::include_proto!("plugin"); +} + +pub use config::{TlsOptions, WorkerOptions}; +pub use registry::{JobHandler, Preview, PreviewProvider, Registry}; +pub use senders::{DetectionSender, ExecutionSender}; +pub use stream::run; diff --git a/seaweed-worker/crates/core/src/registry.rs b/seaweed-worker/crates/core/src/registry.rs new file mode 100644 index 000000000..897aa0a42 --- /dev/null +++ b/seaweed-worker/crates/core/src/registry.rs @@ -0,0 +1,89 @@ +use std::collections::HashMap; +use std::sync::Arc; + +use anyhow::Result; +use async_trait::async_trait; + +use crate::pb::{ExecuteJobRequest, JobTypeCapability, JobTypeDescriptor, RunDetectionRequest}; +use crate::senders::{DetectionSender, ExecutionSender}; + +/// One job type, worker side. Mirrors the Go JobHandler interface in +/// weed/plugin/worker/worker.go so the two stay readable against each other. +#[async_trait] +pub trait JobHandler: Send + Sync { + fn capability(&self) -> JobTypeCapability; + /// The descriptor admin renders as this job's settings page. + fn descriptor(&self) -> JobTypeDescriptor; + async fn detect( + &self, + request: &RunDetectionRequest, + sender: &dyn DetectionSender, + ) -> Result<()>; + async fn execute( + &self, + request: &ExecuteJobRequest, + sender: &dyn ExecutionSender, + ) -> Result<()>; +} + +/// Sample rows of one object, already rendered as text. The worker formats +/// them because it is the only side that knows the object's types. +pub struct Preview { + pub columns: Vec, + pub rows: Vec>, + /// Rows in the object, which is not the number sampled. + pub total_rows: i64, +} + +/// Reads sample rows of a format admin cannot parse itself. +/// +/// This is deliberately not a JobHandler: a preview is answered while someone +/// waits on a page, so it neither schedules nor reports progress. +#[async_trait] +pub trait PreviewProvider: Send + Sync { + /// The format this provider reads, matched case-insensitively against what + /// the catalog recorded. + fn format(&self) -> &str; + async fn preview(&self, object_id: &[String], row_limit: usize) -> Result; +} + +/// The handlers one worker process serves. A process may serve several job +/// types, which is why WorkerHello carries a list of capabilities. +#[derive(Default, Clone)] +pub struct Registry { + handlers: HashMap>, + previews: HashMap>, +} + +impl Registry { + pub fn new() -> Self { + Self::default() + } + + pub fn register(mut self, handler: Arc) -> Self { + self.handlers.insert(handler.capability().job_type, handler); + self + } + + pub fn with_preview(mut self, provider: Arc) -> Self { + self.previews + .insert(provider.format().to_ascii_uppercase(), provider); + self + } + + pub fn get(&self, job_type: &str) -> Option> { + self.handlers.get(job_type).cloned() + } + + pub fn preview_provider(&self, format: &str) -> Option> { + self.previews.get(&format.to_ascii_uppercase()).cloned() + } + + pub fn capabilities(&self) -> Vec { + self.handlers.values().map(|h| h.capability()).collect() + } + + pub fn is_empty(&self) -> bool { + self.handlers.is_empty() + } +} diff --git a/seaweed-worker/crates/core/src/senders.rs b/seaweed-worker/crates/core/src/senders.rs new file mode 100644 index 000000000..70f067453 --- /dev/null +++ b/seaweed-worker/crates/core/src/senders.rs @@ -0,0 +1,75 @@ +use anyhow::Result; +use tokio::sync::mpsc; + +use crate::pb::{ + worker_to_admin_message::Body, ActivityEvent, DetectionComplete, DetectionProposals, + JobCompleted, JobProgressUpdate, WorkerObservations, WorkerToAdminMessage, +}; + +/// Replies to one detection request. +pub trait DetectionSender: Send + Sync { + fn send_proposals(&self, proposals: DetectionProposals) -> Result<()>; + fn send_complete(&self, complete: DetectionComplete) -> Result<()>; + fn send_activity(&self, activity: ActivityEvent) -> Result<()>; + /// Reports what the worker saw while deciding. Admin caches the last one + /// per object and serves it back for display; nothing is scheduled from it. + fn send_observations(&self, observations: WorkerObservations) -> Result<()>; +} + +/// Replies to one execution request. +pub trait ExecutionSender: Send + Sync { + fn send_progress(&self, progress: JobProgressUpdate) -> Result<()>; + fn send_completed(&self, completed: JobCompleted) -> Result<()>; +} + +/// Both senders write to the single outbound stream, so they share one channel. +#[derive(Clone)] +pub struct StreamSender { + worker_id: String, + tx: mpsc::UnboundedSender, +} + +impl StreamSender { + pub fn new(worker_id: String, tx: mpsc::UnboundedSender) -> Self { + Self { worker_id, tx } + } + + pub fn send(&self, body: Body) -> Result<()> { + self.tx.send(WorkerToAdminMessage { + worker_id: self.worker_id.clone(), + sent_at: Some(std::time::SystemTime::now().into()), + body: Some(body), + })?; + Ok(()) + } +} + +impl DetectionSender for StreamSender { + fn send_proposals(&self, proposals: DetectionProposals) -> Result<()> { + self.send(Body::DetectionProposals(proposals)) + } + + fn send_complete(&self, complete: DetectionComplete) -> Result<()> { + self.send(Body::DetectionComplete(complete)) + } + + fn send_activity(&self, _activity: ActivityEvent) -> Result<()> { + // Activity rides inside progress and completion messages rather than + // being a body of its own, so there is nothing to send on its own here. + Ok(()) + } + + fn send_observations(&self, observations: WorkerObservations) -> Result<()> { + self.send(Body::Observations(observations)) + } +} + +impl ExecutionSender for StreamSender { + fn send_progress(&self, progress: JobProgressUpdate) -> Result<()> { + self.send(Body::JobProgressUpdate(progress)) + } + + fn send_completed(&self, completed: JobCompleted) -> Result<()> { + self.send(Body::JobCompleted(completed)) + } +} diff --git a/seaweed-worker/crates/core/src/stream.rs b/seaweed-worker/crates/core/src/stream.rs new file mode 100644 index 000000000..054731890 --- /dev/null +++ b/seaweed-worker/crates/core/src/stream.rs @@ -0,0 +1,408 @@ +use std::sync::Arc; +use std::time::Duration; + +use anyhow::{anyhow, Context, Result}; +use tokio::sync::{mpsc, Semaphore}; +use tokio_stream::wrappers::UnboundedReceiverStream; +use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity}; +use tracing::{info, warn}; + +use crate::config::WorkerOptions; +use crate::pb::{ + admin_to_worker_message::Body as AdminBody, + plugin_control_service_client::PluginControlServiceClient, + worker_to_admin_message::Body as WorkerBody, ConfigSchemaResponse, ExecuteJobRequest, + JobCompleted, ObjectPreviewResponse, PreviewRow, RequestObjectPreview, RunDetectionRequest, + RunningWork, WorkerHeartbeat, WorkerHello, +}; +use crate::registry::Registry; +use crate::senders::StreamSender; + +/// The protocol version this worker speaks, sent in WorkerHello. +const PROTOCOL_VERSION: &str = "1"; + +/// Connect to admin and serve the registry until the context is cancelled, +/// reconnecting on failure. The stream is the only channel: everything admin +/// asks for and everything the worker reports flows through it. +pub async fn run(options: WorkerOptions, registry: Registry) -> Result<()> { + if registry.is_empty() { + return Err(anyhow!("no job handlers registered")); + } + if options.max_detection_concurrency < 1 || options.max_execution_concurrency < 1 { + return Err(anyhow!( + "concurrency limits must be at least 1, got detection={} execution={}", + options.max_detection_concurrency, + options.max_execution_concurrency + )); + } + let slots = Slots::new(&options); + loop { + match serve_once(&options, ®istry, &slots).await { + Err(err) => warn!("worker stream ended: {err:#}"), + // Admin asked this worker to stop, so stop. Reconnecting here would + // make shutdown impossible: the worker would log back in. + Ok(Outcome::ShutdownRequested) => return Ok(()), + // Admin closing a healthy stream is not an error, but reconnecting + // in silence hides the reason - two workers sharing an id evict + // each other and produce nothing but a login every few seconds. + Ok(Outcome::StreamClosed) => warn!( + "admin closed the stream; reconnecting in {:?}. If this repeats, check for \ + another worker using the id {}", + options.reconnect_delay, options.worker_id + ), + } + tokio::time::sleep(options.reconnect_delay).await; + } +} + +/// Why a stream ended. Only one of these means "do not come back". +enum Outcome { + StreamClosed, + ShutdownRequested, +} + +/// The capacity this worker advertises in WorkerHello. Admin schedules against +/// those numbers, so the worker has to actually hold to them - and the heartbeat +/// has to report what is in use, or admin is scheduling blind. +#[derive(Clone)] +struct Slots { + detection: Arc, + execution: Arc, + detection_total: i32, + execution_total: i32, +} + +impl Slots { + fn new(options: &WorkerOptions) -> Self { + Self { + detection: Arc::new(Semaphore::new(options.max_detection_concurrency as usize)), + execution: Arc::new(Semaphore::new(options.max_execution_concurrency as usize)), + detection_total: options.max_detection_concurrency, + execution_total: options.max_execution_concurrency, + } + } + + fn detection_used(&self) -> i32 { + self.detection_total - self.detection.available_permits() as i32 + } + + fn execution_used(&self) -> i32 { + self.execution_total - self.execution.available_permits() as i32 + } +} + +/// Dials admin, over mTLS when certificates are configured. Plaintext is the +/// default and is fine over loopback; anything else carries preview rows and +/// execution commands in the clear, and a cluster with grpc TLS on refuses the +/// connection anyway. +async fn connect(options: &WorkerOptions, grpc_address: &str) -> Result { + let Some(tls) = options.tls.as_ref() else { + return Ok(Channel::from_shared(format!("http://{grpc_address}"))? + .connect_timeout(Duration::from_secs(10)) + .connect() + .await?); + }; + + let ca = tokio::fs::read(&tls.ca_path) + .await + .with_context(|| format!("read CA certificate {}", tls.ca_path))?; + let cert = tokio::fs::read(&tls.client_cert_path) + .await + .with_context(|| format!("read client certificate {}", tls.client_cert_path))?; + let key = tokio::fs::read(&tls.client_key_path) + .await + .with_context(|| format!("read client key {}", tls.client_key_path))?; + + let mut config = ClientTlsConfig::new() + .ca_certificate(Certificate::from_pem(ca)) + .identity(Identity::from_pem(cert, key)); + if let Some(server_name) = tls.server_name.as_ref() { + config = config.domain_name(server_name.clone()); + } + + Ok(Channel::from_shared(format!("https://{grpc_address}"))? + .tls_config(config)? + .connect_timeout(Duration::from_secs(10)) + .connect() + .await?) +} + +async fn serve_once( + options: &WorkerOptions, + registry: &Registry, + slots: &Slots, +) -> Result { + // Operators give the admin's HTTP address; the gRPC port is derived, the + // same way the Go worker does it. + let grpc_address = crate::address::server_to_grpc_address(&options.admin_address) + .ok_or_else(|| anyhow!("cannot parse admin address {}", options.admin_address))?; + let channel = connect(options, &grpc_address).await?; + let mut client = PluginControlServiceClient::new(channel); + + let (tx, rx) = mpsc::unbounded_channel(); + let sender = StreamSender::new(options.worker_id.clone(), tx); + + sender.send(WorkerBody::Hello(WorkerHello { + worker_id: options.worker_id.clone(), + worker_instance_id: options.worker_id.clone(), + address: options.worker_address.clone(), + worker_version: options.worker_version.clone(), + protocol_version: PROTOCOL_VERSION.to_string(), + capabilities: registry.capabilities(), + metadata: Default::default(), + }))?; + + let mut inbound = client + .worker_stream(UnboundedReceiverStream::new(rx)) + .await? + .into_inner(); + + let heartbeat = spawn_heartbeat(sender.clone(), options.clone(), slots.clone()); + + while let Some(message) = inbound.message().await? { + let request_id = message.request_id.clone(); + match message.body { + Some(AdminBody::Hello(hello)) => { + if !hello.accepted { + return Err(anyhow!("admin rejected this worker: {}", hello.message)); + } + info!( + "connected to admin at {} ({})", + options.admin_address, grpc_address + ); + } + Some(AdminBody::RequestConfigSchema(request)) => { + let response = match registry.get(&request.job_type) { + Some(handler) => ConfigSchemaResponse { + request_id: request_id.clone(), + job_type: request.job_type.clone(), + success: true, + error_message: String::new(), + job_type_descriptor: Some(handler.descriptor()), + }, + None => ConfigSchemaResponse { + request_id: request_id.clone(), + job_type: request.job_type.clone(), + success: false, + error_message: format!("unknown job type {}", request.job_type), + job_type_descriptor: None, + }, + }; + sender.send(WorkerBody::ConfigSchemaResponse(response))?; + } + Some(AdminBody::RequestObjectPreview(request)) => { + spawn_preview( + registry.clone(), + sender.clone(), + request_id.clone(), + request, + ); + } + Some(AdminBody::RunDetectionRequest(request)) => { + spawn_detection(registry.clone(), sender.clone(), slots.clone(), request); + } + Some(AdminBody::ExecuteJobRequest(request)) => { + spawn_execution(registry.clone(), sender.clone(), slots.clone(), request); + } + Some(AdminBody::CancelRequest(request)) => { + // Cancellation needs a per-request handle to be honoured; until + // then say so rather than silently continuing to run the job. + warn!( + "cancel requested for {} ({}) but is not implemented", + request.target_id, request.reason + ); + } + Some(AdminBody::Shutdown(shutdown)) => { + info!("admin asked this worker to stop: {}", shutdown.reason); + heartbeat.abort(); + return Ok(Outcome::ShutdownRequested); + } + None => {} + } + } + + heartbeat.abort(); + Ok(Outcome::StreamClosed) +} + +/// Answers one preview request off the stream loop. Reading rows takes as long +/// as it takes, and the stream has heartbeats to keep up meanwhile. +fn spawn_preview( + registry: Registry, + sender: StreamSender, + request_id: String, + request: RequestObjectPreview, +) { + tokio::spawn(async move { + let limit = request.row_limit.max(1) as usize; + let response = match registry.preview_provider(&request.format) { + None => ObjectPreviewResponse { + request_id, + success: false, + error_message: format!("this worker does not read {} objects", request.format), + ..Default::default() + }, + Some(provider) => match provider.preview(&request.object_id, limit).await { + Ok(preview) => ObjectPreviewResponse { + request_id, + success: true, + error_message: String::new(), + columns: preview.columns, + rows: preview + .rows + .into_iter() + .map(|values| PreviewRow { values }) + .collect(), + total_rows: preview.total_rows, + }, + Err(err) => ObjectPreviewResponse { + request_id, + success: false, + error_message: format!("{err:#}"), + ..Default::default() + }, + }, + }; + let _ = sender.send(WorkerBody::ObjectPreviewResponse(response)); + }); +} + +fn spawn_heartbeat( + sender: StreamSender, + options: WorkerOptions, + slots: Slots, +) -> tokio::task::JoinHandle<()> { + // The handle has to be the heartbeat's own, or aborting it aborts nothing + // and every reconnect leaves another ticker running. + tokio::spawn(async move { + let mut ticker = tokio::time::interval(options.heartbeat_interval); + loop { + ticker.tick().await; + let beat = WorkerHeartbeat { + worker_id: options.worker_id.clone(), + running_work: Vec::::new(), + detection_slots_used: slots.detection_used(), + detection_slots_total: slots.detection_total, + execution_slots_used: slots.execution_used(), + execution_slots_total: slots.execution_total, + queued_jobs_by_type: Default::default(), + metadata: Default::default(), + }; + if sender.send(WorkerBody::Heartbeat(beat)).is_err() { + return; + } + } + }) +} + +fn spawn_detection( + registry: Registry, + sender: StreamSender, + slots: Slots, + request: RunDetectionRequest, +) { + tokio::spawn(async move { + let Some(handler) = registry.get(&request.job_type) else { + return; + }; + // Held until the sweep finishes, so the worker keeps to the capacity it + // advertised and the heartbeat reports the truth while it works. + let _permit = slots.detection.acquire().await; + if let Err(err) = handler.detect(&request, &sender).await { + warn!("detection for {} failed: {err:#}", request.job_type); + let _ = sender.send(WorkerBody::DetectionComplete( + crate::pb::DetectionComplete { + request_id: request.request_id.clone(), + job_type: request.job_type.clone(), + success: false, + error_message: format!("{err:#}"), + total_proposals: 0, + }, + )); + } + }); +} + +fn spawn_execution( + registry: Registry, + sender: StreamSender, + slots: Slots, + request: ExecuteJobRequest, +) { + tokio::spawn(async move { + let _permit = slots.execution.acquire().await; + let job_type = request + .job + .as_ref() + .map(|job| job.job_type.clone()) + .unwrap_or_default(); + let job_id = request + .job + .as_ref() + .map(|job| job.job_id.clone()) + .unwrap_or_default(); + let Some(handler) = registry.get(&job_type) else { + return; + }; + if let Err(err) = handler.execute(&request, &sender).await { + warn!("job {job_id} failed: {err:#}"); + let _ = sender.send(WorkerBody::JobCompleted(JobCompleted { + request_id: request.request_id.clone(), + job_id, + job_type, + success: false, + error_message: format!("{err:#}"), + ..Default::default() + })); + } + }); +} + +#[cfg(test)] +mod tests { + use super::*; + + fn options(detection: i32, execution: i32) -> WorkerOptions { + WorkerOptions { + max_detection_concurrency: detection, + max_execution_concurrency: execution, + ..Default::default() + } + } + + // The heartbeat is admin's only view of how busy this worker is; before the + // permits existed it reported zero however much was running. + #[tokio::test] + async fn slots_report_what_is_held() { + let slots = Slots::new(&options(2, 3)); + assert_eq!(slots.detection_used(), 0); + assert_eq!(slots.execution_used(), 0); + + let held = slots.detection.acquire().await.unwrap(); + assert_eq!(slots.detection_used(), 1); + assert_eq!(slots.execution_used(), 0, "the two do not share capacity"); + + drop(held); + assert_eq!(slots.detection_used(), 0); + } + + // A limit of one means the second request waits, rather than running anyway + // as it did when every request simply spawned a task. + #[tokio::test] + async fn a_full_lane_makes_the_next_request_wait() { + let slots = Slots::new(&options(1, 1)); + let held = slots.execution.clone().acquire_owned().await.unwrap(); + assert_eq!(slots.execution_used(), 1); + + let waiter = tokio::spawn({ + let execution = slots.execution.clone(); + async move { execution.acquire_owned().await.unwrap() } + }); + tokio::task::yield_now().await; + assert!(!waiter.is_finished(), "the second request must not start"); + + drop(held); + let _second = waiter.await.expect("the waiter should be handed the slot"); + assert_eq!(slots.execution_used(), 1); + } +} diff --git a/seaweed-worker/crates/lance/Cargo.toml b/seaweed-worker/crates/lance/Cargo.toml new file mode 100644 index 000000000..6b85eed76 --- /dev/null +++ b/seaweed-worker/crates/lance/Cargo.toml @@ -0,0 +1,39 @@ +[package] +name = "weed-lance-worker" +version.workspace = true +edition.workspace = true +description = "SeaweedFS maintenance worker for Lance tables" + +[lib] +name = "weed_lance_worker" + +[[bin]] +name = "weed-lance-worker" +path = "src/main.rs" + +[dependencies] +seaweed-worker-core = { path = "../core" } +# Only the S3 backend: the other object stores lance enables by default are +# build time this worker never spends. +lance = { version = "10", default-features = false, features = ["aws"] } +lance-index = "10" +arrow-schema = "58" +arrow-cast = "58" +chrono = "0.4" +futures = "0.3" +anyhow.workspace = true +async-trait.workspace = true +clap = { version = "4", features = ["derive", "env"] } +reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } +serde = { version = "1", features = ["derive"] } +serde_json = "1" +tokio.workspace = true +tracing.workspace = true +tracing-subscriber.workspace = true + +[dev-dependencies] +tokio = { workspace = true } +arrow-array = "58" +arrow-schema = "58" +arrow-cast = "58" +lance-linalg = "10" diff --git a/seaweed-worker/crates/lance/src/catalog/mod.rs b/seaweed-worker/crates/lance/src/catalog/mod.rs new file mode 100644 index 000000000..002a74c36 --- /dev/null +++ b/seaweed-worker/crates/lance/src/catalog/mod.rs @@ -0,0 +1,10 @@ +//! How the worker finds Lance tables and gets at their bytes. +//! +//! It goes through the Lance namespace rather than the filer: the namespace is +//! the catalog of record, it already knows which tables are Lance, and asking it +//! to describe a table with vend_credentials is how the worker gets storage +//! credentials without holding any of its own. + +pub mod namespace; + +pub use namespace::{parse_id, NamespaceClient, TableDescription}; diff --git a/seaweed-worker/crates/lance/src/catalog/namespace.rs b/seaweed-worker/crates/lance/src/catalog/namespace.rs new file mode 100644 index 000000000..c026ae2c6 --- /dev/null +++ b/seaweed-worker/crates/lance/src/catalog/namespace.rs @@ -0,0 +1,129 @@ +use std::collections::HashMap; + +use std::time::Duration; + +use anyhow::{Context, Result}; +use serde::Deserialize; + +/// The delimiter the Lance namespace joins identifier parts with. +const DELIMITER: &str = "$"; + +#[derive(Debug, Deserialize)] +pub struct TableDescription { + pub location: String, + #[serde(default)] + pub storage_options: HashMap, + #[serde(default)] + pub managed_versioning: bool, +} + +#[derive(Debug, Deserialize)] +struct ListNamespacesResponse { + #[serde(default)] + namespaces: Vec, +} + +#[derive(Debug, Deserialize)] +struct ListTablesResponse { + #[serde(default)] + tables: Vec, +} + +/// A thin client for the operations this worker needs. It deliberately does not +/// wrap the whole spec: a maintenance worker lists, describes, and commits. +pub struct NamespaceClient { + base_url: String, + http: reqwest::Client, +} + +/// A namespace call that has not answered by now is not going to. Without this +/// a gateway that accepts the connection and then goes quiet holds a detection +/// slot open forever, and the sweep never finishes. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(30); +const CONNECT_TIMEOUT: Duration = Duration::from_secs(10); + +impl NamespaceClient { + pub fn new(base_url: impl Into) -> Self { + let http = reqwest::Client::builder() + .timeout(REQUEST_TIMEOUT) + .connect_timeout(CONNECT_TIMEOUT) + .build() + // The builder only fails on a bad TLS backend, which would break + // every call anyway; a client with no timeouts is worse than a panic + // at startup, so keep the default only as a last resort. + .unwrap_or_else(|err| { + tracing::warn!("falling back to an untimed HTTP client: {err}"); + reqwest::Client::new() + }); + Self { + base_url: base_url.into().trim_end_matches('/').to_string(), + http, + } + } + + /// Every table the namespace holds, as delimiter-joined identifiers. + pub async fn list_all_tables(&self) -> Result> { + let url = format!("{}/v1/table", self.base_url); + let response: ListTablesResponse = self + .http + .get(&url) + .send() + .await + .context("list tables")? + .error_for_status()? + .json() + .await?; + Ok(response.tables) + } + + /// Child namespaces of `id`; the root lists table buckets. + pub async fn list_namespaces(&self, id: &[String]) -> Result> { + let url = format!("{}/v1/namespace/{}/list", self.base_url, encode_id(id)); + let response: ListNamespacesResponse = self + .http + .get(&url) + .send() + .await + .context("list namespaces")? + .error_for_status()? + .json() + .await?; + Ok(response.namespaces) + } + + /// Resolve a table to a location and the credentials to reach it. The + /// credentials expire, so a long compaction re-describes rather than + /// carrying one set for the whole job. + pub async fn describe_table(&self, id: &[String]) -> Result { + let url = format!("{}/v1/table/{}/describe", self.base_url, encode_id(id)); + let body = serde_json::json!({ "id": id, "vend_credentials": true }); + let description: TableDescription = self + .http + .post(&url) + .json(&body) + .send() + .await + .context("describe table")? + .error_for_status()? + .json() + .await?; + Ok(description) + } +} + +fn encode_id(id: &[String]) -> String { + if id.is_empty() { + DELIMITER.to_string() + } else { + id.join(DELIMITER) + } +} + +/// Splits a delimiter-joined identifier back into parts. +pub fn parse_id(encoded: &str) -> Vec { + encoded + .split(DELIMITER) + .filter(|part| !part.is_empty()) + .map(|part| part.to_string()) + .collect() +} diff --git a/seaweed-worker/crates/lance/src/dataset.rs b/seaweed-worker/crates/lance/src/dataset.rs new file mode 100644 index 000000000..74effcc92 --- /dev/null +++ b/seaweed-worker/crates/lance/src/dataset.rs @@ -0,0 +1,98 @@ +//! Opening a Lance dataset with credentials the namespace vended. +//! +//! The worker holds no storage credentials of its own: it asks the namespace to +//! describe a table with `vend_credentials`, and the `storage_options` that come +//! back are handed to lance as-is. They expire, so a job that runs longer than +//! their lifetime re-describes rather than carrying one set throughout. + +use std::collections::HashMap; + +use anyhow::{Context, Result}; +use lance::dataset::builder::DatasetBuilder; +use lance::dataset::Dataset; + +use crate::catalog::{NamespaceClient, TableDescription}; + +/// A table the worker is about to work on. +pub struct OpenTable { + pub id: Vec, + pub location: String, + pub dataset: Dataset, +} + +/// What detection needs to decide whether a table is worth a job. Reading it +/// opens the dataset but touches no data files. +pub struct TableStats { + pub fragments: usize, + pub version: u64, + pub total_versions: usize, + pub rows: usize, + /// The Arrow schema as JSON, which is the only description of this table + /// anything outside the format can produce. + pub schema: Option, +} + +/// Storage options an operator supplies for deployments that vend none. +pub type FallbackOptions = HashMap; + +/// Resolve a table through the namespace and open it. +pub async fn open( + client: &NamespaceClient, + id: &[String], + fallback: &FallbackOptions, +) -> Result { + let description = client.describe_table(id).await?; + let dataset = open_at(&description, fallback).await?; + Ok(OpenTable { + id: id.to_vec(), + location: description.location, + dataset, + }) +} + +/// The namespace vends object_store's own option names, so they pass straight +/// through. What it vends always wins: the fallback exists because a deployment +/// without STS vends no credentials at all, and then the worker has no other way +/// to reach the data. +async fn open_at(description: &TableDescription, fallback: &FallbackOptions) -> Result { + let mut options: HashMap = fallback.clone(); + options.extend(description.storage_options.clone()); + DatasetBuilder::from_uri(&description.location) + .with_storage_options(options) + .load() + .await + .with_context(|| format!("open lance dataset at {}", description.location)) +} + +impl OpenTable { + pub async fn stats(&self) -> Result { + let versions = self.dataset.versions().await?; + Ok(TableStats { + fragments: self.dataset.get_fragments().len(), + version: self.dataset.version().version, + total_versions: versions.len(), + rows: self.dataset.count_rows(None).await.unwrap_or(0), + schema: schema_json(&self.dataset), + }) + } +} + +/// Renders the dataset's schema as JSON. Best effort: a schema that will not +/// serialise is not a reason to fail a maintenance sweep. +fn schema_json(dataset: &Dataset) -> Option { + let arrow: arrow_schema::Schema = dataset.schema().into(); + serde_json::to_string( + &arrow + .fields() + .iter() + .map(|f| { + serde_json::json!({ + "name": f.name(), + "type": f.data_type().to_string(), + "nullable": f.is_nullable(), + }) + }) + .collect::>(), + ) + .ok() +} diff --git a/seaweed-worker/crates/lance/src/jobs/cleanup.rs b/seaweed-worker/crates/lance/src/jobs/cleanup.rs new file mode 100644 index 000000000..b24e4642a --- /dev/null +++ b/seaweed-worker/crates/lance/src/jobs/cleanup.rs @@ -0,0 +1,315 @@ +use std::collections::HashMap; + +use anyhow::{anyhow, Context, Result}; +use async_trait::async_trait; +use chrono::{Duration, Utc}; +use lance::dataset::cleanup::{cleanup_old_versions, CleanupPolicy}; +use seaweed_worker_core::config_form::{form, int_or, int_value, number_field}; +use seaweed_worker_core::pb::{ + ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted, + JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor, + RunDetectionRequest, +}; +use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler}; +use tracing::warn; + +use crate::catalog::{parse_id, NamespaceClient}; +use crate::dataset; +use crate::jobs::{clamp, string_list, table_id}; + +pub const JOB_TYPE: &str = "lance_cleanup_versions"; + +const DEFAULT_RETAIN_HOURS: i64 = 168; +const DEFAULT_MIN_VERSIONS: i64 = 5; + +// The ranges the form offers. Duration::hours also panics far outside this one. +const MAX_RETAIN_HOURS: i64 = 8760; +const MAX_MIN_VERSIONS: i64 = 1000; + +/// Lance keeps every version until something removes it. Lance can also do this +/// itself through auto-cleanup, so this job is for deployments that would rather +/// the cluster owned the policy than each writer. +pub struct CleanupVersionsHandler { + namespace_url: String, + fallback: dataset::FallbackOptions, +} + +impl CleanupVersionsHandler { + pub fn new(namespace_url: String) -> Self { + Self { + namespace_url, + fallback: dataset::FallbackOptions::new(), + } + } + + pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self { + self.fallback = fallback; + self + } + + fn client(&self) -> NamespaceClient { + NamespaceClient::new(self.namespace_url.clone()) + } +} + +/// The oldest version that may be removed while still leaving `min_versions` +/// behind, or None when the table has no more than the floor. +async fn version_floor(table: &dataset::OpenTable, min_versions: usize) -> Result> { + let mut versions: Vec = table + .dataset + .versions() + .await? + .iter() + .map(|version| version.version) + .collect(); + if versions.len() <= min_versions { + return Ok(None); + } + versions.sort_unstable(); + Ok(Some(versions[versions.len() - min_versions])) +} + +#[async_trait] +impl JobHandler for CleanupVersionsHandler { + fn capability(&self) -> JobTypeCapability { + JobTypeCapability { + job_type: JOB_TYPE.to_string(), + can_detect: true, + can_execute: true, + max_detection_concurrency: 1, + max_execution_concurrency: 1, + display_name: "Lance Version Cleanup".to_string(), + description: "Remove old Lance versions and the files only they referenced".to_string(), + weight: 10, + } + } + + fn descriptor(&self) -> JobTypeDescriptor { + let mut defaults: HashMap = HashMap::new(); + defaults.insert("retain_hours".to_string(), int_value(DEFAULT_RETAIN_HOURS)); + defaults.insert( + "min_versions_to_keep".to_string(), + int_value(DEFAULT_MIN_VERSIONS), + ); + + JobTypeDescriptor { + job_type: JOB_TYPE.to_string(), + display_name: "Lance Version Cleanup".to_string(), + description: "Age out Lance versions the table no longer needs".to_string(), + icon: "fas fa-broom".to_string(), + descriptor_version: 1, + worker_config_form: Some(form( + "lance-cleanup-worker", + "Version cleanup", + vec![ + number_field( + "retain_hours", + "Retain versions for (hours)", + "Versions younger than this are always kept", + 1, + 8760, + ), + number_field( + "min_versions_to_keep", + "Minimum versions", + "Never leave a table with fewer versions than this, whatever their age", + 1, + 1000, + ), + ], + defaults.clone(), + )), + worker_default_values: defaults, + ..Default::default() + } + } + + async fn detect( + &self, + request: &RunDetectionRequest, + sender: &dyn DetectionSender, + ) -> Result<()> { + let min_versions = clamp( + int_or( + &request.worker_config_values, + "min_versions_to_keep", + DEFAULT_MIN_VERSIONS, + ), + 1, + MAX_MIN_VERSIONS, + ) as usize; + let client = self.client(); + let tables = client.list_all_tables().await?; + + let mut proposals = Vec::new(); + for encoded in &tables { + let id = parse_id(encoded); + let table = match dataset::open(&client, &id, &self.fallback).await { + Ok(table) => table, + Err(err) => { + warn!("skipping {encoded}: {err:#}"); + continue; + } + }; + let stats = match table.stats().await { + Ok(stats) => stats, + Err(err) => { + warn!("skipping {encoded}: reading its stats failed: {err:#}"); + continue; + } + }; + // Age is decided at execution against the retention window; a table + // at or under the floor cannot lose a version whatever its age, so + // proposing one would only produce a job with nothing to do. + if stats.total_versions <= min_versions { + continue; + } + let mut parameters: HashMap = HashMap::new(); + parameters.insert("table_id".to_string(), string_list(&id)); + proposals.push(JobProposal { + proposal_id: format!("{JOB_TYPE}:{encoded}"), + dedupe_key: format!("{JOB_TYPE}:{encoded}"), + job_type: JOB_TYPE.to_string(), + summary: format!("Clean up {encoded} ({} versions)", stats.total_versions), + detail: format!( + "{} versions retained, above the {min_versions} floor; \ + those outside the retention window can go", + stats.total_versions + ), + parameters, + ..Default::default() + }); + } + + let total = proposals.len() as i32; + sender.send_proposals(DetectionProposals { + request_id: request.request_id.clone(), + job_type: JOB_TYPE.to_string(), + proposals, + has_more: false, + })?; + sender.send_complete(DetectionComplete { + request_id: request.request_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + error_message: String::new(), + total_proposals: total, + })?; + Ok(()) + } + + async fn execute( + &self, + request: &ExecuteJobRequest, + sender: &dyn ExecutionSender, + ) -> Result<()> { + let job = request + .job + .as_ref() + .ok_or_else(|| anyhow!("execute request carried no job"))?; + let id = table_id(&job.parameters) + .ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?; + let retain_hours = clamp( + int_or( + &request.worker_config_values, + "retain_hours", + DEFAULT_RETAIN_HOURS, + ), + 0, + MAX_RETAIN_HOURS, + ); + let min_versions = clamp( + int_or( + &request.worker_config_values, + "min_versions_to_keep", + DEFAULT_MIN_VERSIONS, + ), + 1, + MAX_MIN_VERSIONS, + ) as usize; + + let client = self.client(); + let table = dataset::open(&client, &id, &self.fallback).await?; + let before = table.stats().await?; + + // The floor is a promise about how much history survives, so it has to + // be applied here and not only when the job was proposed: by the time it + // runs, versions may have aged past the retention window, and age alone + // would take the table below what the operator asked to keep. + let Some(floor) = version_floor(&table, min_versions).await? else { + sender.send_completed(JobCompleted { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + result: Some(JobResult { + summary: format!( + "kept all {} versions; the {min_versions} version floor leaves none to remove", + before.total_versions + ), + ..Default::default() + }), + ..Default::default() + })?; + return Ok(()); + }; + + sender.send_progress(JobProgressUpdate { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + progress_percent: 10.0, + stage: format!("cleaning up {} versions", before.total_versions), + ..Default::default() + })?; + + let policy = CleanupPolicy { + before_timestamp: Some(Utc::now() - Duration::hours(retain_hours)), + // should_clean() ANDs its clauses, so a version has to be both older + // than the window and below the floor to go. + before_version: Some(floor), + // Files this dataset cannot account for are left alone: they may + // belong to a writer that has not committed yet, and deleting them + // would corrupt a commit in flight. + delete_unverified: false, + // A tagged version is pinned on purpose, so refuse rather than + // silently dropping what someone named. + error_if_tagged_old_versions: true, + ..Default::default() + }; + let stats = cleanup_old_versions(&table.dataset, policy) + .await + .with_context(|| format!("clean up versions of {}", table.location))?; + + let mut output: HashMap = HashMap::new(); + output.insert( + "old_versions_removed".to_string(), + int_value(stats.old_versions as i64), + ); + output.insert( + "bytes_removed".to_string(), + int_value(stats.bytes_removed as i64), + ); + output.insert( + "data_files_removed".to_string(), + int_value(stats.data_files_removed as i64), + ); + + sender.send_completed(JobCompleted { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + result: Some(JobResult { + output_values: output, + summary: format!( + "removed {} versions and {} bytes", + stats.old_versions, stats.bytes_removed + ), + ..Default::default() + }), + ..Default::default() + })?; + Ok(()) + } +} diff --git a/seaweed-worker/crates/lance/src/jobs/compact.rs b/seaweed-worker/crates/lance/src/jobs/compact.rs new file mode 100644 index 000000000..5fc6333d3 --- /dev/null +++ b/seaweed-worker/crates/lance/src/jobs/compact.rs @@ -0,0 +1,296 @@ +use std::collections::HashMap; + +use anyhow::{anyhow, Context, Result}; +use async_trait::async_trait; +use lance::dataset::optimize::{compact_files, CompactionOptions}; +use seaweed_worker_core::config_form::{form, int_or, int_value, number_field, string_value}; +use seaweed_worker_core::pb::{ + ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted, + JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor, + RunDetectionRequest, WorkerObservations, +}; +use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler}; +use tracing::warn; + +use crate::catalog::{parse_id, NamespaceClient}; +use crate::dataset; +use crate::jobs::{clamp, observation, string_list, table_id, FORMAT}; + +pub const JOB_TYPE: &str = "lance_compact"; + +const DEFAULT_TARGET_ROWS: i64 = 1_048_576; +const DEFAULT_MIN_FRAGMENTS: i64 = 8; + +// The ranges the descriptor's form offers. These are also the values that stay +// meaningful: a table needs two fragments before merging them means anything, +// and a fragment target below a thousand rows defeats the purpose of the job. +// Both are cast to usize, where a negative would arrive as an enormous number. +const TARGET_ROWS_FLOOR: i64 = 1024; +const TARGET_ROWS_CEILING: i64 = 16_777_216; +const MIN_FRAGMENTS_FLOOR: i64 = 2; +const MIN_FRAGMENTS_CEILING: i64 = 4096; + +/// Lance writes one fragment per write batch, so a table fed by small appends +/// accumulates small files the same way an Iceberg table does. +pub struct CompactHandler { + namespace_url: String, + fallback: dataset::FallbackOptions, +} + +impl CompactHandler { + pub fn new(namespace_url: String) -> Self { + Self { + namespace_url, + fallback: dataset::FallbackOptions::new(), + } + } + + /// Storage options to use where the namespace vends none. + pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self { + self.fallback = fallback; + self + } + + fn client(&self) -> NamespaceClient { + NamespaceClient::new(self.namespace_url.clone()) + } +} + +#[async_trait] +impl JobHandler for CompactHandler { + fn capability(&self) -> JobTypeCapability { + JobTypeCapability { + job_type: JOB_TYPE.to_string(), + can_detect: true, + can_execute: true, + max_detection_concurrency: 1, + max_execution_concurrency: 1, + display_name: "Lance Compaction".to_string(), + description: "Merge small Lance fragments into fewer, larger ones".to_string(), + weight: 20, + } + } + + fn descriptor(&self) -> JobTypeDescriptor { + let mut defaults: HashMap = HashMap::new(); + defaults.insert( + "target_rows_per_fragment".to_string(), + int_value(DEFAULT_TARGET_ROWS), + ); + defaults.insert( + "min_fragments".to_string(), + int_value(DEFAULT_MIN_FRAGMENTS), + ); + + JobTypeDescriptor { + job_type: JOB_TYPE.to_string(), + display_name: "Lance Compaction".to_string(), + description: "Compact fragments of Lance tables".to_string(), + icon: "fas fa-compress".to_string(), + descriptor_version: 1, + worker_config_form: Some(form( + "lance-compact-worker", + "Compaction", + vec![ + number_field( + "target_rows_per_fragment", + "Target rows per fragment", + "Rows to aim for when rewriting fragments", + 1024, + 16_777_216, + ), + number_field( + "min_fragments", + "Minimum fragments", + "Leave a table alone until it has at least this many fragments", + 2, + 4096, + ), + ], + defaults.clone(), + )), + worker_default_values: defaults, + ..Default::default() + } + } + + /// Propose a job for every table with more fragments than the operator is + /// willing to leave alone. Opening a dataset reads its manifest, not its + /// data, so this stays cheap across a catalog. + async fn detect( + &self, + request: &RunDetectionRequest, + sender: &dyn DetectionSender, + ) -> Result<()> { + let min_fragments = clamp( + int_or( + &request.worker_config_values, + "min_fragments", + DEFAULT_MIN_FRAGMENTS, + ), + MIN_FRAGMENTS_FLOOR, + MIN_FRAGMENTS_CEILING, + ) as usize; + let client = self.client(); + let tables = client.list_all_tables().await?; + + let mut proposals = Vec::new(); + let mut observations = Vec::new(); + for encoded in &tables { + let id = parse_id(encoded); + let table = match dataset::open(&client, &id, &self.fallback).await { + Ok(table) => table, + Err(err) => { + // A table that cannot be opened is the next run's problem, + // not a reason to abandon the whole sweep. + warn!("skipping {encoded}: {err:#}"); + continue; + } + }; + // One unreadable table must not end the sweep: the tables already + // read would lose their proposals, and admin would get no + // completion for this request at all. + let stats = match table.stats().await { + Ok(stats) => stats, + Err(err) => { + warn!("skipping {encoded}: reading its stats failed: {err:#}"); + continue; + } + }; + // Logged because "detection proposed nothing" is otherwise + // indistinguishable from a table the worker could not read. + tracing::info!( + "compaction detection: {encoded} has {} fragments, threshold {min_fragments}", + stats.fragments + ); + + let mut attributes: HashMap = HashMap::new(); + attributes.insert("fragments".to_string(), int_value(stats.fragments as i64)); + attributes.insert("version".to_string(), int_value(stats.version as i64)); + attributes.insert( + "versions".to_string(), + int_value(stats.total_versions as i64), + ); + attributes.insert("rows".to_string(), int_value(stats.rows as i64)); + if let Some(schema) = stats.schema.clone() { + attributes.insert("schema".to_string(), string_value(schema)); + } + observations.push(observation(&id, FORMAT, attributes)); + + if stats.fragments < min_fragments { + continue; + } + let mut parameters: HashMap = HashMap::new(); + parameters.insert("table_id".to_string(), string_list(&id)); + proposals.push(JobProposal { + proposal_id: format!("{JOB_TYPE}:{encoded}"), + dedupe_key: format!("{JOB_TYPE}:{encoded}"), + job_type: JOB_TYPE.to_string(), + summary: format!("Compact {encoded} ({} fragments)", stats.fragments), + detail: format!( + "{} fragments at version {}, above the {min_fragments} the policy allows", + stats.fragments, stats.version + ), + parameters, + ..Default::default() + }); + } + + if !observations.is_empty() { + sender.send_observations(WorkerObservations { + job_type: JOB_TYPE.to_string(), + observations, + })?; + } + + let total = proposals.len() as i32; + sender.send_proposals(DetectionProposals { + request_id: request.request_id.clone(), + job_type: JOB_TYPE.to_string(), + proposals, + has_more: false, + })?; + sender.send_complete(DetectionComplete { + request_id: request.request_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + error_message: String::new(), + total_proposals: total, + })?; + Ok(()) + } + + async fn execute( + &self, + request: &ExecuteJobRequest, + sender: &dyn ExecutionSender, + ) -> Result<()> { + let job = request + .job + .as_ref() + .ok_or_else(|| anyhow!("execute request carried no job"))?; + let id = table_id(&job.parameters) + .ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?; + let target_rows = clamp( + int_or( + &request.worker_config_values, + "target_rows_per_fragment", + DEFAULT_TARGET_ROWS, + ), + TARGET_ROWS_FLOOR, + TARGET_ROWS_CEILING, + ) as usize; + + let client = self.client(); + // Re-resolve rather than trusting the location detection saw: the table + // may have been repointed, and the vended credentials have expired. + let mut table = dataset::open(&client, &id, &self.fallback).await?; + let before = table.stats().await?; + + sender.send_progress(JobProgressUpdate { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + progress_percent: 10.0, + stage: format!("compacting {} fragments", before.fragments), + ..Default::default() + })?; + + let options = CompactionOptions { + target_rows_per_fragment: target_rows, + ..Default::default() + }; + let metrics = compact_files(&mut table.dataset, options, None) + .await + .with_context(|| format!("compact {}", table.location))?; + + let after = table.stats().await?; + let mut output: HashMap = HashMap::new(); + output.insert( + "fragments_removed".to_string(), + int_value(metrics.fragments_removed as i64), + ); + output.insert( + "fragments_added".to_string(), + int_value(metrics.fragments_added as i64), + ); + output.insert( + "files_removed".to_string(), + int_value(metrics.files_removed as i64), + ); + + sender.send_completed(JobCompleted { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + result: Some(JobResult { + output_values: output, + summary: format!("{} fragments became {}", before.fragments, after.fragments), + ..Default::default() + }), + ..Default::default() + })?; + Ok(()) + } +} diff --git a/seaweed-worker/crates/lance/src/jobs/indices.rs b/seaweed-worker/crates/lance/src/jobs/indices.rs new file mode 100644 index 000000000..5c8d70391 --- /dev/null +++ b/seaweed-worker/crates/lance/src/jobs/indices.rs @@ -0,0 +1,256 @@ +use std::collections::HashMap; + +use anyhow::{anyhow, Context, Result}; +use async_trait::async_trait; +use lance::index::DatasetIndexExt; +use lance_index::optimize::OptimizeOptions; +use seaweed_worker_core::config_form::{form, int_or, int_value, number_field}; +use seaweed_worker_core::pb::{ + ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted, + JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor, + RunDetectionRequest, +}; +use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler}; +use tracing::warn; + +use crate::catalog::{parse_id, NamespaceClient}; +use crate::dataset::{self, OpenTable}; +use crate::jobs::{clamp, string_list, table_id}; + +pub const JOB_TYPE: &str = "lance_optimize_indices"; + +const DEFAULT_MAX_UNINDEXED_ROWS: i64 = 100_000; + +// Zero is a real setting - reindex as soon as any row is uncovered - so the +// floor is what stays meaningful rather than what the form offers. The ceiling +// is the form's, and the point of both is that this is cast to u64: a negative +// would arrive as an enormous budget and mean "never reindex". +const MAX_UNINDEXED_FLOOR: i64 = 0; +const MAX_UNINDEXED_CEILING: i64 = 100_000_000; + +/// Rows written after an index was built are not covered by it, so a vector +/// search quietly misses them. This is the job with no Iceberg equivalent, and +/// the reason a neglected Lance table is a correctness problem rather than a +/// slow one. +pub struct OptimizeIndicesHandler { + namespace_url: String, + fallback: dataset::FallbackOptions, +} + +impl OptimizeIndicesHandler { + pub fn new(namespace_url: String) -> Self { + Self { + namespace_url, + fallback: dataset::FallbackOptions::new(), + } + } + + pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self { + self.fallback = fallback; + self + } + + fn client(&self) -> NamespaceClient { + NamespaceClient::new(self.namespace_url.clone()) + } +} + +/// Rows no index covers, summed across a table's indices. A table with no +/// indices at all has nothing to optimize, which is different from a table whose +/// indices have fallen behind. +async fn unindexed_rows(table: &OpenTable) -> Result> { + let indices = table.dataset.load_indices().await?; + if indices.is_empty() { + return Ok(None); + } + + let mut worst = 0u64; + let mut names: Vec = indices.iter().map(|index| index.name.clone()).collect(); + names.sort(); + names.dedup(); + for name in names { + let raw = table.dataset.index_statistics(&name).await?; + let stats: serde_json::Value = serde_json::from_str(&raw) + .with_context(|| format!("parse index statistics for {name}"))?; + let unindexed = stats + .get("num_unindexed_rows") + .and_then(|value| value.as_u64()) + .unwrap_or(0); + worst = worst.max(unindexed); + } + Ok(Some(worst)) +} + +#[async_trait] +impl JobHandler for OptimizeIndicesHandler { + fn capability(&self) -> JobTypeCapability { + JobTypeCapability { + job_type: JOB_TYPE.to_string(), + can_detect: true, + can_execute: true, + max_detection_concurrency: 1, + max_execution_concurrency: 1, + display_name: "Lance Index Optimization".to_string(), + description: "Extend indices to cover rows written since they were built".to_string(), + weight: 30, + } + } + + fn descriptor(&self) -> JobTypeDescriptor { + let mut defaults: HashMap = HashMap::new(); + defaults.insert( + "max_unindexed_rows".to_string(), + int_value(DEFAULT_MAX_UNINDEXED_ROWS), + ); + + JobTypeDescriptor { + job_type: JOB_TYPE.to_string(), + display_name: "Lance Index Optimization".to_string(), + description: "Keep vector and scalar indices covering the whole table".to_string(), + icon: "fas fa-magnifying-glass-chart".to_string(), + descriptor_version: 1, + worker_config_form: Some(form( + "lance-indices-worker", + "Index optimization", + vec![number_field( + "max_unindexed_rows", + "Unindexed row budget", + "Reindex once a table has more rows than this outside its indices", + 1_000, + 100_000_000, + )], + defaults.clone(), + )), + worker_default_values: defaults, + ..Default::default() + } + } + + async fn detect( + &self, + request: &RunDetectionRequest, + sender: &dyn DetectionSender, + ) -> Result<()> { + let budget = clamp( + int_or( + &request.worker_config_values, + "max_unindexed_rows", + DEFAULT_MAX_UNINDEXED_ROWS, + ), + MAX_UNINDEXED_FLOOR, + MAX_UNINDEXED_CEILING, + ) as u64; + let client = self.client(); + let tables = client.list_all_tables().await?; + + let mut proposals = Vec::new(); + for encoded in &tables { + let id = parse_id(encoded); + let table = match dataset::open(&client, &id, &self.fallback).await { + Ok(table) => table, + Err(err) => { + warn!("skipping {encoded}: {err:#}"); + continue; + } + }; + let unindexed = match unindexed_rows(&table).await { + Ok(Some(unindexed)) => unindexed, + Ok(None) => continue, + Err(err) => { + warn!("skipping {encoded}: reading its index stats failed: {err:#}"); + continue; + } + }; + if unindexed <= budget { + continue; + } + let mut parameters: HashMap = HashMap::new(); + parameters.insert("table_id".to_string(), string_list(&id)); + proposals.push(JobProposal { + proposal_id: format!("{JOB_TYPE}:{encoded}"), + dedupe_key: format!("{JOB_TYPE}:{encoded}"), + job_type: JOB_TYPE.to_string(), + summary: format!("Reindex {encoded} ({unindexed} rows uncovered)"), + detail: format!( + "{unindexed} rows sit outside the indices, above the {budget} the policy allows; \ + a search of this table misses them" + ), + parameters, + ..Default::default() + }); + } + + let total = proposals.len() as i32; + sender.send_proposals(DetectionProposals { + request_id: request.request_id.clone(), + job_type: JOB_TYPE.to_string(), + proposals, + has_more: false, + })?; + sender.send_complete(DetectionComplete { + request_id: request.request_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + error_message: String::new(), + total_proposals: total, + })?; + Ok(()) + } + + async fn execute( + &self, + request: &ExecuteJobRequest, + sender: &dyn ExecutionSender, + ) -> Result<()> { + let job = request + .job + .as_ref() + .ok_or_else(|| anyhow!("execute request carried no job"))?; + let id = table_id(&job.parameters) + .ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?; + + let client = self.client(); + let mut table = dataset::open(&client, &id, &self.fallback).await?; + let before = unindexed_rows(&table).await?.unwrap_or(0); + + sender.send_progress(JobProgressUpdate { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + progress_percent: 10.0, + stage: format!("indexing {before} uncovered rows"), + ..Default::default() + })?; + + // Merging every delta keeps read latency from drifting as the table is + // reindexed again and again; leaving them unmerged is how an index ends + // up fast to write and slow to search. + table + .dataset + .optimize_indices(&OptimizeOptions::default()) + .await + .with_context(|| format!("optimize indices of {}", table.location))?; + + let after = unindexed_rows(&table).await?.unwrap_or(0); + let mut output: HashMap = HashMap::new(); + output.insert( + "unindexed_rows_before".to_string(), + int_value(before as i64), + ); + output.insert("unindexed_rows_after".to_string(), int_value(after as i64)); + + sender.send_completed(JobCompleted { + request_id: request.request_id.clone(), + job_id: job.job_id.clone(), + job_type: JOB_TYPE.to_string(), + success: true, + result: Some(JobResult { + output_values: output, + summary: format!("{before} uncovered rows became {after}"), + ..Default::default() + }), + ..Default::default() + })?; + Ok(()) + } +} diff --git a/seaweed-worker/crates/lance/src/jobs/mod.rs b/seaweed-worker/crates/lance/src/jobs/mod.rs new file mode 100644 index 000000000..ae995f7e3 --- /dev/null +++ b/seaweed-worker/crates/lance/src/jobs/mod.rs @@ -0,0 +1,104 @@ +//! One module per job type. Each declares its capability and the settings form +//! admin renders for it, then does the work. +//! +//! Detection opens each table and reads its manifest rather than its data, so a +//! sweep across a catalog stays cheap. Execution re-resolves the table instead +//! of trusting what detection saw: it may have been repointed, and the vended +//! credentials expire. + +pub mod cleanup; +pub mod compact; +pub mod indices; + +use std::collections::HashMap; +use std::sync::Arc; + +use seaweed_worker_core::pb::{config_value::Kind, ConfigValue, ObjectObservation, StringList}; +use seaweed_worker_core::JobHandler; + +use crate::catalog::parse_id; + +/// A table identifier travels in a proposal's parameters and comes back on the +/// job, so both sides agree on one encoding. +pub(crate) fn string_list(parts: &[String]) -> ConfigValue { + ConfigValue { + kind: Some(Kind::StringList(StringList { + values: parts.to_vec(), + })), + } +} + +pub(crate) fn table_id(parameters: &HashMap) -> Option> { + match parameters.get("table_id")?.kind.as_ref()? { + Kind::StringList(list) => Some(list.values.clone()), + Kind::StringValue(encoded) => Some(parse_id(encoded)), + _ => None, + } +} + +/// Every handler this worker serves. A worker process may serve several job +/// types, which is why WorkerHello carries a list. +pub fn handlers( + namespace_url: String, + fallback: crate::dataset::FallbackOptions, +) -> Vec> { + vec![ + Arc::new( + compact::CompactHandler::new(namespace_url.clone()).with_fallback(fallback.clone()), + ), + Arc::new( + indices::OptimizeIndicesHandler::new(namespace_url.clone()) + .with_fallback(fallback.clone()), + ), + Arc::new(cleanup::CleanupVersionsHandler::new(namespace_url).with_fallback(fallback)), + ] +} + +/// Holds a configured value to the range its form offers. A value from outside +/// it is one the UI could not have produced, and every one of these is cast to +/// an unsigned type: a negative arrives as an enormous number, which silently +/// turns a threshold into "never" rather than failing loudly. +pub(crate) fn clamp(value: i64, low: i64, high: i64) -> i64 { + value.max(low).min(high) +} + +/// The format the catalog records for the tables this worker maintains. +pub const FORMAT: &str = "LANCE"; + +/// Builds the observation a detection sweep reports for one table. Detection +/// has already opened the dataset to decide whether it needs work, so saying +/// what it saw costs nothing, and for a format the cluster cannot read this is +/// the only description of the table anything can produce. +pub(crate) fn observation( + id: &[String], + format: &str, + attributes: HashMap, +) -> ObjectObservation { + ObjectObservation { + object_id: id.to_vec(), + object_kind: "table".to_string(), + format: format.to_string(), + attributes, + observed_at: Some(std::time::SystemTime::now().into()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // Every configured threshold is cast to an unsigned type before use. A + // negative one would arrive as an enormous number and quietly mean "never", + // which looks exactly like a worker with nothing to do. + #[test] + fn clamp_keeps_a_negative_from_wrapping() { + assert_eq!(clamp(-1, 2, 4096) as usize, 2); + assert_eq!(clamp(i64::MIN, 0, 100_000_000) as u64, 0); + } + + #[test] + fn clamp_holds_the_ceiling_and_passes_the_middle() { + assert_eq!(clamp(i64::MAX, 0, 8760), 8760); + assert_eq!(clamp(168, 0, 8760), 168); + } +} diff --git a/seaweed-worker/crates/lance/src/lib.rs b/seaweed-worker/crates/lance/src/lib.rs new file mode 100644 index 000000000..ee8b88f39 --- /dev/null +++ b/seaweed-worker/crates/lance/src/lib.rs @@ -0,0 +1,13 @@ +//! Maintenance jobs for Lance tables. +//! +//! A Lance dataset needs three things done to it over time: its fragments +//! compacted, its indices extended to cover rows written after they were built, +//! and its old versions removed. None can run in the Go worker, because all +//! three read and rewrite Lance files. This crate is the worker that can. + +pub mod catalog; +pub mod dataset; +pub mod jobs; +pub mod preview; + +pub use jobs::handlers; diff --git a/seaweed-worker/crates/lance/src/main.rs b/seaweed-worker/crates/lance/src/main.rs new file mode 100644 index 000000000..6bc42a5bd --- /dev/null +++ b/seaweed-worker/crates/lance/src/main.rs @@ -0,0 +1,127 @@ +use std::time::Duration; + +use anyhow::Result; +use clap::Parser; +use std::sync::Arc; + +use seaweed_worker_core::{Registry, TlsOptions, WorkerOptions}; +use weed_lance_worker::handlers; + +/// Mirrors `weed worker`'s flags, because this is the same contract from another +/// language and an operator should not have to learn a second set of names. +#[derive(Parser, Debug)] +#[command( + name = "weed-lance-worker", + about = "SeaweedFS maintenance worker for Lance tables" +)] +struct Args { + /// Admin server gRPC address. + #[arg(long, default_value = "localhost:23646", env = "WEED_ADMIN")] + admin: String, + + /// Worker identity reported to admin. + #[arg(long, default_value = "lance-worker", env = "WEED_WORKER_ID")] + id: String, + + /// Lance namespace the worker lists tables from. + #[arg( + long, + default_value = "http://localhost:9101", + env = "WEED_LANCE_NAMESPACE" + )] + namespace: String, + + #[arg(long, default_value = "10", env = "WEED_HEARTBEAT_SECONDS")] + heartbeat_seconds: u64, + + #[arg(long, default_value = "1")] + max_concurrency: i32, + + /// Storage credentials to use where the namespace vends none. A gateway + /// with STS configured vends its own and these are ignored. + #[arg(long, env = "WEED_S3_ACCESS_KEY")] + access_key: Option, + + #[arg(long, env = "WEED_S3_SECRET_KEY")] + secret_key: Option, + + /// mTLS for the admin stream, the same certificates the Go worker reads + /// from the [grpc.worker] section of security.toml. All three together, or + /// none, in which case the stream is plaintext. + #[arg(long, env = "WEED_GRPC_CA")] + tls_ca: Option, + + #[arg(long, env = "WEED_GRPC_CLIENT_CERT")] + tls_cert: Option, + + #[arg(long, env = "WEED_GRPC_CLIENT_KEY")] + tls_key: Option, + + /// Name to verify admin's certificate against, when it is not the address + /// this worker dials. + #[arg(long)] + tls_server_name: Option, +} + +impl Args { + /// The TLS configuration, or an error when the three certificate paths do + /// not arrive together: a CA on its own would silently give one-way TLS, + /// which the cluster's mutual setup refuses anyway. + fn tls(&self) -> Result> { + match ( + self.tls_ca.clone(), + self.tls_cert.clone(), + self.tls_key.clone(), + ) { + (None, None, None) => Ok(None), + (Some(ca_path), Some(client_cert_path), Some(client_key_path)) => { + Ok(Some(TlsOptions { + ca_path, + client_cert_path, + client_key_path, + server_name: self.tls_server_name.clone(), + })) + } + _ => Err(anyhow::anyhow!( + "--tls-ca, --tls-cert and --tls-key must be given together" + )), + } + } +} + +#[tokio::main] +async fn main() -> Result<()> { + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")), + ) + .init(); + + let args = Args::parse(); + let tls = args.tls()?; + let options = WorkerOptions { + admin_address: args.admin, + worker_id: args.id, + heartbeat_interval: Duration::from_secs(args.heartbeat_seconds), + max_detection_concurrency: args.max_concurrency, + max_execution_concurrency: args.max_concurrency, + tls, + ..Default::default() + }; + + let mut fallback = weed_lance_worker::dataset::FallbackOptions::new(); + if let (Some(access), Some(secret)) = (args.access_key, args.secret_key) { + fallback.insert("aws_access_key_id".to_string(), access); + fallback.insert("aws_secret_access_key".to_string(), secret); + } + + let mut registry = Registry::new().with_preview(Arc::new( + weed_lance_worker::preview::LancePreview::new(args.namespace.clone(), fallback.clone()), + )); + for handler in handlers(args.namespace, fallback) { + registry = registry.register(handler); + } + + seaweed_worker_core::run(options, registry).await +} diff --git a/seaweed-worker/crates/lance/src/preview.rs b/seaweed-worker/crates/lance/src/preview.rs new file mode 100644 index 000000000..092791b69 --- /dev/null +++ b/seaweed-worker/crates/lance/src/preview.rs @@ -0,0 +1,86 @@ +//! Sample rows of a Lance table, for a page that cannot read the format. +//! +//! Admin renders an Iceberg table by reading its Parquet files directly. There +//! is no Go Lance reader, so for Lance the worker does the reading and hands +//! back text. + +use anyhow::{Context, Result}; +use arrow_cast::display::{ArrayFormatter, FormatOptions}; +use async_trait::async_trait; +use futures::TryStreamExt; +use seaweed_worker_core::{Preview, PreviewProvider}; + +use crate::catalog::NamespaceClient; +use crate::dataset::{self, FallbackOptions}; +use crate::jobs::FORMAT; + +pub struct LancePreview { + namespace_url: String, + fallback: FallbackOptions, +} + +impl LancePreview { + pub fn new(namespace_url: String, fallback: FallbackOptions) -> Self { + Self { + namespace_url, + fallback, + } + } +} + +#[async_trait] +impl PreviewProvider for LancePreview { + fn format(&self) -> &str { + FORMAT + } + + async fn preview(&self, object_id: &[String], row_limit: usize) -> Result { + let client = NamespaceClient::new(self.namespace_url.clone()); + let table = dataset::open(&client, object_id, &self.fallback).await?; + + let total_rows = table.dataset.count_rows(None).await.unwrap_or(0) as i64; + let mut scanner = table.dataset.scan(); + scanner.limit(Some(row_limit as i64), None)?; + + let batches: Vec<_> = scanner + .try_into_stream() + .await? + .try_collect() + .await + .with_context(|| format!("read rows from {}", table.location))?; + + let columns = table + .dataset + .schema() + .fields + .iter() + .map(|field| field.name.clone()) + .collect(); + + let mut rows = Vec::new(); + for batch in &batches { + // ArrayFormatter renders each Arrow type the way its own tooling + // does, so a vector column reads as a vector rather than as bytes. + let formatters = batch + .columns() + .iter() + .map(|array| ArrayFormatter::try_new(array.as_ref(), &FormatOptions::default())) + .collect::, _>>()?; + for index in 0..batch.num_rows() { + rows.push( + formatters + .iter() + .map(|formatter| formatter.value(index).to_string()) + .collect(), + ); + } + } + rows.truncate(row_limit); + + Ok(Preview { + columns, + rows, + total_rows, + }) + } +} diff --git a/seaweed-worker/crates/lance/tests/compaction.rs b/seaweed-worker/crates/lance/tests/compaction.rs new file mode 100644 index 000000000..3c83107ed --- /dev/null +++ b/seaweed-worker/crates/lance/tests/compaction.rs @@ -0,0 +1,606 @@ +//! Drives the compaction handler against a live namespace. +//! +//! Skipped unless WEED_LANCE_NAMESPACE names one, the way the Go integration +//! tests skip without Docker: compaction rewrites real files, and there is +//! nothing to learn from it against a fake. + +use std::collections::HashMap; +use std::sync::Mutex; + +use anyhow::Result; +use seaweed_worker_core::pb::{ + config_value::Kind, ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, + JobCompleted, JobProgressUpdate, JobProposal, JobSpec, RunDetectionRequest, +}; +use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler, PreviewProvider}; +use weed_lance_worker::catalog::NamespaceClient; +use weed_lance_worker::jobs::cleanup::CleanupVersionsHandler; +use weed_lance_worker::jobs::compact::{CompactHandler, JOB_TYPE}; +use weed_lance_worker::jobs::indices::OptimizeIndicesHandler; +use weed_lance_worker::preview::LancePreview; + +#[derive(Default)] +struct Recorder { + proposals: Mutex>, + observations: Mutex>, + completed: Mutex>, +} + +impl DetectionSender for Recorder { + fn send_proposals(&self, proposals: DetectionProposals) -> Result<()> { + self.proposals.lock().unwrap().extend(proposals.proposals); + Ok(()) + } + fn send_complete(&self, _complete: DetectionComplete) -> Result<()> { + Ok(()) + } + fn send_activity(&self, _activity: seaweed_worker_core::pb::ActivityEvent) -> Result<()> { + Ok(()) + } + fn send_observations( + &self, + observations: seaweed_worker_core::pb::WorkerObservations, + ) -> Result<()> { + self.observations + .lock() + .unwrap() + .extend(observations.observations); + Ok(()) + } +} + +impl ExecutionSender for Recorder { + fn send_progress(&self, _progress: JobProgressUpdate) -> Result<()> { + Ok(()) + } + fn send_completed(&self, completed: JobCompleted) -> Result<()> { + self.completed.lock().unwrap().push(completed); + Ok(()) + } +} + +/// These tests drive one live gateway and one shared catalog: `list_all_tables` +/// sweeps everything, so a table another test is writing shows up in this test's +/// detection. Rust runs a binary's tests concurrently, so take a lock. +static GATEWAY: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + +fn namespace_url() -> Option { + std::env::var("WEED_LANCE_NAMESPACE") + .ok() + .filter(|s| !s.is_empty()) +} + +fn int_config(name: &str, value: i64) -> HashMap { + let mut values = HashMap::new(); + values.insert( + name.to_string(), + ConfigValue { + kind: Some(Kind::Int64Value(value)), + }, + ); + values +} + +/// Declares a table through the namespace and writes `fragments` one-row +/// appends into it, so a test brings its own state instead of depending on +/// whatever a previous run left behind. +async fn seed_fragmented_table(url: &str, name: &str, fragments: usize) -> Result { + seed_table(url, name, fragments, 1, false).await +} + +/// Writes `batches` appends of `rows_each` into a freshly declared table, and +/// optionally builds a vector index after the first batch so the later ones are +/// rows no index covers. +async fn seed_table( + url: &str, + name: &str, + batches: usize, + rows_each: usize, + with_index: bool, +) -> Result { + use arrow_array::{ + FixedSizeListArray, Float32Array, Int64Array, RecordBatch, RecordBatchIterator, + }; + use arrow_schema::{DataType, Field, Schema}; + use lance::dataset::{Dataset, WriteMode, WriteParams}; + use lance::io::{ObjectStoreParams, StorageOptionsAccessor}; + use std::sync::Arc; + + // Declaring is the namespace's job, not the worker's, so the test asks for + // it directly rather than widening the client the worker uses. The bucket and + // namespace come first: a table cannot be declared under a parent that does + // not exist, and a test that assumes one is a test that only passes twice. + let http = reqwest::Client::new(); + for parent in ["vec", "vec$ml"] { + http.post(format!("{url}/v1/namespace/{parent}/create")) + .json(&serde_json::json!({"mode": "EXIST_OK"})) + .send() + .await? + .error_for_status()?; + } + let encoded = format!("vec$ml${name}"); + http.post(format!("{url}/v1/table/{encoded}/declare")) + .json(&serde_json::json!({})) + .send() + .await? + .error_for_status()?; + + let client = NamespaceClient::new(url.to_string()); + let id = vec!["vec".to_string(), "ml".to_string(), name.to_string()]; + let description = client.describe_table(&id).await?; + + let mut options = description.storage_options.clone(); + options.extend(fallback()); + const DIM: i32 = 16; + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new( + "vec", + DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), DIM), + false, + ), + ])); + + for i in 0..batches { + let ids: Vec = (0..rows_each).map(|r| (i * rows_each + r) as i64).collect(); + let values: Vec = ids + .iter() + .flat_map(|id| (0..DIM).map(move |d| (*id as f32) + d as f32)) + .collect(); + let vectors = FixedSizeListArray::new( + Arc::new(Field::new("item", DataType::Float32, true)), + DIM, + Arc::new(Float32Array::from(values)), + None, + ); + let batch = RecordBatch::try_new( + schema.clone(), + vec![Arc::new(Int64Array::from(ids)), Arc::new(vectors)], + )?; + let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone()); + let params = WriteParams { + mode: if i == 0 { + WriteMode::Overwrite + } else { + WriteMode::Append + }, + store_params: Some(ObjectStoreParams { + storage_options_accessor: Some(std::sync::Arc::new( + StorageOptionsAccessor::with_static_options(options.clone()), + )), + ..Default::default() + }), + ..Default::default() + }; + let dataset = Dataset::write(reader, description.location.as_str(), Some(params)).await?; + + // The index is built after the first batch, so everything appended + // afterwards is a row it does not cover. + if with_index && i == 0 { + use lance::index::vector::VectorIndexParams; + use lance::index::DatasetIndexExt; + use lance_index::vector::{ivf::IvfBuildParams, pq::PQBuildParams}; + use lance_index::IndexType; + + let mut dataset = dataset; + let params = VectorIndexParams::with_ivf_pq_params( + lance_linalg::distance::MetricType::L2, + IvfBuildParams::new(1), + PQBuildParams::new(4, 8), + ); + dataset + .create_index(&["vec"], IndexType::Vector, None, ¶ms, true) + .await?; + } + } + Ok(encoded) +} + +/// A table with more fragments than the policy allows is proposed, and running +/// the proposal leaves it with fewer than it started with. +#[tokio::test] +async fn compacts_a_fragmented_table() { + let _gateway = GATEWAY.lock().await; + let Some(url) = namespace_url() else { + eprintln!("WEED_LANCE_NAMESPACE is unset, skipping"); + return; + }; + let mut fallback = weed_lance_worker::dataset::FallbackOptions::new(); + fallback.insert("aws_access_key_id".to_string(), "any".to_string()); + fallback.insert("aws_secret_access_key".to_string(), "any".to_string()); + // Seeded here rather than by a script, so the test is repeatable: a previous + // run compacts the table it depended on. + let encoded = seed_fragmented_table(&url, "compactme", 12) + .await + .expect("seed a fragmented table"); + let handler = CompactHandler::new(url).with_fallback(fallback); + let recorder = Recorder::default(); + + let request = RunDetectionRequest { + request_id: "detect-1".to_string(), + job_type: JOB_TYPE.to_string(), + worker_config_values: int_config("min_fragments", 4), + ..Default::default() + }; + handler + .detect(&request, &recorder) + .await + .expect("detection failed"); + + let proposals = recorder.proposals.lock().unwrap().clone(); + assert!( + !proposals.is_empty(), + "expected a proposal for the fragmented table" + ); + let proposal = proposals + .iter() + .find(|p| p.summary.contains(encoded.as_str())) + .expect("no proposal for the seeded table"); + + // Detection opened the dataset to decide, so it reports what it saw. This is + // the only description of a Lance table anything outside the format can give. + let observations = recorder.observations.lock().unwrap().clone(); + let observed = observations + .iter() + .find(|o| o.object_id.last().map(String::as_str) == Some("compactme")) + .expect("detection reported no observation for the seeded table"); + assert_eq!(observed.format, "LANCE"); + for attribute in ["fragments", "rows", "versions", "schema"] { + assert!( + observed.attributes.contains_key(attribute), + "observation is missing {attribute}: {:?}", + observed.attributes.keys().collect::>() + ); + } + + let execute = ExecuteJobRequest { + request_id: "execute-1".to_string(), + job: Some(JobSpec { + job_id: "job-1".to_string(), + job_type: JOB_TYPE.to_string(), + parameters: proposal.parameters.clone(), + ..Default::default() + }), + worker_config_values: int_config("target_rows_per_fragment", 1_048_576), + ..Default::default() + }; + handler + .execute(&execute, &recorder) + .await + .expect("execution failed"); + + let completed = recorder.completed.lock().unwrap().clone(); + let result = completed.first().expect("no completion reported"); + assert!( + result.success, + "compaction reported failure: {}", + result.error_message + ); + let summary = result + .result + .as_ref() + .map(|r| r.summary.clone()) + .unwrap_or_default(); + assert!( + summary.contains("became"), + "completion carried no fragment counts: {summary}" + ); + eprintln!("compaction result: {summary}"); +} + +fn fallback() -> weed_lance_worker::dataset::FallbackOptions { + let mut options = weed_lance_worker::dataset::FallbackOptions::new(); + options.insert("aws_access_key_id".to_string(), "any".to_string()); + options.insert("aws_secret_access_key".to_string(), "any".to_string()); + options +} + +/// A table with more versions than the floor is proposed, and running the job +/// reports what it removed. The compaction test above leaves one behind. +#[tokio::test] +async fn cleans_up_old_versions() { + let _gateway = GATEWAY.lock().await; + let Some(url) = namespace_url() else { + eprintln!("WEED_LANCE_NAMESPACE is unset, skipping"); + return; + }; + let encoded = seed_table(&url, "cleanme", 6, 4, false) + .await + .expect("seed a table with versions to clean"); + + let handler = CleanupVersionsHandler::new(url).with_fallback(fallback()); + let recorder = Recorder::default(); + + let request = RunDetectionRequest { + request_id: "detect-cleanup".to_string(), + job_type: "lance_cleanup_versions".to_string(), + worker_config_values: int_config("min_versions_to_keep", 2), + ..Default::default() + }; + handler + .detect(&request, &recorder) + .await + .expect("detection failed"); + let proposals = recorder.proposals.lock().unwrap().clone(); + let proposal = proposals + .iter() + .find(|p| p.summary.contains(encoded.as_str())) + .cloned() + .expect("no cleanup proposal for the seeded table"); + + // Retain nothing, so every version outside the current one is fair game and + // the job has something to report rather than a no-op. + let execute = ExecuteJobRequest { + request_id: "execute-cleanup".to_string(), + job: Some(JobSpec { + job_id: "job-cleanup".to_string(), + job_type: "lance_cleanup_versions".to_string(), + parameters: proposal.parameters.clone(), + ..Default::default() + }), + worker_config_values: int_config("retain_hours", 0), + ..Default::default() + }; + handler + .execute(&execute, &recorder) + .await + .expect("cleanup failed"); + + let completed = recorder.completed.lock().unwrap().clone(); + let result = completed.last().expect("no completion reported"); + assert!( + result.success, + "cleanup reported failure: {}", + result.error_message + ); + eprintln!( + "cleanup result: {}", + result.result.as_ref().unwrap().summary + ); +} + +/// A table with no indices has nothing to optimize, so detection proposes +/// nothing rather than queueing work that would do nothing. +#[tokio::test] +async fn skips_tables_without_indices() { + let _gateway = GATEWAY.lock().await; + let Some(url) = namespace_url() else { + eprintln!("WEED_LANCE_NAMESPACE is unset, skipping"); + return; + }; + let encoded = seed_table(&url, "noindex", 2, 8, false) + .await + .expect("seed a table without an index"); + + let handler = OptimizeIndicesHandler::new(url).with_fallback(fallback()); + let recorder = Recorder::default(); + + let request = RunDetectionRequest { + request_id: "detect-indices".to_string(), + job_type: "lance_optimize_indices".to_string(), + worker_config_values: int_config("max_unindexed_rows", 1), + ..Default::default() + }; + handler + .detect(&request, &recorder) + .await + .expect("detection failed"); + // Judge this table only: the catalog holds every other test's tables too, + // and an indexed one with uncovered rows is supposed to be proposed. + assert!( + !recorder + .proposals + .lock() + .unwrap() + .iter() + .any(|p| p.summary.contains(encoded.as_str())), + "a table with no indices must not be proposed for reindexing" + ); +} + +/// The job with no Iceberg equivalent: rows appended after an index was built +/// are invisible to a search of it until this runs. Needs a table with an index +/// and rows outside it, which `indexed.py` in the scratchpad seeds. +#[tokio::test] +async fn reindexes_rows_an_index_does_not_cover() { + let _gateway = GATEWAY.lock().await; + let Some(url) = namespace_url() else { + eprintln!("WEED_LANCE_NAMESPACE is unset, skipping"); + return; + }; + let encoded = seed_table(&url, "reindexme", 2, 512, true) + .await + .expect("seed an indexed table with uncovered rows"); + let handler = OptimizeIndicesHandler::new(url).with_fallback(fallback()); + let recorder = Recorder::default(); + + let request = RunDetectionRequest { + request_id: "detect-reindex".to_string(), + job_type: "lance_optimize_indices".to_string(), + worker_config_values: int_config("max_unindexed_rows", 100), + ..Default::default() + }; + handler + .detect(&request, &recorder) + .await + .expect("detection failed"); + let proposals = recorder.proposals.lock().unwrap().clone(); + let proposal = proposals + .iter() + .find(|p| p.summary.contains(encoded.as_str())) + .cloned() + .expect("the seeded indexed table was not proposed"); + + let execute = ExecuteJobRequest { + request_id: "execute-reindex".to_string(), + job: Some(JobSpec { + job_id: "job-reindex".to_string(), + job_type: "lance_optimize_indices".to_string(), + parameters: proposal.parameters.clone(), + ..Default::default() + }), + ..Default::default() + }; + handler + .execute(&execute, &recorder) + .await + .expect("reindex failed"); + + let completed = recorder.completed.lock().unwrap().clone(); + let result = completed.last().expect("no completion reported"); + assert!( + result.success, + "reindex reported failure: {}", + result.error_message + ); + let output = &result.result.as_ref().unwrap().output_values; + let after = match output + .get("unindexed_rows_after") + .and_then(|v| v.kind.as_ref()) + { + Some(Kind::Int64Value(value)) => *value, + other => panic!("no unindexed_rows_after in {other:?}"), + }; + assert_eq!( + after, 0, + "rows are still outside the index after optimizing" + ); + eprintln!( + "reindex result: {}", + result.result.as_ref().unwrap().summary + ); +} + +/// The UI's whole reason for asking a worker: admin cannot read a Lance table, +/// so the rows have to come back already rendered. +#[tokio::test] +async fn previews_rows_of_a_table() { + let _gateway = GATEWAY.lock().await; + let Some(url) = namespace_url() else { + eprintln!("set WEED_LANCE_NAMESPACE to run this test"); + return; + }; + seed_table(&url, "previewme", 2, 3, false) + .await + .expect("seed a table to preview"); + + let provider = LancePreview::new(url, fallback()); + let id = vec!["vec".to_string(), "ml".to_string(), "previewme".to_string()]; + let preview = provider.preview(&id, 4).await.expect("preview the table"); + + assert_eq!(preview.columns, vec!["id".to_string(), "vec".to_string()]); + assert_eq!(preview.total_rows, 6, "total is the table, not the sample"); + assert_eq!(preview.rows.len(), 4, "row_limit bounds the sample"); + assert!( + preview.rows[0][1].starts_with('['), + "a vector column should render as a list, got {:?}", + preview.rows[0][1] + ); +} + +/// The claim that removed managed versioning is not "one writer wins the +/// conditional PUT" - that is only the mechanism. It is that concurrent writers +/// lose nothing: the loser sees the conflict, rebases, and commits again. Eight +/// writers appending at once must leave all eight batches in the table. +#[tokio::test] +async fn concurrent_writers_keep_every_commit() { + let _gateway = GATEWAY.lock().await; + let Some(url) = namespace_url() else { + eprintln!("set WEED_LANCE_NAMESPACE to run this test"); + return; + }; + const WRITERS: i64 = 8; + const ROWS_EACH: i64 = 4; + + seed_table(&url, "racers", 1, ROWS_EACH as usize, false) + .await + .expect("seed the table the writers will append to"); + + let client = NamespaceClient::new(url.clone()); + let id = vec!["vec".to_string(), "ml".to_string(), "racers".to_string()]; + let description = client.describe_table(&id).await.expect("describe"); + let mut options = description.storage_options.clone(); + options.extend(fallback()); + + let writes = (0..WRITERS).map(|writer| { + let location = description.location.clone(); + let options = options.clone(); + tokio::spawn( + async move { append_rows(&location, &options, writer * 1000, ROWS_EACH).await }, + ) + }); + for (writer, handle) in writes.enumerate() { + handle + .await + .expect("writer panicked") + .unwrap_or_else(|err| panic!("writer {writer} failed to commit: {err:#}")); + } + + let table = weed_lance_worker::dataset::open(&client, &id, &fallback()) + .await + .expect("reopen the table"); + let rows = table.dataset.count_rows(None).await.expect("count rows"); + let expected = (ROWS_EACH + WRITERS * ROWS_EACH) as usize; + assert_eq!( + rows, expected, + "concurrent commits lost data: {rows} rows, want {expected}" + ); +} + +/// Appends one batch to an existing dataset, the way an independent writer would. +async fn append_rows( + location: &str, + options: &HashMap, + first_id: i64, + rows: i64, +) -> Result<()> { + use arrow_array::{ + FixedSizeListArray, Float32Array, Int64Array, RecordBatch, RecordBatchIterator, + }; + use arrow_schema::{DataType, Field, Schema}; + use lance::dataset::{Dataset, WriteMode, WriteParams}; + use lance::io::{ObjectStoreParams, StorageOptionsAccessor}; + use std::sync::Arc; + + const DIM: i32 = 16; + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new( + "vec", + DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), DIM), + false, + ), + ])); + let ids: Vec = (0..rows).map(|r| first_id + r).collect(); + let values: Vec = ids + .iter() + .flat_map(|id| (0..DIM).map(move |d| (*id as f32) + d as f32)) + .collect(); + let vectors = FixedSizeListArray::new( + Arc::new(Field::new("item", DataType::Float32, true)), + DIM, + Arc::new(Float32Array::from(values)), + None, + ); + let batch = RecordBatch::try_new( + schema.clone(), + vec![Arc::new(Int64Array::from(ids)), Arc::new(vectors)], + )?; + let params = WriteParams { + mode: WriteMode::Append, + store_params: Some(ObjectStoreParams { + storage_options_accessor: Some(Arc::new(StorageOptionsAccessor::with_static_options( + options.clone(), + ))), + ..Default::default() + }), + ..Default::default() + }; + Dataset::write( + RecordBatchIterator::new(vec![Ok(batch)], schema.clone()), + location, + Some(params), + ) + .await?; + Ok(()) +} diff --git a/test/s3/distributed_lock/distributed_lock_cluster_test.go b/test/s3/distributed_lock/distributed_lock_cluster_test.go index 2c09d0db9..dd9cc107a 100644 --- a/test/s3/distributed_lock/distributed_lock_cluster_test.go +++ b/test/s3/distributed_lock/distributed_lock_cluster_test.go @@ -310,6 +310,7 @@ func (c *distributedLockCluster) startS3(index int) error { "-port=" + strconv.Itoa(c.s3Ports[index]), "-port.grpc=" + strconv.Itoa(c.s3GrpcPorts[index]), "-port.iceberg=0", + "-port.lance=0", "-filer=" + strings.Join(filers, ","), "-config=" + c.s3Config, "-iam.readOnly=false", diff --git a/test/s3tables/catalog/Dockerfile.lance b/test/s3tables/catalog/Dockerfile.lance new file mode 100644 index 000000000..5824eeb1d --- /dev/null +++ b/test/s3tables/catalog/Dockerfile.lance @@ -0,0 +1,13 @@ +# Lance client container for Lance Namespace REST compatibility testing. +# The point of this image is to exercise the namespace with the real client +# rather than hand-built HTTP: every serious bug in this surface so far looked +# fine to a request we wrote ourselves. +FROM python:3.11-slim + +WORKDIR /app + +RUN pip install --no-cache-dir lance-namespace pylance pyarrow + +COPY test_lance_namespace.py /app/ + +CMD ["python3", "/app/test_lance_namespace.py", "--help"] diff --git a/test/s3tables/catalog/duckdb_oauth_test.go b/test/s3tables/catalog/duckdb_oauth_test.go index 4d9eb1f1b..865a7ee8d 100644 --- a/test/s3tables/catalog/duckdb_oauth_test.go +++ b/test/s3tables/catalog/duckdb_oauth_test.go @@ -60,7 +60,12 @@ func newOAuthTestEnv(t *testing.T) *oauthTestEnv { } weedBinary := filepath.Join(seaweedDir, "weed", "weed") - if info, err := os.Stat(weedBinary); err != nil || info.IsDir() { + if info, err := os.Stat(weedBinary); err == nil && !info.IsDir() { + // Say which binary and how old it is. `make test` rebuilds first, but + // `go test` on its own happily runs a weeks-old binary and reports a + // pass for code that is not being exercised. + t.Logf("using %s, built %s", weedBinary, info.ModTime().Format(time.RFC3339)) + } else { weedBinary = "weed" if _, err := exec.LookPath(weedBinary); err != nil { t.Skip("weed binary not found, skipping integration test") diff --git a/test/s3tables/catalog/iceberg_catalog_test.go b/test/s3tables/catalog/iceberg_catalog_test.go index 50d187de7..b45f3714a 100644 --- a/test/s3tables/catalog/iceberg_catalog_test.go +++ b/test/s3tables/catalog/iceberg_catalog_test.go @@ -62,6 +62,7 @@ type TestEnvironment struct { s3Port int s3GrpcPort int icebergPort int + lancePort int masterPort int masterGrpcPort int filerPort int @@ -92,7 +93,12 @@ func newTestEnvironmentForMain() (*TestEnvironment, error) { // Check for weed binary weedBinary := filepath.Join(seaweedDir, "weed", "weed") - if _, err := os.Stat(weedBinary); os.IsNotExist(err) { + if info, statErr := os.Stat(weedBinary); statErr == nil { + // Name the binary and its age. `make test` rebuilds first, but a plain + // `go test` will happily drive a weeks-old binary and report a pass for + // code it never ran. + fmt.Fprintf(os.Stderr, "using %s, built %s\n", weedBinary, info.ModTime().Format(time.RFC3339)) + } else if os.IsNotExist(statErr) { weedBinary = "weed" if _, err := exec.LookPath(weedBinary); err != nil { return nil, fmt.Errorf("weed binary not found") @@ -105,9 +111,9 @@ func newTestEnvironmentForMain() (*TestEnvironment, error) { return nil, fmt.Errorf("create temp dir: %w", err) } - // Allocate 9 unique ports atomically: s3, iceberg, s3Grpc, master, masterGrpc, - // filer, filerGrpc, volume, volumeGrpc - ports, err := testutil.AllocatePorts(9) + // Allocate 10 unique ports atomically: s3, iceberg, s3Grpc, master, masterGrpc, + // filer, filerGrpc, volume, volumeGrpc, lance + ports, err := testutil.AllocatePorts(10) if err != nil { return nil, fmt.Errorf("allocate ports: %w", err) } @@ -125,6 +131,7 @@ func newTestEnvironmentForMain() (*TestEnvironment, error) { filerGrpcPort: ports[6], volumePort: ports[7], volumeGrpcPort: ports[8], + lancePort: ports[9], dockerAvailable: testutil.HasDocker(), }, nil } @@ -155,6 +162,7 @@ func (env *TestEnvironment) startSeaweedFSForMain() error { "-s3.port", fmt.Sprintf("%d", env.s3Port), "-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort), "-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort), + "-s3.port.lance", fmt.Sprintf("%d", env.lancePort), "-ip.bind", "0.0.0.0", "-dir", env.dataDir, ) @@ -211,6 +219,11 @@ func (env *TestEnvironment) IcebergURL() string { return fmt.Sprintf("http://127.0.0.1:%d", env.icebergPort) } +// LanceURL returns the Lance Namespace server URL +func (env *TestEnvironment) LanceURL() string { + return fmt.Sprintf("http://127.0.0.1:%d", env.lancePort) +} + // TestIcebergConfig tests the /v1/config endpoint func TestIcebergConfig(t *testing.T) { if testing.Short() { @@ -252,7 +265,7 @@ func TestIcebergNamespaces(t *testing.T) { // Create the default table bucket first via S3 bucketName := "warehouse-ns-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") // Test GET /v1/namespaces (should return empty list initially) resp, err := http.Get(env.IcebergURL() + icebergPath(bucketName, "/v1/namespaces")) @@ -275,7 +288,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) { env := sharedEnv bucketName := "warehouse-stage-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") namespace := "stage_ns_" + randomSuffix() tableName := "orders" @@ -369,7 +382,7 @@ func TestCommitMissingTableWithoutAssertCreate(t *testing.T) { env := sharedEnv bucketName := "warehouse-missing-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") namespace := "stage_missing_assert_ns_" + randomSuffix() tableName := "missing_table" @@ -458,11 +471,16 @@ func icebergPath(prefix, path string) string { // The request is AWS V4 signed for SERVICE=s3tables so the S3 Tables // route matcher accepts it; signing with regular SERVICE=s3 would let // the request fall through to the S3 CreateBucket handler. -func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) { +// createTableBucket makes a table bucket of the given format. An empty format +// leaves it to the server, which means ICEBERG. +func createTableBucket(t *testing.T, env *TestEnvironment, bucketName, format string) { t.Helper() endpoint := fmt.Sprintf("http://localhost:%d/buckets", env.s3Port) reqBody := fmt.Sprintf(`{"name":"%s"}`, bucketName) + if format != "" { + reqBody = fmt.Sprintf(`{"name":"%s","format":"%s"}`, bucketName, format) + } req, err := http.NewRequest(http.MethodPut, endpoint, strings.NewReader(reqBody)) if err != nil { diff --git a/test/s3tables/catalog/issue_9103_test.go b/test/s3tables/catalog/issue_9103_test.go index 4e00ab8f5..be3dddbd1 100644 --- a/test/s3tables/catalog/issue_9103_test.go +++ b/test/s3tables/catalog/issue_9103_test.go @@ -52,7 +52,7 @@ func TestIssue9103_ConfigDoesNotVendWarehousePrefix(t *testing.T) { env := sharedEnv bucketName := "warehouse-9103cfg-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") warehouse := fmt.Sprintf("s3://%s/", bucketName) u := fmt.Sprintf("%s/v1/config?warehouse=%s", env.IcebergURL(), url.QueryEscape(warehouse)) @@ -88,7 +88,7 @@ func TestIssue9103_BareNamespacesListMissesNamespaceInAttachedBucket(t *testing. env := sharedEnv bucketName := "warehouse-9103ns-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") namespace := "ovirt" status, _, err := doIcebergJSONRequest(env, http.MethodPost, diff --git a/test/s3tables/catalog/lance_client_test.go b/test/s3tables/catalog/lance_client_test.go new file mode 100644 index 000000000..e44d03206 --- /dev/null +++ b/test/s3tables/catalog/lance_client_test.go @@ -0,0 +1,63 @@ +package catalog + +import ( + "fmt" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" + "os" + "os/exec" + "path/filepath" + "testing" +) + +// TestLanceNamespaceRealClient drives the namespace with the Lance client rather +// than hand-built HTTP. The Go tests beside this one cover the catalog surface; +// what this adds is that the location and storage_options the namespace vends +// are actually enough to write and read a dataset, which needs the S3 layout +// guard, the endpoint and the credentials all to be right at once. +// +// To run manually: +// +// cd test/s3tables/catalog +// docker build -t lance-namespace-test -f Dockerfile.lance . +func TestLanceNamespaceRealClient(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + + env := sharedEnv + if !env.dockerAvailable { + t.Skip("Docker not available, skipping Lance client integration test") + } + + bucketName := "lance-client-test-" + randomSuffix() + createTableBucket(t, env, bucketName, s3tables.FormatLance) + + testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog") + + buildCmd := exec.Command("docker", "build", "-t", "lance-namespace-test", "-f", "Dockerfile.lance", ".") + buildCmd.Dir = testDir + if out, err := buildCmd.CombinedOutput(); err != nil { + t.Fatalf("Failed to build test image: %v\n%s", err, string(out)) + } + + namespaceURL := fmt.Sprintf("http://host.docker.internal:%d", env.lancePort) + s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port) + + cmd := exec.Command("docker", "run", "--rm", + "--add-host", "host.docker.internal:host-gateway", + "-v", fmt.Sprintf("%s:/app:ro", testDir), + "lance-namespace-test", + "python3", "/app/test_lance_namespace.py", + "--namespace-url", namespaceURL, + "--s3-endpoint", s3Endpoint, + "--bucket", bucketName, + ) + cmd.Dir = testDir + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + + t.Logf("Running Lance client test against %s", namespaceURL) + if err := cmd.Run(); err != nil { + t.Errorf("Lance client test failed: %v", err) + } +} diff --git a/test/s3tables/catalog/lance_namespace_test.go b/test/s3tables/catalog/lance_namespace_test.go new file mode 100644 index 000000000..acf7dba96 --- /dev/null +++ b/test/s3tables/catalog/lance_namespace_test.go @@ -0,0 +1,378 @@ +// Integration tests for the Lance Namespace REST server, driven against a live +// gateway rather than an in-memory filer. The bugs this surface has produced - +// a deregister that deleted the dataset, an S3 door that refused every Lance +// file - all looked fine against a fake. +package catalog + +import ( + "bytes" + "encoding/json" + "fmt" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" + "io" + "net/http" + "strings" + "sync" + "testing" +) + +// lanceCall posts to the Lance namespace and returns the status and body. +func lanceCall(t *testing.T, env *TestEnvironment, method, path, body string) (int, []byte) { + t.Helper() + req, err := http.NewRequest(method, env.LanceURL()+path, strings.NewReader(body)) + if err != nil { + t.Fatalf("build request: %v", err) + } + req.Header.Set("Content-Type", "application/json") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("%s %s: %v", method, path, err) + } + defer resp.Body.Close() + payload, _ := io.ReadAll(resp.Body) + return resp.StatusCode, payload +} + +func lanceMust(t *testing.T, env *TestEnvironment, method, path, body string, want int) []byte { + t.Helper() + status, payload := lanceCall(t, env, method, path, body) + if status != want { + t.Fatalf("%s %s = %d (%s), want %d", method, path, status, payload, want) + } + return payload +} + +// filerEntryExists reports whether a path exists on storage, which is how these +// tests tell "the catalog forgot the table" from "the data is gone". +func filerEntryExists(t *testing.T, env *TestEnvironment, path string) bool { + t.Helper() + url := fmt.Sprintf("http://127.0.0.1:%d%s", env.filerPort, path) + resp, err := http.Get(url) + if err != nil { + t.Fatalf("filer GET %s: %v", path, err) + } + defer resp.Body.Close() + io.Copy(io.Discard, resp.Body) + return resp.StatusCode == http.StatusOK +} + +// lanceTestBucket makes a uniquely named table bucket of the given format. Most +// tests want LANCE; the one that checks what the Lance surface hides needs an +// Iceberg bucket, because a declared bucket holds one format only. +func lanceTestBucket(t *testing.T, env *TestEnvironment, prefix, format string) string { + t.Helper() + bucket := prefix + "-" + randomSuffix() + createTableBucket(t, env, bucket, format) + return bucket +} + +func TestLanceNamespaceLifecycle(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + env := sharedEnv + bucket := lanceTestBucket(t, env, "lance-ns", s3tables.FormatLance) + + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/create", `{}`, http.StatusOK) + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/exists", `{}`, http.StatusOK) + + // The root lists table buckets, which is the first namespace level here. + var roots struct { + Namespaces []string `json:"namespaces"` + } + if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK), &roots); err != nil { + t.Fatalf("decode root listing: %v", err) + } + found := false + for _, name := range roots.Namespaces { + if name == bucket { + found = true + } + } + if !found { + t.Fatalf("root listing %v does not include %s", roots.Namespaces, bucket) + } + + var children struct { + Namespaces []string `json:"namespaces"` + } + if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"/list", "", http.StatusOK), &children); err != nil { + t.Fatalf("decode namespace listing: %v", err) + } + if len(children.Namespaces) != 1 || children.Namespaces[0] != "sales" { + t.Fatalf("namespace listing = %v, want [sales]", children.Namespaces) + } + + // A namespace is never created as a side effect of naming one inside a + // bucket that does not exist. + status, _ := lanceCall(t, env, http.MethodPost, "/v1/namespace/nosuchbucket$ns/create", `{}`) + if status != http.StatusNotFound { + t.Fatalf("create under a missing bucket = %d, want 404", status) + } +} + +func TestLanceTableLifecyclePreservesData(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + env := sharedEnv + bucket := lanceTestBucket(t, env, "lance-tbl", s3tables.FormatLance) + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK) + + table := "/v1/table/" + bucket + "$ml$vectors" + var declared struct { + Location string `json:"location"` + StorageOptions map[string]string `json:"storage_options"` + } + if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil { + t.Fatalf("decode declare: %v", err) + } + want := fmt.Sprintf("s3://%s/ml/vectors", bucket) + if declared.Location != want { + t.Fatalf("declared location = %q, want %q", declared.Location, want) + } + + datasetPath := fmt.Sprintf("/buckets/%s/ml/vectors/", bucket) + if !filerEntryExists(t, env, datasetPath) { + t.Fatal("declare did not create the dataset directory") + } + + var listed struct { + Tables []string `json:"tables"` + } + if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$ml/table/list", "", http.StatusOK), &listed); err != nil { + t.Fatalf("decode listing: %v", err) + } + if len(listed.Tables) != 1 || listed.Tables[0] != bucket+"$ml$vectors" { + t.Fatalf("table listing = %v, want the full identifier", listed.Tables) + } + + // Deregistering forgets the table but keeps every byte. The catalog entry is + // the dataset directory, so a drop of the entry would take the data too. + lanceMust(t, env, http.MethodPost, table+"/deregister", `{}`, http.StatusOK) + lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusNotFound) + if !filerEntryExists(t, env, datasetPath) { + t.Fatal("deregister deleted the dataset") + } + + registerBody := fmt.Sprintf(`{"location":%q}`, want) + lanceMust(t, env, http.MethodPost, table+"/register", registerBody, http.StatusOK) + lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusOK) + + // Dropping is the operation that does remove the data. + lanceMust(t, env, http.MethodPost, table+"/drop", `{}`, http.StatusOK) + if filerEntryExists(t, env, datasetPath) { + t.Fatal("drop left the dataset behind") + } +} + +// A Lance client must never resolve an Iceberg table's location, or it writes a +// dataset over a table another engine owns. +func TestLanceHidesIcebergTables(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + env := sharedEnv + // An Iceberg bucket, because a bucket that declares LANCE cannot hold an + // Iceberg table at all now. The invariant still matters from this side: the + // Lance surface can be pointed at any bucket, and must not describe or list + // a table whose format it does not serve. + bucket := lanceTestBucket(t, env, "lance-mixed", s3tables.FormatIceberg) + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$mixed/create", `{}`, http.StatusOK) + + createIcebergTable(t, env, bucket, "mixed", "ledger") + + status, _ := lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/describe", `{}`) + if status != http.StatusNotFound { + t.Fatalf("describing an iceberg table through lance = %d, want 404", status) + } + status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/declare", `{}`) + if status != http.StatusConflict { + t.Fatalf("declaring over an iceberg table = %d, want 409", status) + } + // And a new Lance table cannot be smuggled in beside it either. + status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$vectors/declare", `{}`) + if status != http.StatusConflict { + t.Fatalf("declaring a lance table in an iceberg bucket = %d, want 409", status) + } + + var listed struct { + Tables []string `json:"tables"` + } + if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$mixed/table/list", "", http.StatusOK), &listed); err != nil { + t.Fatalf("decode listing: %v", err) + } + if len(listed.Tables) != 0 { + t.Fatalf("lance listing shows iceberg tables: %v", listed.Tables) + } +} + +// The S3 door validates every object written into a table bucket. A Lance +// dataset's files have to get through it, or the catalog is decorative. +func TestLanceFilesAreAcceptedByTheS3Door(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + env := sharedEnv + bucket := lanceTestBucket(t, env, "lance-layout", s3tables.FormatLance) + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK) + lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK) + + accepted := []string{ + "ml/vectors/data/0111111010111000110101116.lance", + "ml/vectors/_versions/18446744073709551614.manifest", + "ml/vectors/_versions/18446744073709551614.manifest-a3a292ad", + "ml/vectors/_transactions/0-ddb27ab7.txn", + "ml/vectors/_indices/85814508-fts/index.idx", + "ml/vectors/.lance-reserved", + } + for _, object := range accepted { + if status := putS3Object(t, env, bucket, object); status != http.StatusOK { + t.Errorf("PUT %s = %d, want 200", object, status) + } + } + + // The guard still rejects files that belong to no table layout. + for _, object := range []string{"ml/vectors/random.txt", "ml/vectors/notadir/x.lance"} { + if status := putS3Object(t, env, bucket, object); status == http.StatusOK { + t.Errorf("PUT %s = 200, want a rejection", object) + } + } +} + +// A Lance commit is a conditional PUT of the next manifest: object_store turns +// PutMode::Create into If-None-Match: *, and lance treats the refusal as a +// commit conflict and rebases. So the whole safety of concurrent writers rests +// on this store answering that precondition atomically. It does, because the +// gateway reduces the header to a filer WriteCondition evaluated at the +// object's owner under a per-path lock. +// +// This is the reason the namespace does not manage versions itself. +func TestLanceCommitPreconditionAdmitsOneWriter(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + env := sharedEnv + bucket := lanceTestBucket(t, env, "lance-commit", s3tables.FormatLance) + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK) + lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK) + + const writers = 8 + var wg sync.WaitGroup + statuses := make([]int, writers) + failures := make([]error, writers) + for i := 0; i < writers; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + // Not t.Fatalf: from a goroutine it ends only that goroutine, and + // the status stays zero, which reads as a bogus response later. + statuses[i], failures[i] = putIfAbsent(env, bucket, "ml/vectors/_versions/1.manifest") + }(i) + } + wg.Wait() + + for i, err := range failures { + if err != nil { + t.Fatalf("writer %d could not reach the gateway: %v", i, err) + } + } + + won, refused := 0, 0 + for _, status := range statuses { + switch status { + case http.StatusOK: + won++ + case http.StatusPreconditionFailed: + refused++ + default: + t.Fatalf("unexpected status %d committing a manifest", status) + } + } + if won != 1 { + t.Fatalf("%d writers committed version 1; exactly one may win, %d were refused", won, refused) + } +} + +// The namespace answers managed_versioning=false: the dataset owns its version +// history, and a reader that never goes through this catalog still sees all of +// it. +func TestLanceDoesNotManageVersions(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + env := sharedEnv + bucket := lanceTestBucket(t, env, "lance-ver", s3tables.FormatLance) + lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK) + table := "/v1/table/" + bucket + "$ml$vectors" + + var declared struct { + ManagedVersioning bool `json:"managed_versioning"` + } + if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil { + t.Fatalf("decode declare: %v", err) + } + if declared.ManagedVersioning { + t.Fatal("managed_versioning must be false; the dataset owns its versions") + } + + // The version ops answer with the spec's Unsupported code rather than a + // bare 404, so a client that asks learns why. + status, _ := lanceCall(t, env, http.MethodPost, table+"/version/list", `{}`) + if status != http.StatusNotImplemented { + t.Fatalf("version/list = %d, want 501", status) + } +} + +// putIfAbsent writes an object only if the key is free, the way a Lance commit +// does. It returns an error rather than failing the test, so it is safe to call +// from the racing goroutines. +func putIfAbsent(env *TestEnvironment, bucket, object string) (int, error) { + url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object) + req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("manifest"))) + if err != nil { + return 0, err + } + req.Header.Set("If-None-Match", "*") + resp, err := http.DefaultClient.Do(req) + if err != nil { + return 0, err + } + defer resp.Body.Close() + io.Copy(io.Discard, resp.Body) + return resp.StatusCode, nil +} + +// putS3Object writes an object through the S3 gateway and returns the status. +func putS3Object(t *testing.T, env *TestEnvironment, bucket, object string) int { + t.Helper() + url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object) + req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("x"))) + if err != nil { + t.Fatalf("build request: %v", err) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("PUT %s: %v", object, err) + } + defer resp.Body.Close() + io.Copy(io.Discard, resp.Body) + return resp.StatusCode +} + +// createIcebergTable registers an ordinary Iceberg table so the mixed-catalog +// tests have one to be confused by. +func createIcebergTable(t *testing.T, env *TestEnvironment, bucket, namespace, name string) { + t.Helper() + body := fmt.Sprintf(`{"name":%q,"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}}`, name) + url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.IcebergURL(), bucket, namespace) + resp, err := http.Post(url, "application/json", strings.NewReader(body)) + if err != nil { + t.Fatalf("create iceberg table: %v", err) + } + defer resp.Body.Close() + payload, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + t.Fatalf("create iceberg table = %d: %s", resp.StatusCode, payload) + } +} diff --git a/test/s3tables/catalog/pyiceberg_test.go b/test/s3tables/catalog/pyiceberg_test.go index 54f4f84b8..0a9fa6f1d 100644 --- a/test/s3tables/catalog/pyiceberg_test.go +++ b/test/s3tables/catalog/pyiceberg_test.go @@ -34,7 +34,7 @@ func TestPyIcebergRestCatalog(t *testing.T) { // Create the test bucket first bucketName := "pyiceberg-compat-test-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") // Build the test working directory path testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog") @@ -93,7 +93,7 @@ func TestPyIcebergRestCatalogAuthenticated(t *testing.T) { // Create the test bucket first (using unauthenticated request, which works with DefaultAllow) bucketName := "pyiceberg-auth-test-" + randomSuffix() - createTableBucket(t, env, bucketName) + createTableBucket(t, env, bucketName, "") // Build the test working directory path testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog") diff --git a/test/s3tables/catalog/test_lance_namespace.py b/test/s3tables/catalog/test_lance_namespace.py new file mode 100755 index 000000000..1c160add7 --- /dev/null +++ b/test/s3tables/catalog/test_lance_namespace.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Drive the SeaweedFS Lance Namespace with the real Lance client. + +The catalog half can be checked with plain HTTP, and the Go integration tests +do. What only a real client proves is that the location and storage_options it +hands back are enough to write and read a dataset: the layout guard on the S3 +door, the endpoint and allow_http options, and the credentials all have to be +right at once, and a hand-built request checks none of that. +""" + +import argparse +import sys +import warnings + +warnings.filterwarnings("ignore") + +import lance +import lance_namespace as ln +import pyarrow as pa + + +def sample_table(): + return pa.table( + { + "id": pa.array([1, 2, 3, 4]), + "vec": pa.array([[1.0, 2.0]] * 4, type=pa.list_(pa.float32(), 2)), + } + ) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--namespace-url", required=True, help="Lance namespace REST URL") + parser.add_argument("--s3-endpoint", required=True, help="S3 endpoint the dataset lives behind") + parser.add_argument("--bucket", required=True, help="table bucket to use, already created") + parser.add_argument("--access-key", default="any") + parser.add_argument("--secret-key", default="any") + args = parser.parse_args() + + ns = ln.connect("rest", {"uri": args.namespace_url}) + ns.create_namespace(ln.CreateNamespaceRequest(id=[args.bucket, "ml"])) + table_id = [args.bucket, "ml", "vectors"] + + declared = ns.declare_table(ln.DeclareTableRequest(id=table_id)) + print(f"declared {table_id} at {declared.location}") + if not declared.location: + print("FAIL: declare returned no location", file=sys.stderr) + return 1 + + described = ns.describe_table(ln.DescribeTableRequest(id=table_id)) + if described.location != declared.location: + print( + f"FAIL: describe location {described.location} != declare {declared.location}", + file=sys.stderr, + ) + return 1 + + options = dict(described.storage_options or {}) + print(f"storage options from the namespace: {sorted(options)}") + # The endpoint is overridden with the container's view of the same gateway: + # what the namespace vends is correct for its own host, and a test harness + # bound to a wildcard address vends nothing at all. That the namespace vends + # a usable endpoint when it can is covered by the unit tests; what matters + # here is that everything else it hands back is enough to reach the data. + options["aws_endpoint"] = args.s3_endpoint + options["allow_http"] = "true" + # A deployment without STS still needs credentials to sign with. + options.setdefault("aws_access_key_id", args.access_key) + options.setdefault("aws_secret_access_key", args.secret_key) + + lance.write_dataset(sample_table(), described.location, storage_options=options, + mode="overwrite") + dataset = lance.dataset(described.location, storage_options=options) + rows = dataset.count_rows() + print(f"wrote and read back {rows} rows at version {dataset.version}") + if rows != 4: + print(f"FAIL: read back {rows} rows, want 4", file=sys.stderr) + return 1 + + # The table is listed, and a client that skips the namespace entirely can + # still open the dataset by URI. + listed = ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables + if f"{args.bucket}$ml$vectors" not in listed: + print(f"FAIL: {listed} does not contain the table", file=sys.stderr) + return 1 + + if lance.dataset(declared.location, storage_options=options).count_rows() != 4: + print("FAIL: the dataset is not readable straight off its URI", file=sys.stderr) + return 1 + + # Deregistering hides the table and keeps the data, which is the difference + # between it and a drop. + ns.deregister_table(ln.DeregisterTableRequest(id=table_id)) + if f"{args.bucket}$ml$vectors" in ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables: + print("FAIL: a deregistered table is still listed", file=sys.stderr) + return 1 + if lance.dataset(declared.location, storage_options=options).count_rows() != 4: + print("FAIL: deregister destroyed the dataset", file=sys.stderr) + return 1 + + print("PASS") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test/testutil/ports.go b/test/testutil/ports.go index 103f5a440..cf14c47f4 100644 --- a/test/testutil/ports.go +++ b/test/testutil/ports.go @@ -38,6 +38,7 @@ var miniDefaultPorts = []int{ 9340, // volume.port 8333, // s3.port 8181, // s3.port.iceberg + 9101, // s3.port.lance 7333, // webdav.port 23646, // admin.port } diff --git a/weed/admin/dash/admin_server.go b/weed/admin/dash/admin_server.go index 9643d8b01..137a976f0 100644 --- a/weed/admin/dash/admin_server.go +++ b/weed/admin/dash/admin_server.go @@ -156,11 +156,12 @@ type AdminServer struct { s3TablesManager *s3tables.Manager icebergPort int + lancePort int } // Type definitions moved to types.go -func NewAdminServer(masters string, filerGroup string, templateFS http.FileSystem, dataDir string, icebergPort int) *AdminServer { +func NewAdminServer(masters string, filerGroup string, templateFS http.FileSystem, dataDir string, icebergPort, lancePort int) *AdminServer { grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.admin") // Create master client with multiple master support @@ -196,6 +197,7 @@ func NewAdminServer(masters string, filerGroup string, templateFS http.FileSyste collectionStatsCacheThreshold: defaultStatsCacheTimeout, s3TablesManager: newS3TablesManager(), icebergPort: icebergPort, + lancePort: lancePort, pluginLock: lockManager, adminPresenceLock: presenceLock, bgCancel: bgCancel, diff --git a/weed/admin/dash/iceberg_data_preview.go b/weed/admin/dash/iceberg_data_preview.go index adb7a9e3b..1c9e90809 100644 --- a/weed/admin/dash/iceberg_data_preview.go +++ b/weed/admin/dash/iceberg_data_preview.go @@ -14,6 +14,7 @@ import ( "github.com/apache/iceberg-go" "github.com/parquet-go/parquet-go" + "github.com/seaweedfs/seaweedfs/weed/admin/plugin" "github.com/seaweedfs/seaweedfs/weed/filer" "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" @@ -27,6 +28,10 @@ const ( icebergPreviewMaxListed = 500 icebergPreviewMaxCellChars = 200 icebergPreviewMaxMetaBytes = 64 << 20 + + // workerPreviewTimeout bounds one round trip to a worker. A page waiting on + // a worker that has gone quiet should say so rather than hang. + workerPreviewTimeout = 15 * time.Second ) type IcebergDataFileInfo struct { @@ -60,6 +65,10 @@ type IcebergDataPreviewData struct { PreviewNotes []string `json:"preview_notes,omitempty"` PreviewError string `json:"preview_error,omitempty"` LastUpdated time.Time `json:"last_updated"` + // Format is what the catalog recorded. Anything but ICEBERG means the rows + // below, if any, came from a worker rather than from metadata this server read. + Format string `json:"format,omitempty"` + PreviewedBy string `json:"previewed_by,omitempty"` } // GetIcebergTableDataPreview walks the selected snapshot's manifests and reads @@ -92,6 +101,11 @@ func (s *AdminServer) GetIcebergTableDataPreview(ctx context.Context, catalogNam if err := s.executeS3TablesOperation(ctx, "GetTable", req, &resp); err != nil { return data, err } + data.Format = resp.Format + if !strings.EqualFold(resp.Format, s3tables.FormatIceberg) && resp.Format != "" { + s.applyWorkerPreview(ctx, &data, bucketArn, namespaceParts, tableName, rowLimit) + return data, nil + } if resp.Metadata == nil || len(resp.Metadata.FullMetadata) == 0 { data.PreviewError = "Table has no Iceberg metadata." return data, nil @@ -467,3 +481,56 @@ func (w *sliceWriter) Write(p []byte) (int, error) { w.n += c return c, nil } + +// applyWorkerPreview asks the worker that last described this table for sample +// rows. Admin has no reader for a format it does not implement, so this is the +// only way the page shows anything but a location. +// +// The rows are fetched, never cached: they are the table's data rather than a +// description of it, and a stale copy sitting in admin would be worse than +// asking. +func (s *AdminServer) applyWorkerPreview(ctx context.Context, data *IcebergDataPreviewData, bucketArn string, namespaceParts []string, tableName string, rowLimit int) { + plugin := s.GetPlugin() + if plugin == nil { + data.PreviewError = fmt.Sprintf("Reading a %s table needs a plugin worker, and none is configured.", data.Format) + return + } + bucketName, err := s3tables.ParseBucketNameFromARN(bucketArn) + if err != nil { + data.PreviewError = err.Error() + return + } + + objectID := append(append([]string{bucketName}, namespaceParts...), tableName) + requestCtx, cancel := context.WithTimeout(ctx, workerPreviewTimeout) + defer cancel() + + response, err := plugin.RequestObjectPreview(requestCtx, objectID, data.Format, rowLimit) + if err != nil { + data.PreviewError = fmt.Sprintf("Could not read this %s table: %v", data.Format, err) + return + } + + data.Columns = response.Columns + data.Rows = make([][]string, 0, len(response.Rows)) + for _, row := range response.Rows { + cells := make([]string, len(row.Values)) + for i, value := range row.Values { + cells[i] = truncateCell(value) + } + data.Rows = append(data.Rows, cells) + } + data.TotalRecords = response.TotalRows + data.PreviewedBy = pluginWorkerForObject(plugin, objectID, data.Format) + if int64(len(data.Rows)) < response.TotalRows { + data.PreviewNotes = append(data.PreviewNotes, fmt.Sprintf("Showing %d of %d rows.", len(data.Rows), response.TotalRows)) + } +} + +// pluginWorkerForObject names the worker that answered, for the page to show. +func pluginWorkerForObject(p *plugin.Plugin, objectID []string, format string) string { + if observed, ok := p.Observations().GetFormat(objectID, format); ok { + return observed.WorkerID + } + return "" +} diff --git a/weed/admin/dash/plugin_api.go b/weed/admin/dash/plugin_api.go index e21a4e1d1..c38b0aa26 100644 --- a/weed/admin/dash/plugin_api.go +++ b/weed/admin/dash/plugin_api.go @@ -982,3 +982,27 @@ func parsePositiveInt(raw string, defaultValue int) int { } // cloneConfigValueMap is now exported by the plugin package as CloneConfigValueMap + +// GetPluginObservationsAPI returns what workers last reported about the objects +// they inspected. Accepts an optional ?format= filter. +// +// These are cached, not live: a worker reports what it saw when it last looked, +// and the timestamp is served with each one so a reader can judge the age. +func (s *AdminServer) GetPluginObservationsAPI(w http.ResponseWriter, r *http.Request) { + plugin := s.GetPlugin() + if plugin == nil { + writeJSON(w, http.StatusOK, []interface{}{}) + return + } + + formatFilter := strings.TrimSpace(r.URL.Query().Get("format")) + observed := plugin.Observations().List() + payload := make([]interface{}, 0, len(observed)) + for _, o := range observed { + if formatFilter != "" && !strings.EqualFold(o.Format, formatFilter) { + continue + } + payload = append(payload, o) + } + writeJSON(w, http.StatusOK, payload) +} diff --git a/weed/admin/dash/s3tables_management.go b/weed/admin/dash/s3tables_management.go index 21aca5aed..d1f1c641b 100644 --- a/weed/admin/dash/s3tables_management.go +++ b/weed/admin/dash/s3tables_management.go @@ -25,6 +25,7 @@ type S3TablesBucketsData struct { Buckets []S3TablesBucketSummary `json:"buckets"` TotalBuckets int `json:"total_buckets"` IcebergPort int `json:"iceberg_port"` + LancePort int `json:"lance_port"` LastUpdated time.Time `json:"last_updated"` } @@ -33,29 +34,39 @@ type S3TablesBucketSummary struct { Name string `json:"name"` OwnerAccountID string `json:"ownerAccountId"` CreatedAt time.Time `json:"createdAt"` + // Format is empty for a bucket created before formats were declared. Such a + // bucket takes tables of either format, which is what it always did. + Format string `json:"format,omitempty"` } type S3TablesNamespacesData struct { Username string `json:"username"` BucketARN string `json:"bucket_arn"` + BucketFormat string `json:"bucket_format,omitempty"` Namespaces []s3tables.NamespaceSummary `json:"namespaces"` TotalNamespaces int `json:"total_namespaces"` LastUpdated time.Time `json:"last_updated"` } type S3TablesTablesData struct { - Username string `json:"username"` - BucketARN string `json:"bucket_arn"` - Namespace string `json:"namespace"` - Tables []s3tables.TableSummary `json:"tables"` - TotalTables int `json:"total_tables"` - LastUpdated time.Time `json:"last_updated"` + Username string `json:"username"` + BucketARN string `json:"bucket_arn"` + BucketFormat string `json:"bucket_format,omitempty"` + Namespace string `json:"namespace"` + Tables []s3tables.TableSummary `json:"tables"` + TotalTables int `json:"total_tables"` + // ObservedRows holds what a worker last counted, by table name, for formats + // this server cannot read itself. A table nothing has looked at is absent + // rather than zero: those are different facts. + ObservedRows map[string]string `json:"observed_rows,omitempty"` + LastUpdated time.Time `json:"last_updated"` } type tableBucketMetadata struct { Name string `json:"name"` CreatedAt time.Time `json:"createdAt"` OwnerAccountID string `json:"ownerAccountId"` + Format string `json:"format,omitempty"` } // S3Tables manager helpers @@ -137,6 +148,7 @@ func (s *AdminServer) GetS3TablesBucketsData(ctx context.Context) (S3TablesBucke Name: entry.Entry.Name, OwnerAccountID: metadata.OwnerAccountID, CreatedAt: metadata.CreatedAt, + Format: metadata.Format, }) } return nil @@ -148,10 +160,60 @@ func (s *AdminServer) GetS3TablesBucketsData(ctx context.Context) (S3TablesBucke Buckets: buckets, TotalBuckets: len(buckets), IcebergPort: s.icebergPort, + LancePort: s.lancePort, LastUpdated: time.Now(), }, nil } +// observedRowCounts collects what workers last reported for these tables. For a +// format admin cannot read, this is the only row count that exists. +func (s *AdminServer) observedRowCounts(bucketArn string, namespaceParts []string, tables []s3tables.TableSummary) map[string]string { + plugin := s.GetPlugin() + if plugin == nil || len(tables) == 0 { + return nil + } + bucketName, err := s3tables.ParseBucketNameFromARN(bucketArn) + if err != nil { + return nil + } + counts := make(map[string]string) + for _, table := range tables { + objectID := append(append([]string{bucketName}, namespaceParts...), table.Name) + if observed, ok := plugin.Observations().GetFormat(objectID, table.Format); ok { + if rows := observed.AttributeString("rows"); rows != "" { + counts[table.Name] = rows + } + } + } + if len(counts) == 0 { + return nil + } + return counts +} + +// catalogPortForFormat is the port serving a format, or 0 when this cluster +// does not run that catalog. +func (s *AdminServer) catalogPortForFormat(format string) int { + if strings.EqualFold(format, s3tables.FormatLance) { + return s.lancePort + } + return s.icebergPort +} + +// tableBucketFormat reports what the bucket says it holds, or "" for one made +// before the declaration existed. A page inside a bucket asks so it can label +// itself and constrain what can be created; failing to read it is not worth +// failing the page over, so it degrades to undeclared. +func (s *AdminServer) tableBucketFormat(ctx context.Context, bucketArn string) string { + var resp s3tables.GetTableBucketResponse + req := &s3tables.GetTableBucketRequest{TableBucketARN: bucketArn} + if err := s.executeS3TablesOperation(ctx, "GetTableBucket", req, &resp); err != nil { + glog.V(1).Infof("S3Tables: failed to read format of %s: %v", bucketArn, err) + return "" + } + return resp.Format +} + func (s *AdminServer) GetS3TablesNamespacesData(ctx context.Context, bucketArn string) (S3TablesNamespacesData, error) { var resp s3tables.ListNamespacesResponse req := &s3tables.ListNamespacesRequest{TableBucketARN: bucketArn, MaxNamespaces: s3TablesAdminListLimit} @@ -160,6 +222,7 @@ func (s *AdminServer) GetS3TablesNamespacesData(ctx context.Context, bucketArn s } return S3TablesNamespacesData{ BucketARN: bucketArn, + BucketFormat: s.tableBucketFormat(ctx, bucketArn), Namespaces: resp.Namespaces, TotalNamespaces: len(resp.Namespaces), LastUpdated: time.Now(), @@ -180,13 +243,16 @@ func (s *AdminServer) GetS3TablesTablesData(ctx context.Context, bucketArn, name if err := s.executeS3TablesOperation(ctx, "ListTables", req, &resp); err != nil { return S3TablesTablesData{}, err } - return S3TablesTablesData{ - BucketARN: bucketArn, - Namespace: namespace, - Tables: resp.Tables, - TotalTables: len(resp.Tables), - LastUpdated: time.Now(), - }, nil + data := S3TablesTablesData{ + BucketARN: bucketArn, + BucketFormat: s.tableBucketFormat(ctx, bucketArn), + Namespace: namespace, + Tables: resp.Tables, + TotalTables: len(resp.Tables), + LastUpdated: time.Now(), + } + data.ObservedRows = s.observedRowCounts(bucketArn, ns, resp.Tables) + return data, nil } // Iceberg Catalog data providers @@ -299,9 +365,75 @@ func (s *AdminServer) GetIcebergTableDetailsData(ctx context.Context, catalogNam } applyIcebergMetadata(resp.Metadata, &details) + s.applyWorkerObservation(&details, bucketArn, namespaceParts, resp.Name) return details, nil } +// applyWorkerObservation fills in what a plugin worker last reported about a +// table this server cannot read itself. Only the catalog knows a Lance table +// exists; only a worker with the format's runtime can say what is in it, so +// without this the page has nothing to show but a location. +// +// The observation is cached, not live, which is why the page carries the time +// and the worker that took it. +func (s *AdminServer) applyWorkerObservation(details *IcebergTableDetailsData, bucketArn string, namespaceParts []string, tableName string) { + if len(details.SchemaFields) > 0 { + return + } + plugin := s.GetPlugin() + if plugin == nil { + return + } + bucketName, err := s3tables.ParseBucketNameFromARN(bucketArn) + if err != nil { + return + } + objectID := append(append([]string{bucketName}, namespaceParts...), tableName) + observed, ok := plugin.Observations().GetFormat(objectID, details.Format) + if !ok { + return + } + + details.ObservedBy = observed.WorkerID + details.ObservedAt = observed.ObservedAt + details.SchemaFields = observationSchemaFields(observed.AttributeString("schema")) + for _, name := range []string{"rows", "fragments", "versions"} { + if value := observed.AttributeString(name); value != "" { + details.Properties = append(details.Properties, IcebergPropertyInfo{Key: name, Value: value}) + } + } +} + +// observationSchemaFields parses the schema a worker reported. A schema it +// could not render is not worth failing the page over. +func observationSchemaFields(schema string) []IcebergSchemaFieldInfo { + if schema == "" { + return nil + } + var reported []struct { + Name string `json:"name"` + Type string `json:"type"` + Nullable bool `json:"nullable"` + } + if err := json.Unmarshal([]byte(schema), &reported); err != nil { + return nil + } + fields := make([]IcebergSchemaFieldInfo, 0, len(reported)) + for i, field := range reported { + encoded, err := json.Marshal(field.Type) + if err != nil { + continue + } + fields = append(fields, IcebergSchemaFieldInfo{ + ID: i + 1, + Name: field.Name, + Type: encoded, + Required: !field.Nullable, + }) + } + return fields +} + type icebergFullMetadata struct { FormatVersion int `json:"format-version"` TableUUID string `json:"table-uuid"` @@ -582,9 +714,10 @@ func (s *AdminServer) CreateS3TablesBucket(w http.ResponseWriter, r *http.Reques return } var req struct { - Name string `json:"name"` - Tags map[string]string `json:"tags"` - Owner string `json:"owner"` + Name string `json:"name"` + Tags map[string]string `json:"tags"` + Owner string `json:"owner"` + Format string `json:"format"` } if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil { writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error()) @@ -605,7 +738,24 @@ func (s *AdminServer) CreateS3TablesBucket(w http.ResponseWriter, r *http.Reques return } } - createReq := &s3tables.CreateTableBucketRequest{Name: req.Name, Tags: req.Tags} + format := s3tables.FormatIceberg + if req.Format != "" { + normalized, ok := s3tables.NormalizeFormat(req.Format) + if !ok { + writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Unsupported format %q", req.Format)) + return + } + format = normalized + } + // A bucket of a format this cluster does not serve is a bucket no client can + // reach. The picker disables the option; refuse it here too, since the API + // is reachable without the page. + if port := s.catalogPortForFormat(format); port == 0 { + writeJSONError(w, http.StatusBadRequest, + fmt.Sprintf("No %s endpoint is configured, so a %s bucket would be unreachable", format, format)) + return + } + createReq := &s3tables.CreateTableBucketRequest{Name: req.Name, Tags: req.Tags, Format: format} var resp s3tables.CreateTableBucketResponse if err := s.executeS3TablesOperation(r.Context(), "CreateTableBucket", createReq, &resp); err != nil { writeS3TablesError(w, err) diff --git a/weed/admin/dash/types.go b/weed/admin/dash/types.go index 708d99de1..a161f8bf7 100644 --- a/weed/admin/dash/types.go +++ b/weed/admin/dash/types.go @@ -778,4 +778,8 @@ type IcebergTableDetailsData struct { TotalSizeBytes int64 `json:"total_size_bytes"` HasTotalSize bool `json:"has_total_size"` MetadataError string `json:"metadata_error,omitempty"` + // Set when the details came from a plugin worker rather than from metadata + // this server can read, so the page can say whose account it is and when. + ObservedBy string `json:"observed_by,omitempty"` + ObservedAt time.Time `json:"observed_at,omitempty"` } diff --git a/weed/admin/handlers/admin_handlers.go b/weed/admin/handlers/admin_handlers.go index 73029f260..85095ef37 100644 --- a/weed/admin/handlers/admin_handlers.go +++ b/weed/admin/handlers/admin_handlers.go @@ -266,6 +266,7 @@ func (h *AdminHandlers) registerAPIRoutes(api *mux.Router, enforceWrite bool) { pluginApi.HandleFunc("/jobs/{jobId}/detail", h.adminServer.GetPluginJobDetailAPI).Methods(http.MethodGet) pluginApi.HandleFunc("/activities", h.adminServer.GetPluginActivitiesAPI).Methods(http.MethodGet) pluginApi.HandleFunc("/scheduler-states", h.adminServer.GetPluginSchedulerStatesAPI).Methods(http.MethodGet) + pluginApi.HandleFunc("/observations", h.adminServer.GetPluginObservationsAPI).Methods(http.MethodGet) pluginApi.HandleFunc("/scheduler-status", h.adminServer.GetPluginSchedulerStatusAPI).Methods(http.MethodGet) pluginApi.HandleFunc("/job-types/{jobType}/descriptor", h.adminServer.GetPluginJobTypeDescriptorAPI).Methods(http.MethodGet) pluginApi.HandleFunc("/job-types/{jobType}/schema", h.adminServer.RequestPluginJobTypeSchemaAPI).Methods(http.MethodPost) diff --git a/weed/admin/plugin/observations.go b/weed/admin/plugin/observations.go new file mode 100644 index 000000000..67f4b2a0e --- /dev/null +++ b/weed/admin/plugin/observations.go @@ -0,0 +1,179 @@ +package plugin + +import ( + "fmt" + "sort" + "strconv" + "strings" + "sync" + "time" + + "github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb" +) + +// maxObservations caps the store. Observations are a convenience for display, +// so a cluster with more objects than this loses the oldest rather than growing +// admin's memory without limit. +const maxObservations = 10000 + +// observationKey identifies one observed object across workers. +func observationKey(objectID []string) string { + return strings.Join(objectID, "\x1f") +} + +// ObservationStore keeps the last thing a worker said about each object. +// +// It is deliberately not authoritative: a worker reports what it saw when it +// last looked, and admin serves that back with its timestamp so a reader can +// judge how stale it is. Nothing schedules work from it. +type ObservationStore struct { + mu sync.RWMutex + entries map[string]*Observation +} + +// Observation is one object as a worker last reported it. +type Observation struct { + ObjectID []string `json:"object_id"` + ObjectKind string `json:"object_kind"` + Format string `json:"format"` + // Attributes are flattened on the way in: they exist to be displayed and + // served as JSON, and the typed form is the worker's business. + Attributes map[string]interface{} `json:"attributes"` + JobType string `json:"job_type"` + WorkerID string `json:"worker_id"` + ObservedAt time.Time `json:"observed_at"` +} + +func NewObservationStore() *ObservationStore { + return &ObservationStore{entries: make(map[string]*Observation)} +} + +// Record stores what one worker reported, replacing whatever was there for the +// same object. A later observation from a different worker still wins: the +// newest look at an object is the useful one. +func (s *ObservationStore) Record(workerID string, report *plugin_pb.WorkerObservations) { + if report == nil { + return + } + s.mu.Lock() + defer s.mu.Unlock() + + for _, observed := range report.Observations { + if observed == nil || len(observed.ObjectId) == 0 { + continue + } + at := time.Now() + if observed.ObservedAt != nil { + at = observed.ObservedAt.AsTime() + } + s.entries[observationKey(observed.ObjectId)] = &Observation{ + ObjectID: observed.ObjectId, + ObjectKind: observed.ObjectKind, + Format: observed.Format, + Attributes: flattenAttributes(observed.Attributes), + JobType: report.JobType, + WorkerID: workerID, + ObservedAt: at, + } + } + s.evictOldest() +} + +// evictOldest trims the store back under the cap. Called with the lock held. +func (s *ObservationStore) evictOldest() { + if len(s.entries) <= maxObservations { + return + } + keys := make([]string, 0, len(s.entries)) + for key := range s.entries { + keys = append(keys, key) + } + sort.Slice(keys, func(i, j int) bool { + return s.entries[keys[i]].ObservedAt.Before(s.entries[keys[j]].ObservedAt) + }) + for _, key := range keys[:len(s.entries)-maxObservations] { + delete(s.entries, key) + } +} + +// flattenAttributes unwraps the ConfigValue envelope that configValueMapToPlain +// preserves. An observation exists to be read, and {"int64_value":"16"} is not +// a number anyone wants to render; a protojson int64 also arrives as a string, +// so it is converted back. +func flattenAttributes(values map[string]*plugin_pb.ConfigValue) map[string]interface{} { + plain := configValueMapToPlain(values) + if plain == nil { + return nil + } + flat := make(map[string]interface{}, len(plain)) + for name, value := range plain { + wrapper, ok := value.(map[string]interface{}) + if !ok || len(wrapper) != 1 { + flat[name] = value + continue + } + for kind, inner := range wrapper { + if kind == "int64_value" { + if text, ok := inner.(string); ok { + if parsed, err := strconv.ParseInt(text, 10, 64); err == nil { + flat[name] = parsed + continue + } + } + } + flat[name] = inner + } + } + return flat +} + +// Get returns the last observation of one object, if any. +func (s *ObservationStore) Get(objectID []string) (*Observation, bool) { + s.mu.RLock() + defer s.mu.RUnlock() + observed, ok := s.entries[observationKey(objectID)] + return observed, ok +} + +// GetFormat returns the last observation of one object only when it describes +// the format asked for. A table can be dropped and remade in another format at +// the same path, and the stale observation would then describe something that +// no longer exists. +func (s *ObservationStore) GetFormat(objectID []string, format string) (*Observation, bool) { + observed, ok := s.Get(objectID) + if !ok || !strings.EqualFold(observed.Format, format) { + return nil, false + } + return observed, true +} + +// List returns every observation, newest first. +func (s *ObservationStore) List() []*Observation { + s.mu.RLock() + defer s.mu.RUnlock() + all := make([]*Observation, 0, len(s.entries)) + for _, observed := range s.entries { + all = append(all, observed) + } + sort.Slice(all, func(i, j int) bool { + return all[i].ObservedAt.After(all[j].ObservedAt) + }) + return all +} + +// AttributeString renders one attribute for display, or "" when it is absent. +func (o *Observation) AttributeString(name string) string { + if o == nil { + return "" + } + value, ok := o.Attributes[name] + if !ok || value == nil { + return "" + } + return fmt.Sprintf("%v", value) +} + +// Observations exposes the store so admin handlers can serve it. +func (r *Plugin) Observations() *ObservationStore { + return r.observations +} diff --git a/weed/admin/plugin/observations_test.go b/weed/admin/plugin/observations_test.go new file mode 100644 index 000000000..f719339f5 --- /dev/null +++ b/weed/admin/plugin/observations_test.go @@ -0,0 +1,126 @@ +package plugin + +import ( + "testing" + "time" + + "github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb" + "google.golang.org/protobuf/types/known/timestamppb" +) + +func intValue(v int64) *plugin_pb.ConfigValue { + return &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: v}} +} + +func stringValue(v string) *plugin_pb.ConfigValue { + return &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_StringValue{StringValue: v}} +} + +// An observation exists to be rendered, so the ConfigValue envelope comes off +// on the way in. A page asking for "fragments" wants 16, not a wrapper holding +// the string "16". +func TestObservationAttributesAreFlattened(t *testing.T) { + store := NewObservationStore() + store.Record("worker-1", &plugin_pb.WorkerObservations{ + JobType: "lance_compact", + Observations: []*plugin_pb.ObjectObservation{{ + ObjectId: []string{"bucket", "ns", "table"}, + ObjectKind: "table", + Format: "LANCE", + Attributes: map[string]*plugin_pb.ConfigValue{ + "fragments": intValue(16), + "schema": stringValue(`[{"name":"id"}]`), + }, + ObservedAt: timestamppb.New(time.Now()), + }}, + }) + + observed, ok := store.Get([]string{"bucket", "ns", "table"}) + if !ok { + t.Fatal("observation was not stored") + } + if got, ok := observed.Attributes["fragments"].(int64); !ok || got != 16 { + t.Fatalf("fragments = %#v, want int64 16", observed.Attributes["fragments"]) + } + if got := observed.AttributeString("fragments"); got != "16" { + t.Fatalf("AttributeString(fragments) = %q, want \"16\"", got) + } + if got := observed.AttributeString("schema"); got != `[{"name":"id"}]` { + t.Fatalf("AttributeString(schema) = %q", got) + } + if observed.WorkerID != "worker-1" || observed.JobType != "lance_compact" { + t.Fatalf("provenance lost: %+v", observed) + } +} + +// The newest look at an object is the useful one, whichever worker took it. +func TestObservationIsReplacedByALaterOne(t *testing.T) { + store := NewObservationStore() + record := func(worker string, fragments int64) { + store.Record(worker, &plugin_pb.WorkerObservations{ + JobType: "lance_compact", + Observations: []*plugin_pb.ObjectObservation{{ + ObjectId: []string{"bucket", "ns", "table"}, + Format: "LANCE", + Attributes: map[string]*plugin_pb.ConfigValue{"fragments": intValue(fragments)}, + ObservedAt: timestamppb.New(time.Now()), + }}, + }) + } + record("worker-1", 16) + record("worker-2", 1) + + observed, _ := store.Get([]string{"bucket", "ns", "table"}) + if got := observed.AttributeString("fragments"); got != "1" { + t.Fatalf("fragments = %q, want the later observation's 1", got) + } + if observed.WorkerID != "worker-2" { + t.Fatalf("worker = %q, want worker-2", observed.WorkerID) + } + if len(store.List()) != 1 { + t.Fatalf("store holds %d entries, want 1", len(store.List())) + } +} + +// Missing attributes read as empty rather than panicking a template. +func TestObservationAttributeStringHandlesAbsence(t *testing.T) { + var absent *Observation + if got := absent.AttributeString("fragments"); got != "" { + t.Fatalf("nil observation returned %q", got) + } + store := NewObservationStore() + store.Record("worker-1", &plugin_pb.WorkerObservations{ + Observations: []*plugin_pb.ObjectObservation{{ + ObjectId: []string{"b", "n", "t"}, + }}, + }) + observed, _ := store.Get([]string{"b", "n", "t"}) + if got := observed.AttributeString("nothing"); got != "" { + t.Fatalf("absent attribute returned %q", got) + } +} + +// A table path can be dropped and remade in another format. The observation left +// behind describes something that is no longer there, so a caller asking about +// the new format must not be handed it. +func TestObservationLookupIsScopedToFormat(t *testing.T) { + store := NewObservationStore() + objectID := []string{"bucket", "ns", "table"} + store.Record("worker-1", &plugin_pb.WorkerObservations{ + Observations: []*plugin_pb.ObjectObservation{{ + ObjectId: objectID, + Format: "LANCE", + ObservedAt: timestamppb.New(time.Now()), + }}, + }) + + if _, ok := store.GetFormat(objectID, "ICEBERG"); ok { + t.Fatal("a LANCE observation was returned for an ICEBERG table") + } + if _, ok := store.GetFormat(objectID, "lance"); !ok { + t.Fatal("format matching must not depend on case") + } + if _, ok := store.GetFormat(objectID, ""); ok { + t.Fatal("an unknown format must not match a recorded one") + } +} diff --git a/weed/admin/plugin/plugin.go b/weed/admin/plugin/plugin.go index 8f1944d8c..e94e0fc79 100644 --- a/weed/admin/plugin/plugin.go +++ b/weed/admin/plugin/plugin.go @@ -97,6 +97,11 @@ type Plugin struct { pendingExecutionMu sync.Mutex pendingExecution map[string]chan *plugin_pb.JobCompleted + pendingPreviewMu sync.Mutex + pendingPreview map[string]chan *plugin_pb.ObjectPreviewResponse + + observations *ObservationStore + jobsMu sync.RWMutex jobs map[string]*TrackedJob // serialize stale job cleanup to avoid duplicate expirations @@ -180,6 +185,8 @@ func New(options Options) (*Plugin, error) { pendingSchema: make(map[string]chan *plugin_pb.ConfigSchemaResponse), pendingDetection: make(map[string]*pendingDetectionState), pendingExecution: make(map[string]chan *plugin_pb.JobCompleted), + pendingPreview: make(map[string]chan *plugin_pb.ObjectPreviewResponse), + observations: NewObservationStore(), nextDetectionAt: make(map[string]time.Time), detectionInFlight: make(map[string]bool), detectorLeases: make(map[string]string), @@ -235,6 +242,13 @@ func (r *Plugin) Shutdown() { } r.pendingSchemaMu.Unlock() + r.pendingPreviewMu.Lock() + for requestID, ch := range r.pendingPreview { + close(ch) + delete(r.pendingPreview, requestID) + } + r.pendingPreviewMu.Unlock() + r.pendingDetectionMu.Lock() for requestID, state := range r.pendingDetection { close(state.complete) @@ -978,6 +992,10 @@ func (r *Plugin) handleWorkerMessage(workerID string, message *plugin_pb.WorkerT r.handleJobProgressUpdate(workerID, body.JobProgressUpdate) case *plugin_pb.WorkerToAdminMessage_JobCompleted: r.handleJobCompleted(body.JobCompleted) + case *plugin_pb.WorkerToAdminMessage_Observations: + r.observations.Record(workerID, body.Observations) + case *plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse: + r.handleObjectPreviewResponse(body.ObjectPreviewResponse) case *plugin_pb.WorkerToAdminMessage_Acknowledge: if !body.Acknowledge.Accepted { glog.Warningf("Plugin worker %s rejected request %s: %s", workerID, body.Acknowledge.RequestId, body.Acknowledge.Message) diff --git a/weed/admin/plugin/preview.go b/weed/admin/plugin/preview.go new file mode 100644 index 000000000..4b03ad579 --- /dev/null +++ b/weed/admin/plugin/preview.go @@ -0,0 +1,105 @@ +package plugin + +import ( + "context" + "fmt" + + "github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb" + "google.golang.org/protobuf/types/known/timestamppb" +) + +// maxPreviewRows caps what admin will ask for. A preview is a look at the +// object, not an export, and the rows cross the control stream. +const maxPreviewRows = 200 + +// RequestObjectPreview asks the worker that last described an object for sample +// rows of it, and waits for the answer. +// +// The worker is chosen from the observation store rather than by job type: the +// one that last looked at this object is the one that can read it, and it says +// so by having reported. That also means a preview is only available once +// detection has run, which the caller should surface rather than hide. +func (r *Plugin) RequestObjectPreview(ctx context.Context, objectID []string, format string, rowLimit int) (*plugin_pb.ObjectPreviewResponse, error) { + if len(objectID) == 0 { + return nil, fmt.Errorf("preview needs an object id") + } + // The conversion happens only inside the bound check, so what reaches the + // wire cannot depend on int being wider than int32 here. + requestedRows := int32(maxPreviewRows) + if rowLimit >= 1 && rowLimit <= maxPreviewRows { + requestedRows = int32(rowLimit) + } + + observed, ok := r.observations.GetFormat(objectID, format) + if !ok { + return nil, fmt.Errorf("no worker has described this object as %s yet", format) + } + if _, connected := r.registry.Get(observed.WorkerID); !connected { + return nil, fmt.Errorf("worker %s is not connected", observed.WorkerID) + } + + requestID, err := newRequestID("preview") + if err != nil { + return nil, err + } + + responseCh := make(chan *plugin_pb.ObjectPreviewResponse, 1) + r.pendingPreviewMu.Lock() + r.pendingPreview[requestID] = responseCh + r.pendingPreviewMu.Unlock() + defer func() { + r.pendingPreviewMu.Lock() + delete(r.pendingPreview, requestID) + r.pendingPreviewMu.Unlock() + }() + + request := &plugin_pb.AdminToWorkerMessage{ + RequestId: requestID, + SentAt: timestamppb.Now(), + Body: &plugin_pb.AdminToWorkerMessage_RequestObjectPreview{ + RequestObjectPreview: &plugin_pb.RequestObjectPreview{ + ObjectId: objectID, + Format: format, + RowLimit: requestedRows, + }, + }, + } + if err := r.sendToWorker(observed.WorkerID, request); err != nil { + return nil, err + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case response, ok := <-responseCh: + if !ok { + return nil, fmt.Errorf("preview request %s interrupted", requestID) + } + if response == nil { + return nil, fmt.Errorf("preview request %s returned nothing", requestID) + } + if !response.Success { + return nil, fmt.Errorf("worker %s could not preview the object: %s", observed.WorkerID, response.ErrorMessage) + } + return response, nil + } +} + +// handleObjectPreviewResponse routes one reply back to whoever is waiting. +func (r *Plugin) handleObjectPreviewResponse(response *plugin_pb.ObjectPreviewResponse) { + if response == nil { + return + } + // Held across the send: Shutdown closes these channels under the same lock, + // and a send that raced it would panic on a closed channel. + r.pendingPreviewMu.Lock() + defer r.pendingPreviewMu.Unlock() + ch := r.pendingPreview[response.RequestId] + if ch == nil { + return + } + select { + case ch <- response: + default: + } +} diff --git a/weed/admin/plugin/preview_test.go b/weed/admin/plugin/preview_test.go new file mode 100644 index 000000000..a5f434580 --- /dev/null +++ b/weed/admin/plugin/preview_test.go @@ -0,0 +1,202 @@ +package plugin + +import ( + "context" + "strings" + "sync" + "testing" + "time" + + "github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb" + "google.golang.org/protobuf/types/known/timestamppb" +) + +func previewPlugin(t *testing.T, workerID string) (*Plugin, *streamSession) { + t.Helper() + pluginSvc, err := New(Options{}) + if err != nil { + t.Fatalf("New plugin error: %v", err) + } + t.Cleanup(pluginSvc.Shutdown) + + pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID}) + session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 4), done: make(chan struct{})} + pluginSvc.putSession(session) + return pluginSvc, session +} + +func observeObject(p *Plugin, workerID string, objectID []string) { + p.observations.Record(workerID, &plugin_pb.WorkerObservations{ + JobType: "lance_compact", + Observations: []*plugin_pb.ObjectObservation{{ + ObjectId: objectID, + Format: "LANCE", + ObservedAt: timestamppb.Now(), + }}, + }) +} + +// The round trip the details page makes: ask the worker that described this +// object, and render what it sends back. +func TestRequestObjectPreviewReturnsTheWorkersRows(t *testing.T) { + t.Parallel() + const workerID = "lance-worker-1" + objectID := []string{"vectors", "ml", "embeddings"} + pluginSvc, session := previewPlugin(t, workerID) + observeObject(pluginSvc, workerID, objectID) + + type result struct { + response *plugin_pb.ObjectPreviewResponse + err error + } + results := make(chan result, 1) + go func() { + response, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 2) + results <- result{response, err} + }() + + var request *plugin_pb.AdminToWorkerMessage + select { + case request = <-session.outgoing: + case <-time.After(2 * time.Second): + t.Fatal("no preview request reached the worker") + } + asked := request.GetRequestObjectPreview() + if asked == nil { + t.Fatalf("expected a preview request, got %T", request.Body) + } + if asked.RowLimit != 2 || asked.Format != "LANCE" { + t.Fatalf("request lost its parameters: %+v", asked) + } + + pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{ + WorkerId: workerID, + Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{ + ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{ + RequestId: request.RequestId, + Success: true, + Columns: []string{"id", "vec"}, + Rows: []*plugin_pb.PreviewRow{{Values: []string{"1", "[0.1, 0.2]"}}}, + TotalRows: 1024, + }, + }, + }) + + select { + case got := <-results: + if got.err != nil { + t.Fatalf("preview failed: %v", got.err) + } + if len(got.response.Rows) != 1 || got.response.Rows[0].Values[1] != "[0.1, 0.2]" { + t.Fatalf("rows did not survive the round trip: %+v", got.response.Rows) + } + if got.response.TotalRows != 1024 { + t.Fatalf("total_rows = %d, want 1024", got.response.TotalRows) + } + case <-time.After(2 * time.Second): + t.Fatal("preview never returned") + } +} + +// A worker that cannot read the object says so, and the page shows its reason +// rather than an empty table. +func TestRequestObjectPreviewSurfacesTheWorkersError(t *testing.T) { + t.Parallel() + const workerID = "lance-worker-1" + objectID := []string{"vectors", "ml", "embeddings"} + pluginSvc, session := previewPlugin(t, workerID) + observeObject(pluginSvc, workerID, objectID) + + errs := make(chan error, 1) + go func() { + _, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10) + errs <- err + }() + + request := <-session.outgoing + pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{ + WorkerId: workerID, + Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{ + ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{ + RequestId: request.RequestId, + Success: false, + ErrorMessage: "open lance dataset: access denied", + }, + }, + }) + + select { + case err := <-errs: + if err == nil || !strings.Contains(err.Error(), "access denied") { + t.Fatalf("error = %v, want the worker's reason", err) + } + case <-time.After(2 * time.Second): + t.Fatal("preview never returned") + } +} + +// Nothing has looked at the object, so there is nobody to ask. Worth its own +// message: it means detection has not run, not that the table is unreadable. +func TestRequestObjectPreviewNeedsAnObservation(t *testing.T) { + t.Parallel() + pluginSvc, _ := previewPlugin(t, "lance-worker-1") + + _, err := pluginSvc.RequestObjectPreview(context.Background(), []string{"vectors", "ml", "nothing"}, "LANCE", 10) + if err == nil || !strings.Contains(err.Error(), "described") { + t.Fatalf("error = %v, want one naming the missing description", err) + } +} + +// The worker that described the object has since gone away. +func TestRequestObjectPreviewNeedsAConnectedWorker(t *testing.T) { + t.Parallel() + objectID := []string{"vectors", "ml", "embeddings"} + pluginSvc, _ := previewPlugin(t, "lance-worker-1") + observeObject(pluginSvc, "lance-worker-gone", objectID) + + _, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10) + if err == nil || !strings.Contains(err.Error(), "not connected") { + t.Fatalf("error = %v, want one naming the absent worker", err) + } +} + +// Shutdown closes every pending channel, and a reply can arrive at the same +// moment; delivering it outside the lock panics with "send on closed channel". +// The window is narrow enough that this test does not reliably reproduce it - +// widening it with a Gosched between the lookup and the send does, every time - +// so treat this as exercising the path rather than as a regression alarm. +func TestObjectPreviewResponseRacesShutdown(t *testing.T) { + t.Parallel() + const workerID = "lance-worker-1" + objectID := []string{"vectors", "ml", "embeddings"} + + for i := 0; i < 50; i++ { + pluginSvc, session := previewPlugin(t, workerID) + observeObject(pluginSvc, workerID, objectID) + + go func() { + _, _ = pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10) + }() + request := <-session.outgoing + + var wg sync.WaitGroup + wg.Add(2) + go func() { + defer wg.Done() + pluginSvc.Shutdown() + }() + go func() { + defer wg.Done() + pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{ + WorkerId: workerID, + Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{ + ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{ + RequestId: request.RequestId, + Success: true, + }, + }, + }) + }() + wg.Wait() + } +} diff --git a/weed/admin/plugin/scheduler_lane.go b/weed/admin/plugin/scheduler_lane.go index abb69145c..070b0025f 100644 --- a/weed/admin/plugin/scheduler_lane.go +++ b/weed/admin/plugin/scheduler_lane.go @@ -22,11 +22,15 @@ const ( // LaneLifecycle handles S3 object store lifecycle management // (expiration, transition, abort incomplete multipart uploads). LaneLifecycle SchedulerLane = "lifecycle" + + // LaneLance handles table-bucket Lance maintenance: fragment compaction, + // index optimization and version cleanup. + LaneLance SchedulerLane = "lance" ) // AllLanes returns every defined scheduler lane in a stable order. func AllLanes() []SchedulerLane { - return []SchedulerLane{LaneDefault, LaneIceberg, LaneLifecycle} + return []SchedulerLane{LaneDefault, LaneIceberg, LaneLifecycle, LaneLance} } // laneIdleSleep maps each lane to its default idle sleep duration. @@ -36,6 +40,7 @@ var laneIdleSleep = map[SchedulerLane]time.Duration{ LaneDefault: 61 * time.Second, LaneIceberg: 61 * time.Second, LaneLifecycle: 5 * time.Minute, + LaneLance: 61 * time.Second, } // laneRequiresLock maps each lane to whether its job types must be @@ -48,6 +53,9 @@ var laneRequiresLock = map[SchedulerLane]bool{ LaneDefault: true, LaneIceberg: false, LaneLifecycle: false, + // Lance maintenance rewrites files inside one table and shares no global + // state, so it has no more need of the cluster admin lock than Iceberg does. + LaneLance: false, } // LaneRequiresLock returns true if the given lane serialises its job types @@ -83,6 +91,13 @@ var jobTypeLaneMap = map[string]SchedulerLane{ // S3 lifecycle management "s3_lifecycle": LaneLifecycle, + + // Lance table maintenance. Without these the job types fall back to the + // default lane, which serialises everything under the cluster admin lock + // and would queue a compaction behind volume balancing. + "lance_compact": LaneLance, + "lance_optimize_indices": LaneLance, + "lance_cleanup_versions": LaneLance, } // JobTypeLane returns the scheduler lane for the given job type. diff --git a/weed/admin/plugin/scheduler_lane_test.go b/weed/admin/plugin/scheduler_lane_test.go index 1bdd19436..e9c47380f 100644 --- a/weed/admin/plugin/scheduler_lane_test.go +++ b/weed/admin/plugin/scheduler_lane_test.go @@ -5,10 +5,16 @@ import ( ) func TestJobTypeLaneMapCoversKnownTypes(t *testing.T) { - // Every job type in the map must resolve to a valid lane. + // Every job type in the map must resolve to a lane the scheduler runs. + // Checked against AllLanes rather than a list spelled out here, so adding a + // lane does not silently leave its job types unscheduled. + known := make(map[SchedulerLane]bool, len(AllLanes())) + for _, lane := range AllLanes() { + known[lane] = true + } for jobType, lane := range jobTypeLaneMap { - if lane != LaneDefault && lane != LaneIceberg && lane != LaneLifecycle { - t.Errorf("jobTypeLaneMap[%q] = %q, want a known lane", jobType, lane) + if !known[lane] { + t.Errorf("jobTypeLaneMap[%q] = %q, want a lane from AllLanes()", jobType, lane) } } } diff --git a/weed/admin/static/js/s3tables.js b/weed/admin/static/js/s3tables.js index 1c7bc8c4d..2ea7d9709 100644 --- a/weed/admin/static/js/s3tables.js +++ b/weed/admin/static/js/s3tables.js @@ -108,7 +108,8 @@ function initS3TablesBuckets() { const tagsInput = document.getElementById('s3tablesBucketTags').value.trim(); const tags = parseTagsInput(tagsInput); if (tags === null) return; - const payload = { name: name, tags: tags, owner: owner }; + const formatInput = document.querySelector('#s3tablesBucketFormatPicker input[name="format"]:checked'); + const payload = { name: name, tags: tags, owner: owner, format: formatInput ? formatInput.value : 'ICEBERG' }; try { const response = await fetch(s3tBasePath('/api/s3tables/buckets'), { @@ -129,6 +130,51 @@ function initS3TablesBuckets() { }); } + // The endpoint is the whole reason the format matters, so show it changing + // rather than making the operator work it out after the fact. + const formatPicker = document.getElementById('s3tablesBucketFormatPicker'); + const formatHint = document.getElementById('s3tablesBucketFormatHint'); + if (formatPicker && formatHint) { + const icebergPort = formatHint.dataset.icebergPort; + const lancePort = formatHint.dataset.lancePort; + const origin = window.location.protocol + '//' + window.location.hostname; + const describeEndpoint = function () { + const chosen = formatPicker.querySelector('input[name="format"]:checked'); + const isLance = chosen && chosen.value === 'LANCE'; + const port = isLance ? lancePort : icebergPort; + const name = (document.getElementById('s3tablesBucketName').value || '').trim(); + // The bucket name is whatever the operator is typing, so it goes in + // as text. Building this with innerHTML would run their input. + formatHint.textContent = ''; + if (!port || port === '0') { + formatHint.appendChild(document.createTextNode('A bucket holds one format. Tables of the other are refused. ')); + const warning = document.createElement('span'); + warning.className = 'text-warning'; + warning.textContent = 'No server is running for this format.'; + formatHint.appendChild(warning); + return; + } + const path = isLance + ? '/v1/namespace/' + (name || '') + '/list' + : '/v1/' + (name || '') + '/namespaces'; + formatHint.appendChild(document.createTextNode('Clients reach this bucket at ')); + const endpoint = document.createElement('code'); + endpoint.textContent = origin + ':' + port + path; + formatHint.appendChild(endpoint); + }; + formatPicker.addEventListener('change', describeEndpoint); + const bucketNameField = document.getElementById('s3tablesBucketName'); + if (bucketNameField) { + bucketNameField.addEventListener('input', describeEndpoint); + } + describeEndpoint(); + } + + // The banner prints localhost server-side; the browser knows the real host. + document.querySelectorAll('.s3tables-origin').forEach(function (el) { + el.textContent = window.location.protocol + '//' + window.location.hostname + ':' + el.dataset.port; + }); + const policyForm = document.getElementById('s3tablesBucketPolicyForm'); if (policyForm) { policyForm.addEventListener('submit', async function (e) { diff --git a/weed/admin/static_gz/js/s3tables.js.gz b/weed/admin/static_gz/js/s3tables.js.gz index f8430e4f8..c97a2aca9 100644 Binary files a/weed/admin/static_gz/js/s3tables.js.gz and b/weed/admin/static_gz/js/s3tables.js.gz differ diff --git a/weed/admin/view/app/iceberg_table_data.templ b/weed/admin/view/app/iceberg_table_data.templ index 1fefcd06a..119452002 100644 --- a/weed/admin/view/app/iceberg_table_data.templ +++ b/weed/admin/view/app/iceberg_table_data.templ @@ -55,105 +55,107 @@ templ IcebergTableData(data dash.IcebergDataPreviewData) { { note } } -
-
-
-
-
-
-
- Snapshot -
-
- if data.SnapshotID != 0 { - { fmt.Sprintf("%d", data.SnapshotID) } - if data.SnapshotID == data.CurrentSnapshotID { - current + if !previewIsWorkerSourced(data) { +
+
+
+
+
+
+
+ Snapshot +
+
+ if data.SnapshotID != 0 { + { fmt.Sprintf("%d", data.SnapshotID) } + if data.SnapshotID == data.CurrentSnapshotID { + current + } + } else { + - } - } else { - - +
+ if !data.SnapshotTime.IsZero() { +
{ data.SnapshotTime.Format("2006-01-02 15:04:05") }
}
- if !data.SnapshotTime.IsZero() { -
{ data.SnapshotTime.Format("2006-01-02 15:04:05") }
- } -
-
- if len(data.Snapshots) > 1 { -
- - +
+ } else { + + } +
+
+
+
+
+
+
+
+
+
+
+ Data Files
- } else { - - } +
+ { formatNumber(int64(data.TotalDataFiles)) } +
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+ Total Records +
+
+ { formatNumber(data.TotalRecords) } +
+
+
+ +
-
-
-
-
-
-
- Data Files -
-
- { formatNumber(int64(data.TotalDataFiles)) } -
-
-
- -
-
-
-
-
-
-
-
-
-
-
- Total Records -
-
- { formatNumber(data.TotalRecords) } -
-
-
- -
-
-
-
-
-
+ }
@@ -199,9 +201,15 @@ templ IcebergTableData(data dash.IcebergDataPreviewData) {
-
- { fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles) } -
+ if previewIsWorkerSourced(data) { +
+ { fmt.Sprintf("Showing %d row(s), read from the %s dataset by worker %s.", len(data.Rows), data.Format, data.PreviewedBy) } +
+ } else { +
+ { fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles) } +
+ } } else {
No rows to preview.
} @@ -209,50 +217,52 @@ templ IcebergTableData(data dash.IcebergDataPreviewData) { -
-
-
-
-
- Data Files -
-
-
-
- - - - - - - - - - - - for _, file := range data.DataFiles { + if !previewIsWorkerSourced(data) { +
+
+
+
+
+ Data Files +
+
+
+
+
PathFormatRecordsSize
+ - - - - - + + + + + - } - if len(data.DataFiles) == 0 { - - - - } - -
{ file.Path }{ file.Format }{ formatNumber(file.RecordCount) }{ formatBytes(file.SizeBytes) } - - Preview - - PathFormatRecordsSize
No data files in this snapshot.
+ + + for _, file := range data.DataFiles { + + { file.Path } + { file.Format } + { formatNumber(file.RecordCount) } + { formatBytes(file.SizeBytes) } + + + Preview + + + + } + if len(data.DataFiles) == 0 { + + No data files in this snapshot. + + } + + +
- + } } diff --git a/weed/admin/view/app/iceberg_table_data_templ.go b/weed/admin/view/app/iceberg_table_data_templ.go index 91ef699f5..e999a0906 100644 --- a/weed/admin/view/app/iceberg_table_data_templ.go +++ b/weed/admin/view/app/iceberg_table_data_templ.go @@ -1,6 +1,6 @@ // Code generated by templ - DO NOT EDIT. -// templ: version: v0.3.1001 +// templ: version: v0.3.1020 package app //lint:file-ignore SA4006 This context is only used if a nested component is present. @@ -183,479 +183,515 @@ func IcebergTableData(data dash.IcebergDataPreviewData) templ.Component { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 14, "
Snapshot
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if data.SnapshotID != 0 { - var templ_7745c5c3_Var12 string - templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.SnapshotID)) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 69, Col: 45} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12)) + if !previewIsWorkerSourced(data) { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 14, "
Snapshot
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 15, " ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if data.SnapshotID == data.CurrentSnapshotID { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "current") + if data.SnapshotID != 0 { + var templ_7745c5c3_Var12 string + templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.SnapshotID)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 70, Col: 46} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 15, " ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if data.SnapshotID == data.CurrentSnapshotID { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "current") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "-") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "-") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if !data.SnapshotTime.IsZero() { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "
") + if !data.SnapshotTime.IsZero() { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var13 string + templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(data.SnapshotTime.Format("2006-01-02 15:04:05")) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 79, Col: 88} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var13 string - templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(data.SnapshotTime.Format("2006-01-02 15:04:05")) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 78, Col: 87} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if len(data.Snapshots) > 1 { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "
Data Files
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "") + var templ_7745c5c3_Var19 string + templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(int64(data.TotalDataFiles))) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 128, Col: 51} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "
Total Records
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var20 string + templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(data.TotalRecords)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 147, Col: 42} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "
Data Files
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var19 string - templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(int64(data.TotalDataFiles))) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 127, Col: 50} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "
Total Records
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var20 string - templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(data.TotalRecords)) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 146, Col: 41} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "
Sample Rows ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "
Sample Rows ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } if data.SelectedFile != "" { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var21 string templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinStringErrs(path.Base(data.SelectedFile)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 164, Col: 75} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 166, Col: 75} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var21)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, " all files") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "\">all files") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } for _, limit := range []int{25, 50, 100, 200} { if limit == data.RowLimit { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "\">") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var24 string templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", limit)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 171, Col: 203} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 173, Col: 203} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var24)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "\">") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var26 string templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", limit)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 173, Col: 211} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 175, Col: 211} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var26)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } if len(data.Columns) > 0 { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "
#
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } for _, col := range data.Columns { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } for i, row := range data.Rows { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } for _, cell := range row { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "
#") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var27 string templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinStringErrs(col) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 186, Col: 20} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 188, Col: 20} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var28 string templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", i+1)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 193, Col: 58} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 195, Col: 58} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var29 string templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinStringErrs(cell) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 195, Col: 42} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 197, Col: 42} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var30 string - templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles)) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 203, Col: 98} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err + if previewIsWorkerSourced(data) { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var30 string + templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("Showing %d row(s), read from the %s dataset by worker %s.", len(data.Rows), data.Format, data.PreviewedBy)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 206, Col: 129} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var31 string + templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 210, Col: 99} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } } } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "
No rows to preview.
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "
No rows to preview.
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "
Data Files
") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - for _, file := range data.DataFiles { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } - if len(data.DataFiles) == 0 { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "
PathFormatRecordsSize
") + if !previewIsWorkerSourced(data) { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "
Data Files
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var31 string - templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(file.Path) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 235, Col: 45} + for _, file := range data.DataFiles { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31)) + if len(data.DataFiles) == 0 { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "
PathFormatRecordsSize
") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var32 string + templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinStringErrs(file.Path) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 244, Col: 46} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var33 string + templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinStringErrs(file.Format) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 245, Col: 61} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var34 string + templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(file.RecordCount)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 246, Col: 47} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var35 string + templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(file.SizeBytes)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 247, Col: 44} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "Preview
No data files in this snapshot.
") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "		") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var32 string - templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinStringErrs(file.Format) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 236, Col: 60} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var33 string - templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(file.RecordCount)) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 237, Col: 46} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var34 string - templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(file.SizeBytes)) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 238, Col: 43} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "Preview
No data files in this snapshot.
") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err } return nil }) diff --git a/weed/admin/view/app/iceberg_table_data_urls.go b/weed/admin/view/app/iceberg_table_data_urls.go index a106ceda5..82ce7d5b3 100644 --- a/weed/admin/view/app/iceberg_table_data_urls.go +++ b/weed/admin/view/app/iceberg_table_data_urls.go @@ -3,6 +3,10 @@ package app import ( "net/url" "strconv" + "strings" + + "github.com/seaweedfs/seaweedfs/weed/admin/dash" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" ) // icebergTableDataURL builds the table data-preview page URL. Zero snapshotID @@ -26,3 +30,10 @@ func icebergTableDataURL(catalog, namespace, table string, snapshotID int64, lim } return u } + +// previewIsWorkerSourced reports whether these rows came from a plugin worker +// rather than from metadata admin read itself. Snapshots and data files are +// Iceberg's shape, and showing them empty for another format reads as a fault. +func previewIsWorkerSourced(data dash.IcebergDataPreviewData) bool { + return data.Format != "" && !strings.EqualFold(data.Format, s3tables.FormatIceberg) +} diff --git a/weed/admin/view/app/iceberg_table_details.templ b/weed/admin/view/app/iceberg_table_details.templ index b6a286a19..0d4ee7ead 100644 --- a/weed/admin/view/app/iceberg_table_details.templ +++ b/weed/admin/view/app/iceberg_table_details.templ @@ -27,7 +27,12 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) { { data.NamespaceName } -
  • { data.TableName }
    • +
  • + { data.TableName } + if data.Format != "" { + { formatLabel(data.Format) } + } +
    • @@ -223,6 +228,11 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) {
    Schema + if data.ObservedBy != "" { + + as seen { data.ObservedAt.Format("2006-01-02 15:04") } + + }
    @@ -253,7 +263,13 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) { } if len(data.SchemaFields) == 0 { - No schema available. + if data.Format != "" && data.Format != "ICEBERG" { + + The catalog records where this { data.Format } table lives, not what is in it. Run a { data.Format } plugin worker and its schema appears here. + + } else { + No schema available. + } } @@ -263,104 +279,143 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) {
    -
    -
    -
    -
    -
    - Partitions -
    -
    -
    -
    - - - - - - - - - - - for _, field := range data.PartitionFields { + if !isLanceFormat(data.Format) { +
    +
    +
    +
    +
    + Partitions +
    +
    +
    +
    +
    NameTransformSource IDField ID
    + - - - - + + + + - } - if len(data.PartitionFields) == 0 { - - - - } - -
    { field.Name }{ field.Transform }{ fmt.Sprintf("%d", field.SourceID) }{ fmt.Sprintf("%d", field.FieldID) }NameTransformSource IDField ID
    No partitions defined.
    + + + for _, field := range data.PartitionFields { + + { field.Name } + { field.Transform } + { fmt.Sprintf("%d", field.SourceID) } + { fmt.Sprintf("%d", field.FieldID) } + + } + if len(data.PartitionFields) == 0 { + + No partitions defined. + + } + + +
    - -
    -
    -
    -
    -
    - Snapshot History -
    -
    -
    -
    - - - - - - - - - - - for _, snapshot := range data.Snapshots { + } + if !isLanceFormat(data.Format) { +
    +
    +
    +
    +
    + Snapshot History +
    +
    +
    +
    +
    Snapshot IDTimestampOperationManifest List
    + - - - - + + + + - } - if len(data.Snapshots) == 0 { - - - - } - -
    { fmt.Sprintf("%d", snapshot.SnapshotID) } - if snapshot.Timestamp.IsZero() { - - - } else { - { snapshot.Timestamp.Format("2006-01-02 15:04") } - } - - if snapshot.Operation != "" { - { snapshot.Operation } - } else { - - - } - - if snapshot.ManifestList != "" { - { snapshot.ManifestList } - } else { - - - } - Snapshot IDTimestampOperationManifest List
    No snapshots available.
    + + + for _, snapshot := range data.Snapshots { + + { fmt.Sprintf("%d", snapshot.SnapshotID) } + + if snapshot.Timestamp.IsZero() { + - + } else { + { snapshot.Timestamp.Format("2006-01-02 15:04") } + } + + + if snapshot.Operation != "" { + { snapshot.Operation } + } else { + - + } + + + if snapshot.ManifestList != "" { + { snapshot.ManifestList } + } else { + - + } + + + } + if len(data.Snapshots) == 0 { + + if data.Format != "" && data.Format != "ICEBERG" { + + A { data.Format } table keeps its own version history; the catalog does not mirror it here. + + } else { + No snapshots available. + } + + } + + +
    - + } + if isLanceFormat(data.Format) { +
    +
    +
    +
    +
    + Versions + if data.ObservedBy != "" { + + as seen { data.ObservedAt.Format("2006-01-02 15:04") } + + } +
    +
    +
    + if len(data.Properties) > 0 { +

    + A { data.Format } dataset keeps its own version history. What a worker last saw is listed under Properties above. +

    + } else { +

    + No worker has reported on this table yet, so its version history is unknown here. It lives in the dataset either way. +

    + } +
    +
    +
    +
    + }
    Delete Namespace

    Are you sure you want to delete the namespace ?

    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } diff --git a/weed/admin/view/app/s3tables_tables.templ b/weed/admin/view/app/s3tables_tables.templ index e27f2320a..84ce15481 100644 --- a/weed/admin/view/app/s3tables_tables.templ +++ b/weed/admin/view/app/s3tables_tables.templ @@ -12,6 +12,7 @@ templ S3TablesTables(data dash.S3TablesTablesData) {

    S3 Tables + { formatLabel(data.BucketFormat) }

    @@ -93,6 +94,8 @@ templ S3TablesTables(data dash.S3TablesTablesData) { Name + Format + Rows Table ARN Created Modified @@ -106,6 +109,16 @@ templ S3TablesTables(data dash.S3TablesTablesData) { {{ tableName := table.Name }} { tableName } + + { formatLabel(table.Format) } + + + if rows, ok := data.ObservedRows[tableName]; ok { + { rows } + } else { + — + } + { table.TableARN } { table.CreatedAt.Format("2006-01-02 15:04") } { table.ModifiedAt.Format("2006-01-02 15:04") } @@ -126,7 +139,7 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
    if parseErr == nil { - + } else { @@ -149,7 +162,7 @@ templ S3TablesTables(data dash.S3TablesTablesData) { } if len(data.Tables) == 0 { - +
    No tables found
    @@ -186,9 +199,16 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
    - + if data.BucketFormat != "" { + +
    Set by the bucket, which holds one format.
    + } else { + +
    This bucket was created before formats were declared, so either is allowed.
    + }
    diff --git a/weed/admin/view/app/s3tables_tables_templ.go b/weed/admin/view/app/s3tables_tables_templ.go index 113c8b861..88fe8227e 100644 --- a/weed/admin/view/app/s3tables_tables_templ.go +++ b/weed/admin/view/app/s3tables_tables_templ.go @@ -1,6 +1,6 @@ // Code generated by templ - DO NOT EDIT. -// templ: version: v0.3.1001 +// templ: version: v0.3.1020 package app //lint:file-ignore SA4006 This context is only used if a nested component is present. @@ -37,359 +37,511 @@ func S3TablesTables(data dash.S3TablesTablesData) templ.Component { templ_7745c5c3_Var1 = templ.NopComponent } ctx = templ.ClearChildren(ctx) - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "

    S3 Tables

    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "

    S3 Tables ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - bucketName, parseErr := s3tables.ParseBucketNameFromARN(data.BucketARN) - if parseErr == nil { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, "Back to Namespaces ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 4, " ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } + var templ_7745c5c3_Var2 = []any{formatBadgeClass(data.BucketFormat), "ms-2 align-middle"} + templ_7745c5c3_Err = templ.RenderCSSItems(ctx, templ_7745c5c3_Buffer, templ_7745c5c3_Var2...) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 5, "Bucket ARN: ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, " Namespace: ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 3, "\" title=\"") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var4 string - templ_7745c5c3_Var4, templ_7745c5c3_Err = templ.JoinStringErrs(data.Namespace) + templ_7745c5c3_Var4, templ_7745c5c3_Err = templ.ResolveAttributeValue(formatBadgeTitle(data.BucketFormat)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 36, Col: 59} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 15, Col: 119} } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var4)) + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var4) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 7, " ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if parseErr != nil { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 8, "Invalid bucket ARN") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 9, "

    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var5 string - templ_7745c5c3_Var5, templ_7745c5c3_Err = templ.JoinStringErrs(data.BucketARN) + templ_7745c5c3_Var5, templ_7745c5c3_Err = templ.JoinStringErrs(formatLabel(data.BucketFormat)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 41, Col: 67} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 15, Col: 154} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var5)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 10, "\" data-namespace=\"") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 5, "
    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var6 string - templ_7745c5c3_Var6, templ_7745c5c3_Err = templ.JoinStringErrs(data.Namespace) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 41, Col: 101} + bucketName, parseErr := s3tables.ParseBucketNameFromARN(data.BucketARN) + if parseErr == nil { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 6, "Back to Namespaces ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 8, " ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var6)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 11, "\">
    Total Tables
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 9, "Bucket ARN: ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var7 string - templ_7745c5c3_Var7, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.TotalTables)) + templ_7745c5c3_Var7, templ_7745c5c3_Err = templ.JoinStringErrs(data.BucketARN) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 52, Col: 46} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 36, Col: 60} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var7)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 12, "
    Last Updated
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 10, " Namespace: ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var8 string - templ_7745c5c3_Var8, templ_7745c5c3_Err = templ.JoinStringErrs(data.LastUpdated.Format("15:04")) + templ_7745c5c3_Var8, templ_7745c5c3_Err = templ.JoinStringErrs(data.Namespace) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 71, Col: 43} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 37, Col: 59} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var8)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 13, "
    Tables
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 11, " ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if parseErr != nil { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 12, "Invalid bucket ARN") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 13, "
    Total Tables
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var11 string + templ_7745c5c3_Var11, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.TotalTables)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 53, Col: 46} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var11)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "
    Last Updated
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var12 string + templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(data.LastUpdated.Format("15:04")) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 72, Col: 43} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "
    Tables
    NameTable ARNCreatedModifiedS3 LocationMetadataActions
    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } for _, table := range data.Tables { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 14, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } tableName := table.Name - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 15, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } if len(data.Tables) == 0 { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "
    NameFormatRowsTable ARNCreatedModifiedS3 LocationMetadataActions
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var9 string - templ_7745c5c3_Var9, templ_7745c5c3_Err = templ.JoinStringErrs(tableName) + var templ_7745c5c3_Var13 string + templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(tableName) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 108, Col: 26} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 111, Col: 26} } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var9)) + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var10 string - templ_7745c5c3_Var10, templ_7745c5c3_Err = templ.JoinStringErrs(table.TableARN) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 109, Col: 56} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var10)) + var templ_7745c5c3_Var14 = []any{formatBadgeClass(table.Format)} + templ_7745c5c3_Err = templ.RenderCSSItems(ctx, templ_7745c5c3_Buffer, templ_7745c5c3_Var14...) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "\" title=\"") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var12 string - templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(table.ModifiedAt.Format("2006-01-02 15:04")) + var templ_7745c5c3_Var16 string + templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.ResolveAttributeValue(formatBadgeTitle(table.Format)) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 111, Col: 60} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 113, Col: 97} } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12)) + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var16) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "\">") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - if parseErr == nil { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "s3://") + var templ_7745c5c3_Var17 string + templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(formatLabel(table.Format)) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 113, Col: 127} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if rows, ok := data.ObservedRows[tableName]; ok { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var13 string - templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(bucketName) + var templ_7745c5c3_Var18 string + templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinStringErrs(rows) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 114, Col: 50} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 117, Col: 61} } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13)) + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "/") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var14 string - templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinStringErrs(data.Namespace) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 114, Col: 69} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "/") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var15 string - templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(tableName) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 114, Col: 83} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "/") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "-") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "—") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if table.MetadataLocation != "" { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var16 string - templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(table.MetadataLocation) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 121, Col: 68} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "-") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "
    ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - if parseErr == nil { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, " ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } else { - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, " ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "
    		") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var19 string templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(table.TableARN) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 140, Col: 128} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 122, Col: 56} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "\" data-table-name=\"") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var20 string - templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(tableName) + templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(table.CreatedAt.Format("2006-01-02 15:04")) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 140, Col: 158} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 123, Col: 59} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "\" title=\"Table Policy\"> ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } var templ_7745c5c3_Var21 string - templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinStringErrs(tableName) + templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinStringErrs(table.ModifiedAt.Format("2006-01-02 15:04")) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 143, Col: 126} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 124, Col: 60} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var21)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "\" title=\"Delete\">
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if parseErr == nil { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "s3://") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var22 string + templ_7745c5c3_Var22, templ_7745c5c3_Err = templ.JoinStringErrs(bucketName) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 127, Col: 50} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var22)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "/") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var23 string + templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinStringErrs(data.Namespace) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 127, Col: 69} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var23)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "/") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var24 string + templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.JoinStringErrs(tableName) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 127, Col: 83} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var24)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "/") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "-") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if table.MetadataLocation != "" { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var25 string + templ_7745c5c3_Var25, templ_7745c5c3_Err = templ.JoinStringErrs(table.MetadataLocation) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/s3tables_tables.templ`, Line: 134, Col: 68} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var25)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "-") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if parseErr == nil { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, " ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, " ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "
    No tables found

    Create your first table to start storing data.

    No tables found

    Create your first table to start storing data.

    Create Table
    Delete Table

    Are you sure you want to delete the table ?

    Table Policy
    Resource Tags
    Loading...
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "
    Create Table
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if data.BucketFormat != "" { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "
    Set by the bucket, which holds one format.
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "
    This bucket was created before formats were declared, so either is allowed.
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "
    Delete Table

    Are you sure you want to delete the table ?

    Table Policy
    Resource Tags
    Loading...
    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } diff --git a/weed/admin/view/app/table_format.go b/weed/admin/view/app/table_format.go new file mode 100644 index 000000000..8dfdff0aa --- /dev/null +++ b/weed/admin/view/app/table_format.go @@ -0,0 +1,63 @@ +package app + +import ( + "fmt" + "strings" + + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// Shared rendering for the table format a bucket or table carries. A bucket is a +// catalog and a catalog serves one protocol, so the format decides which +// endpoint a client uses and which panels on a page mean anything. + +// formatLabel is what the badge says. A bucket created before formats were +// declared has none; that is a fact about its age, not a fault, so it says so +// plainly rather than claiming a format it never chose. +func formatLabel(format string) string { + if strings.TrimSpace(format) == "" { + return "unset" + } + return strings.ToUpper(format) +} + +// formatBadgeClass keeps the two formats distinguishable at a glance without +// reaching for a colour that means something else in this UI. +func formatBadgeClass(format string) string { + switch strings.ToUpper(strings.TrimSpace(format)) { + case s3tables.FormatIceberg: + return "badge bg-primary" + case s3tables.FormatLance: + return "badge bg-success" + default: + return "badge bg-light text-muted border" + } +} + +// formatBadgeTitle explains the badge on hover, which is the only place there is +// room to say what an undeclared bucket does. +func formatBadgeTitle(format string) string { + switch strings.ToUpper(strings.TrimSpace(format)) { + case s3tables.FormatIceberg: + return "Served over the Iceberg REST catalog" + case s3tables.FormatLance: + return "Served over the Lance Namespace API" + default: + return "Created before formats were declared; accepts tables of either format" + } +} + +// isLanceFormat reports whether this is a format the cluster records but cannot +// read, which is what decides between the Iceberg panels and the worker's. +func isLanceFormat(format string) bool { + return strings.EqualFold(strings.TrimSpace(format), s3tables.FormatLance) +} + +// bucketCatalogPath is the path a client uses to reach one bucket, which differs +// per format because the two are different protocols on different ports. +func bucketCatalogPath(format, bucket string) string { + if isLanceFormat(format) { + return fmt.Sprintf("/v1/namespace/%s/list", bucket) + } + return fmt.Sprintf("/v1/%s/namespaces", bucket) +} diff --git a/weed/admin/view/layout/layout.templ b/weed/admin/view/layout/layout.templ index 829e73485..9733919de 100644 --- a/weed/admin/view/layout/layout.templ +++ b/weed/admin/view/layout/layout.templ @@ -44,6 +44,7 @@ templ Layout(view ViewContext, content templ.Component) { strings.HasPrefix(currentPath, "/plugin/lanes/default/") isIcebergWorkerPage := currentPath == "/plugin/lanes/iceberg" || strings.HasPrefix(currentPath, "/plugin/lanes/iceberg/") isLifecycleWorkerPage := currentPath == "/plugin/lanes/lifecycle" || strings.HasPrefix(currentPath, "/plugin/lanes/lifecycle/") + isLanceWorkerPage := currentPath == "/plugin/lanes/lance" || strings.HasPrefix(currentPath, "/plugin/lanes/lance/") }} @@ -294,6 +295,17 @@ templ Layout(view ViewContext, content templ.Component) { } +
  • + if isLifecycleWorkerPage { + + Lifecycle + + } else { + + Lifecycle + + } +
  • if isIcebergWorkerPage { @@ -306,13 +318,13 @@ templ Layout(view ViewContext, content templ.Component) { }
  • - if isLifecycleWorkerPage { - - Lifecycle + if isLanceWorkerPage { + + Lance } else { - - Lifecycle + + Lance }
    • diff --git a/weed/admin/view/layout/layout_templ.go b/weed/admin/view/layout/layout_templ.go index d37f0a4dc..5c7fa667f 100644 --- a/weed/admin/view/layout/layout_templ.go +++ b/weed/admin/view/layout/layout_templ.go @@ -1,6 +1,6 @@ // Code generated by templ - DO NOT EDIT. -// templ: version: v0.3.1001 +// templ: version: v0.3.1020 package layout //lint:file-ignore SA4006 This context is only used if a nested component is present. @@ -77,16 +77,17 @@ func Layout(view ViewContext, content templ.Component) templ.Component { strings.HasPrefix(currentPath, "/plugin/lanes/default/") isIcebergWorkerPage := currentPath == "/plugin/lanes/iceberg" || strings.HasPrefix(currentPath, "/plugin/lanes/iceberg/") isLifecycleWorkerPage := currentPath == "/plugin/lanes/lifecycle" || strings.HasPrefix(currentPath, "/plugin/lanes/lifecycle/") + isLanceWorkerPage := currentPath == "/plugin/lanes/lance" || strings.HasPrefix(currentPath, "/plugin/lanes/lance/") templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "SeaweedFS AdminIceberg") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "\">Lifecycle") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } @@ -727,15 +728,15 @@ func Layout(view ViewContext, content templ.Component) templ.Component { return templ_7745c5c3_Err } var templ_7745c5c3_Var46 templ.SafeURL - templ_7745c5c3_Var46, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/iceberg")) + templ_7745c5c3_Var46, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lifecycle")) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 303, Col: 90} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 304, Col: 92} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var46)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "\">Iceberg") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "\">Lifecycle") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } @@ -744,21 +745,21 @@ func Layout(view ViewContext, content templ.Component) templ.Component { if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - if isLifecycleWorkerPage { + if isIcebergWorkerPage { templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "Lifecycle") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "\">Iceberg") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } @@ -768,20 +769,61 @@ func Layout(view ViewContext, content templ.Component) templ.Component { return templ_7745c5c3_Err } var templ_7745c5c3_Var48 templ.SafeURL - templ_7745c5c3_Var48, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lifecycle")) + templ_7745c5c3_Var48, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/iceberg")) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 314, Col: 92} + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 315, Col: 90} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var48)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "\">Lifecycle") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "\">Iceberg") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "
  • ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + if isLanceWorkerPage { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "Lance") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } else { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "Lance") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 74, "
    • ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } @@ -789,127 +831,127 @@ func Layout(view ViewContext, content templ.Component) templ.Component { if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "
    © ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 75, "
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 84, "\">") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } @@ -933,140 +975,140 @@ func LoginForm(title string, errorMessage string, csrfToken string) templ.Compon }() } ctx = templ.InitializeContext(ctx) - templ_7745c5c3_Var57 := templ.GetChildren(ctx) - if templ_7745c5c3_Var57 == nil { - templ_7745c5c3_Var57 = templ.NopComponent + templ_7745c5c3_Var59 := templ.GetChildren(ctx) + if templ_7745c5c3_Var59 == nil { + templ_7745c5c3_Var59 = templ.NopComponent } ctx = templ.ClearChildren(ctx) prefix := dash.URLPrefixFromContext(ctx) - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 82, "") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "<!doctype html><html lang=\"en\"><head><meta charset=\"UTF-8\"><title>") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var58 string - templ_7745c5c3_Var58, templ_7745c5c3_Err = templ.JoinStringErrs(title) + var templ_7745c5c3_Var60 string + templ_7745c5c3_Var60, templ_7745c5c3_Err = templ.JoinStringErrs(title) if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 373, Col: 17} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var58)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 83, " - Login

    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "\" type=\"image/x-icon\">

    Please sign in to continue

    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "\" rel=\"stylesheet\"> ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - var templ_7745c5c3_Var63 string - templ_7745c5c3_Var63, templ_7745c5c3_Err = templ.JoinStringErrs(errorMessage) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 394, Col: 45} - } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var63)) - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "
    ") - if templ_7745c5c3_Err != nil { - return templ_7745c5c3_Err - } + var templ_7745c5c3_Var63 templ.SafeURL + templ_7745c5c3_Var63, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/css/fontawesome.min.css") + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 389, Col: 59} } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "

    ") if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 398, Col: 85} + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var64 string + templ_7745c5c3_Var64, templ_7745c5c3_Err = templ.JoinStringErrs(title) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 399, Col: 57} } _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var64)) if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "\">

    Please sign in to continue

    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } - var templ_7745c5c3_Var65 string - templ_7745c5c3_Var65, templ_7745c5c3_Err = templ.JoinStringErrs(csrfToken) - if templ_7745c5c3_Err != nil { - return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 399, Col: 84} + if errorMessage != "" { + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, "
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + var templ_7745c5c3_Var65 string + templ_7745c5c3_Var65, templ_7745c5c3_Err = templ.JoinStringErrs(errorMessage) + if templ_7745c5c3_Err != nil { + return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 406, Col: 45} + } + _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var65)) + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, "
    ") + if templ_7745c5c3_Err != nil { + return templ_7745c5c3_Err + } } - _, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var65)) + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "
    ") + templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, "\">
    ") if templ_7745c5c3_Err != nil { return templ_7745c5c3_Err } diff --git a/weed/command/admin.go b/weed/command/admin.go index 910848cdd..098b97d5b 100644 --- a/weed/command/admin.go +++ b/weed/command/admin.go @@ -51,6 +51,7 @@ type AdminOptions struct { readOnlyPassword *string dataDir *string icebergPort *int + lancePort *int urlPrefix *string metricsHttpPort *int metricsHttpIp *string @@ -74,6 +75,7 @@ func init() { a.readOnlyUser = cmdAdmin.Flag.String("readOnlyUser", "", "read-only user username (optional, for view-only access)") a.readOnlyPassword = cmdAdmin.Flag.String("readOnlyPassword", "", "read-only user password (optional, for view-only access; requires adminPassword to be set)") a.icebergPort = cmdAdmin.Flag.Int("iceberg.port", 8181, "Iceberg REST Catalog port (0 to hide in UI)") + a.lancePort = cmdAdmin.Flag.Int("lance.port", 9101, "Lance Namespace port (0 to hide in UI)") a.urlPrefix = cmdAdmin.Flag.String("urlPrefix", "", "URL path prefix when running behind a reverse proxy under a subdirectory (e.g. /seaweedfs)") a.metricsHttpPort = cmdAdmin.Flag.Int("metricsPort", 0, "Prometheus metrics listen port") a.metricsHttpIp = cmdAdmin.Flag.String("metricsIp", "", "metrics listen ip. If empty, listens on all interfaces.") @@ -314,7 +316,7 @@ func runAdmin(cmd *Command, args []string) bool { } // Start the admin server with all masters (UI enabled by default) - err := startAdminServer(ctx, a, true, *a.icebergPort, urlPrefix) + err := startAdminServer(ctx, a, true, *a.icebergPort, *a.lancePort, urlPrefix) if err != nil { fmt.Printf("Admin server error: %v\n", err) return false @@ -325,7 +327,7 @@ func runAdmin(cmd *Command, args []string) bool { } // startAdminServer starts the actual admin server -func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, icebergPort int, urlPrefix string) error { +func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, icebergPort, lancePort int, urlPrefix string) error { // Create router r := mux.NewRouter() r.Use(loggingMiddleware) @@ -388,7 +390,7 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", admin.StaticHandler())) // Create admin server (plugin is always enabled) - adminServer := dash.NewAdminServer(*options.master, *options.filerGroup, nil, dataDir, icebergPort) + adminServer := dash.NewAdminServer(*options.master, *options.filerGroup, nil, dataDir, icebergPort, lancePort) if err := adminServer.ApplyPluginConfigFromToml(util.GetViper()); err != nil { return fmt.Errorf("apply admin.toml to plugin config: %w", err) diff --git a/weed/command/filer.go b/weed/command/filer.go index a0a90b251..90777925e 100644 --- a/weed/command/filer.go +++ b/weed/command/filer.go @@ -157,6 +157,7 @@ func init() { filerS3Options.cipher = cmdFiler.Flag.Bool("s3.encryptVolumeData", false, "encrypt data on volume servers for S3 uploads") filerS3Options.iamReadOnly = cmdFiler.Flag.Bool("s3.iam.readOnly", true, "disable IAM write operations on this server") filerS3Options.portIceberg = cmdFiler.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)") + filerS3Options.portLance = cmdFiler.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)") filerS3Options.externalUrl = cmdFiler.Flag.String("s3.externalUrl", "", "the external URL clients use to connect (e.g. https://api.example.com:9000). Used for S3 signature verification behind a reverse proxy. Falls back to S3_EXTERNAL_URL env var.") filerS3Options.defaultFileMode = cmdFiler.Flag.String("s3.defaultFileMode", "", "default file mode for S3 uploaded objects, e.g. 0660, 0644, 0666") filerS3Options.cacheSizeMB = cmdFiler.Flag.Int64("s3.cacheCapacityMB", 0, "in-memory chunk cache capacity in MB for S3 GETs shared across requests (0 disables)") diff --git a/weed/command/mini.go b/weed/command/mini.go index c5ce98bfa..9b8020c67 100644 --- a/weed/command/mini.go +++ b/weed/command/mini.go @@ -212,6 +212,9 @@ func miniStartupServices() []string { if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 { services = append(services, "Iceberg") } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + services = append(services, "Lance") + } } services = append(services, "Admin") return services @@ -498,6 +501,7 @@ func initMiniS3Flags() { miniS3Options.portHttps = cmdMini.Flag.Int("s3.port.https", 0, "s3 server https listen port") miniS3Options.portGrpc = cmdMini.Flag.Int("s3.port.grpc", 0, "s3 server grpc listen port") miniS3Options.portIceberg = cmdMini.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)") + miniS3Options.portLance = cmdMini.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)") miniS3Options.icebergCredentialRole = cmdMini.Flag.String("s3.iceberg.credentialRole", "", "IAM role ARN the Iceberg catalog assumes to vend table-scoped credentials (empty disables vending)") miniS3Options.icebergCredentialDuration = cmdMini.Flag.Int("s3.iceberg.credentialDurationSeconds", 3600, "lifetime of credentials vended by the Iceberg catalog") miniS3Options.domainName = cmdMini.Flag.String("s3.domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}") @@ -884,6 +888,14 @@ func ensureAllPortsAvailableOnIP(bindIp string) error { grpcPtr *int }{miniS3Options.portIceberg, "Iceberg", "s3.port.iceberg", nil}) } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + portConfigs = append(portConfigs, struct { + port *int + name string + flagName string + grpcPtr *int + }{miniS3Options.portLance, "Lance", "s3.port.lance", nil}) + } } portConfigs = append(portConfigs, struct { port *int @@ -935,9 +947,13 @@ func ensureAllPortsAvailableOnIP(bindIp string) error { if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 { icebergPortStr = fmt.Sprintf("%d", *miniS3Options.portIceberg) } - glog.V(1).Infof("Final port configuration - Master: %d, Filer: %d, Volume: %d, S3: %d, Iceberg: %s, WebDAV: %d, Admin: %d", + lancePortStr := "disabled" + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + lancePortStr = fmt.Sprintf("%d", *miniS3Options.portLance) + } + glog.V(1).Infof("Final port configuration - Master: %d, Filer: %d, Volume: %d, S3: %d, Iceberg: %s, Lance: %s, WebDAV: %d, Admin: %d", *miniMasterOptions.port, *miniFilerOptions.port, *miniOptions.v.port, - *miniS3Options.port, icebergPortStr, *miniWebDavOptions.port, *miniAdminOptions.port) + *miniS3Options.port, icebergPortStr, lancePortStr, *miniWebDavOptions.port, *miniAdminOptions.port) // Log gRPC ports too (now finalized) glog.V(1).Infof("gRPC port configuration - Master: %d, Filer: %d, Volume: %d, S3: %d, Admin: %d", @@ -967,6 +983,9 @@ func initializeGrpcPortsOnIP(bindIp string) { if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 { allocatedPorts[*miniS3Options.portIceberg] = true } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + allocatedPorts[*miniS3Options.portLance] = true + } } grpcConfigs := []struct { @@ -1421,10 +1440,14 @@ func startMiniServices(miniWhiteList []string, allServicesReady chan struct{}) { go func() { defer done() defer reportMiniStopped("S3") - // Iceberg lives inside the S3 server; report it stopped alongside. + // Iceberg and Lance live inside the S3 server; report them stopped + // alongside it. if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 { defer reportMiniStopped("Iceberg") } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + defer reportMiniStopped("Lance") + } startMiniService("S3", startS3Service, *miniS3Options.port) }() } @@ -1452,6 +1475,12 @@ func startMiniServices(miniWhiteList []string, allServicesReady chan struct{}) { } waitForServiceReady("Iceberg", *miniS3Options.portIceberg, bindIp) } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + if miniProgressBoard != nil { + miniProgressBoard.starting("Lance") + } + waitForServiceReady("Lance", *miniS3Options.portLance, bindIp) + } } if *miniEnableWebDAV { waitForServiceReady("WebDAV", *miniWebDavOptions.port, bindIp) @@ -1585,11 +1614,18 @@ func startMiniAdminWithWorker(allServicesReady chan struct{}) { go func() { defer done() defer reportMiniStopped("Admin") - var icebergPort int - if miniS3Options.portIceberg != nil { - icebergPort = *miniS3Options.portIceberg + // Only advertise a catalog port when S3 is actually running: with -s3=false + // the admin UI would otherwise print an endpoint nothing is listening on. + var icebergPort, lancePort int + if miniEnableS3 != nil && *miniEnableS3 { + if miniS3Options.portIceberg != nil { + icebergPort = *miniS3Options.portIceberg + } + if miniS3Options.portLance != nil { + lancePort = *miniS3Options.portLance + } } - if err := startAdminServer(ctx, miniAdminOptions, *miniEnableAdminUI, icebergPort, urlPrefix); err != nil { + if err := startAdminServer(ctx, miniAdminOptions, *miniEnableAdminUI, icebergPort, lancePort, urlPrefix); err != nil { glog.Errorf("Admin server error: %v", err) } }() @@ -1856,6 +1892,9 @@ func printWelcomeMessage() { if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 { fmt.Fprintf(&sb, " Iceberg Catalog: http://%s:%d\n", *miniIp, *miniS3Options.portIceberg) } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + fmt.Fprintf(&sb, " Lance Namespace: http://%s:%d\n", *miniIp, *miniS3Options.portLance) + } } if *miniEnableAdminUI { fmt.Fprintf(&sb, " Admin UI: http://%s:%d\n", *miniIp, *miniAdminOptions.port) @@ -1984,6 +2023,15 @@ func ensureMiniTableBuckets(bucketSpec string) error { return nil } + // A bucket holds one format, and the format decides which catalog serves it. + // Creating one in a format this mini does not serve leaves a bucket no + // client can reach, so take the format from the endpoint that is running. + format := miniTableBucketFormat() + if format == "" { + glog.Warningf("not creating table buckets %q: neither the Iceberg nor the Lance endpoint is enabled, so nothing could reach them", bucketSpec) + return nil + } + filerAddress := pb.NewServerAddress(*miniIp, *miniFilerOptions.port, *miniFilerOptions.portGrpc) grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.client") @@ -1992,12 +2040,12 @@ func ensureMiniTableBuckets(bucketSpec string) error { mgrClient := s3tables.NewManagerClient(client) for _, name := range names { ctx, cancel := context.WithTimeout(miniClientsCtx(), 5*time.Second) - req := &s3tables.CreateTableBucketRequest{Name: name} + req := &s3tables.CreateTableBucketRequest{Name: name, Format: format} var resp s3tables.CreateTableBucketResponse err := manager.Execute(ctx, mgrClient, "CreateTableBucket", req, &resp, s3tables.DefaultAccountID) cancel() if err == nil { - glog.V(0).Infof("created table bucket %s", name) + glog.V(0).Infof("created %s table bucket %s", format, name) continue } var s3Err *s3tables.S3TablesError @@ -2011,6 +2059,22 @@ func ensureMiniTableBuckets(bucketSpec string) error { }) } +// miniTableBucketFormat is the format a pre-created table bucket should hold: +// Iceberg when its catalog is running, else Lance, else none because neither +// server is up. +func miniTableBucketFormat() string { + if miniEnableS3 == nil || !*miniEnableS3 { + return "" + } + if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 { + return s3tables.FormatIceberg + } + if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 { + return s3tables.FormatLance + } + return "" +} + // parseBucketList splits a comma-separated bucket spec into a deduplicated list // of trimmed, non-empty names, preserving the order they were given. func parseBucketList(spec string) []string { diff --git a/weed/command/s3.go b/weed/command/s3.go index 7d7c4ad7c..fc8150c5f 100644 --- a/weed/command/s3.go +++ b/weed/command/s3.go @@ -24,7 +24,9 @@ import ( "github.com/seaweedfs/seaweedfs/weed/pb/s3_pb" "github.com/seaweedfs/seaweedfs/weed/s3api" "github.com/seaweedfs/seaweedfs/weed/s3api/iceberg" + "github.com/seaweedfs/seaweedfs/weed/s3api/lance" "github.com/seaweedfs/seaweedfs/weed/s3api/s3err" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" "github.com/seaweedfs/seaweedfs/weed/security" stats_collect "github.com/seaweedfs/seaweedfs/weed/stats" "github.com/seaweedfs/seaweedfs/weed/util" @@ -48,6 +50,7 @@ type S3Options struct { portHttps *int portGrpc *int portIceberg *int + portLance *int icebergCredentialRole *string icebergCredentialDuration *int config *string @@ -93,6 +96,7 @@ func init() { s3StandaloneOptions.portHttps = cmdS3.Flag.Int("port.https", 0, "s3 server https listen port") s3StandaloneOptions.portGrpc = cmdS3.Flag.Int("port.grpc", 0, "s3 server grpc listen port") s3StandaloneOptions.portIceberg = cmdS3.Flag.Int("port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)") + s3StandaloneOptions.portLance = cmdS3.Flag.Int("port.lance", 9101, "Lance Namespace server listen port (0 to disable); credential vending uses -iceberg.credentialRole") s3StandaloneOptions.icebergCredentialRole = cmdS3.Flag.String("iceberg.credentialRole", "", "IAM role ARN the Iceberg catalog assumes to vend table-scoped credentials (empty disables vending)") s3StandaloneOptions.icebergCredentialDuration = cmdS3.Flag.Int("iceberg.credentialDurationSeconds", 3600, "lifetime of credentials vended by the Iceberg catalog") s3StandaloneOptions.domainName = cmdS3.Flag.String("domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}") @@ -385,6 +389,11 @@ func (s3opt *S3Options) startS3Server() bool { go s3opt.startIcebergServer(s3ApiServer) } + // Start Lance Namespace server if enabled + if s3opt.portLance != nil && *s3opt.portLance > 0 { + go s3opt.startLanceServer(s3ApiServer) + } + if runtime.GOOS != "windows" { localSocket := *s3opt.localSocket if localSocket == "" { @@ -588,6 +597,82 @@ func (s3opt *S3Options) startIcebergServer(s3ApiServer *s3api.S3ApiServer) { } } +// startLanceServer starts the Lance Namespace server on a separate port. It +// shares the Iceberg catalog's credential role: one deployment vends table +// credentials one way, whichever catalog the client speaks to. +func (s3opt *S3Options) startLanceServer(s3ApiServer *s3api.S3ApiServer) { + lanceRouter := mux.NewRouter().SkipClean(true) + lanceRouter.Use(util_http.EscapeSemicolonsInQuery) + + lanceServer := lance.NewServer(s3ApiServer, s3ApiServer) + if s3opt.icebergCredentialRole != nil && *s3opt.icebergCredentialRole != "" { + lanceServer.SetCredentialVendor(lanceCredentialVendor{s3ApiServer}) + } + lanceServer.SetS3Endpoint(s3opt.deriveLanceStorageEndpoint()) + lanceServer.SetS3Region(s3tables.DefaultRegion) + lanceServer.RegisterRoutes(lanceRouter) + + listenAddress := fmt.Sprintf("%s:%d", *s3opt.bindIp, *s3opt.portLance) + lanceListener, lanceLocalListener, err := util.NewIpAndLocalListeners( + *s3opt.bindIp, *s3opt.portLance, time.Duration(*s3opt.idleTimeout)*time.Second) + if err != nil { + glog.Fatalf("Lance Namespace listener on %s error: %v", listenAddress, err) + } + + glog.V(0).Infof("Start Lance Namespace Server at http://%s", listenAddress) + + httpS := newHttpServer(lanceRouter, nil) + if s3opt.shutdownCtx != nil { + go func() { + <-s3opt.shutdownCtx.Done() + httpS.Shutdown(context.Background()) + }() + } + if lanceLocalListener != nil { + go func() { + if err := httpS.Serve(lanceLocalListener); err != nil && err != http.ErrServerClosed { + glog.V(0).Infof("Lance localhost listener error: %v", err) + } + }() + } + if err = httpS.Serve(lanceListener); err != nil && err != http.ErrServerClosed { + glog.Fatalf("Lance Namespace Server Fail to serve: %v", err) + } +} + +// deriveLanceStorageEndpoint picks the endpoint the Lance namespace puts in +// storage_options. It falls back to the advertised -ip where the Iceberg +// derivation gives up, because the two clients are not in the same position: a +// Spark or Trino Iceberg client brings its own s3.endpoint and advertising the +// wrong one hijacks it, whereas storage_options is the only place a Lance +// client learns where the store is. Without one, object_store quietly falls +// back to real AWS S3 and the failure reads like a credentials problem. +func (s3opt *S3Options) deriveLanceStorageEndpoint() string { + if endpoint := s3opt.deriveS3AdvertisedEndpoint(); endpoint != "" { + return endpoint + } + host := "" + if s3opt.ip != nil { + host = *s3opt.ip + } + switch host { + case "", "0.0.0.0", "::", "[::]": + return "" + } + scheme := "http" + port := 0 + if s3opt.port != nil { + port = *s3opt.port + } + if s3opt.tlsPrivateKey != nil && *s3opt.tlsPrivateKey != "" { + scheme = "https" + if s3opt.portHttps != nil && *s3opt.portHttps > 0 { + port = *s3opt.portHttps + } + } + return fmt.Sprintf("%s://%s", scheme, util.JoinHostPort(host, port)) +} + // deriveS3AdvertisedEndpoint builds the S3 endpoint URL to advertise to // Iceberg catalog clients as part of LoadTable FileIO config. To avoid // hijacking correctly-configured clients (Spark/Trino/PyIceberg all bring @@ -646,3 +731,22 @@ func (v icebergCredentialVendor) VendTableCredentials(ctx context.Context, princ Expiration: credentials.Expiration, }, nil } + +// lanceCredentialVendor adapts the S3 gateway's STS-backed vending to the Lance +// namespace's interface, keeping the two packages independent of each other. +type lanceCredentialVendor struct { + server *s3api.S3ApiServer +} + +func (v lanceCredentialVendor) VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*lance.VendedCredentials, error) { + credentials, err := v.server.VendTableCredentials(ctx, principal, bucket, prefix) + if err != nil || credentials == nil { + return nil, err + } + return &lance.VendedCredentials{ + AccessKeyID: credentials.AccessKeyID, + SecretAccessKey: credentials.SecretAccessKey, + SessionToken: credentials.SessionToken, + Expiration: credentials.Expiration, + }, nil +} diff --git a/weed/command/server.go b/weed/command/server.go index 0eaca136d..625f87d3d 100644 --- a/weed/command/server.go +++ b/weed/command/server.go @@ -164,6 +164,7 @@ func init() { s3Options.portHttps = cmdServer.Flag.Int("s3.port.https", 0, "s3 server https listen port") s3Options.portGrpc = cmdServer.Flag.Int("s3.port.grpc", 0, "s3 server grpc listen port") s3Options.portIceberg = cmdServer.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)") + s3Options.portLance = cmdServer.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)") s3Options.icebergCredentialRole = cmdServer.Flag.String("s3.iceberg.credentialRole", "", "IAM role ARN the Iceberg catalog assumes to vend table-scoped credentials (empty disables vending)") s3Options.icebergCredentialDuration = cmdServer.Flag.Int("s3.iceberg.credentialDurationSeconds", 3600, "lifetime of credentials vended by the Iceberg catalog") s3Options.domainName = cmdServer.Flag.String("s3.domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}") diff --git a/weed/pb/plugin.proto b/weed/pb/plugin.proto index 9333a456c..ffef11de5 100644 --- a/weed/pb/plugin.proto +++ b/weed/pb/plugin.proto @@ -27,6 +27,8 @@ message WorkerToAdminMessage { DetectionComplete detection_complete = 15; JobProgressUpdate job_progress_update = 16; JobCompleted job_completed = 17; + WorkerObservations observations = 18; + ObjectPreviewResponse object_preview_response = 19; } } @@ -42,6 +44,7 @@ message AdminToWorkerMessage { ExecuteJobRequest execute_job_request = 13; CancelRequest cancel_request = 14; AdminShutdown shutdown = 15; + RequestObjectPreview request_object_preview = 16; } } @@ -452,3 +455,61 @@ enum ActivitySource { ACTIVITY_SOURCE_DETECTOR = 2; ACTIVITY_SOURCE_EXECUTOR = 3; } + +// WorkerObservations reports what a worker learned about the objects it +// inspected. A worker opens a table to decide whether it needs a job; what it +// saw is worth keeping either way, because for a format the cluster cannot +// read, the worker is the only thing that can describe it. +// +// Observations are not work. Admin caches them and serves them back, so a +// worker that goes away leaves its last ones standing rather than blanking a +// page. +message WorkerObservations { + string job_type = 1; + repeated ObjectObservation observations = 2; +} + +// ObjectObservation is one object as a worker last saw it. +message ObjectObservation { + // Identifier of the object, e.g. ["bucket", "namespace", "table"]. + repeated string object_id = 1; + // What kind of object this is, e.g. "table". + string object_kind = 2; + // The object's format, e.g. "LANCE", so a reader can tell whose observation + // this is without parsing the attributes. + string format = 3; + // Whatever the worker can cheaply say: schema, row count, fragment count. + // The keys are the worker's to choose. + map attributes = 4; + google.protobuf.Timestamp observed_at = 5; +} + +// RequestObjectPreview asks a worker for sample rows of an object admin cannot +// read itself. Unlike an observation this is not cached: it is fetched when +// someone opens the page, because rows are the object's data rather than a +// description of it, and holding a copy in admin is neither fresh nor its +// business. +message RequestObjectPreview { + repeated string object_id = 1; + // The format admin believes this object is, so a worker that does not own it + // can decline instead of guessing. + string format = 2; + int32 row_limit = 3; +} + +// ObjectPreviewResponse carries the sample back, already rendered as text. The +// worker is the only thing that knows the object's types, so it formats them; +// admin displays what it is given. +message ObjectPreviewResponse { + string request_id = 6; + bool success = 1; + string error_message = 2; + repeated string columns = 3; + repeated PreviewRow rows = 4; + // Rows in the object, which is not the number sampled. + int64 total_rows = 5; +} + +message PreviewRow { + repeated string values = 1; +} diff --git a/weed/pb/plugin_pb/plugin.pb.go b/weed/pb/plugin_pb/plugin.pb.go index 06c0b02e8..dd2505688 100644 --- a/weed/pb/plugin_pb/plugin.pb.go +++ b/weed/pb/plugin_pb/plugin.pb.go @@ -453,6 +453,8 @@ type WorkerToAdminMessage struct { // *WorkerToAdminMessage_DetectionComplete // *WorkerToAdminMessage_JobProgressUpdate // *WorkerToAdminMessage_JobCompleted + // *WorkerToAdminMessage_Observations + // *WorkerToAdminMessage_ObjectPreviewResponse Body isWorkerToAdminMessage_Body `protobuf_oneof:"body"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -581,6 +583,24 @@ func (x *WorkerToAdminMessage) GetJobCompleted() *JobCompleted { return nil } +func (x *WorkerToAdminMessage) GetObservations() *WorkerObservations { + if x != nil { + if x, ok := x.Body.(*WorkerToAdminMessage_Observations); ok { + return x.Observations + } + } + return nil +} + +func (x *WorkerToAdminMessage) GetObjectPreviewResponse() *ObjectPreviewResponse { + if x != nil { + if x, ok := x.Body.(*WorkerToAdminMessage_ObjectPreviewResponse); ok { + return x.ObjectPreviewResponse + } + } + return nil +} + type isWorkerToAdminMessage_Body interface { isWorkerToAdminMessage_Body() } @@ -617,6 +637,14 @@ type WorkerToAdminMessage_JobCompleted struct { JobCompleted *JobCompleted `protobuf:"bytes,17,opt,name=job_completed,json=jobCompleted,proto3,oneof"` } +type WorkerToAdminMessage_Observations struct { + Observations *WorkerObservations `protobuf:"bytes,18,opt,name=observations,proto3,oneof"` +} + +type WorkerToAdminMessage_ObjectPreviewResponse struct { + ObjectPreviewResponse *ObjectPreviewResponse `protobuf:"bytes,19,opt,name=object_preview_response,json=objectPreviewResponse,proto3,oneof"` +} + func (*WorkerToAdminMessage_Hello) isWorkerToAdminMessage_Body() {} func (*WorkerToAdminMessage_Heartbeat) isWorkerToAdminMessage_Body() {} @@ -633,6 +661,10 @@ func (*WorkerToAdminMessage_JobProgressUpdate) isWorkerToAdminMessage_Body() {} func (*WorkerToAdminMessage_JobCompleted) isWorkerToAdminMessage_Body() {} +func (*WorkerToAdminMessage_Observations) isWorkerToAdminMessage_Body() {} + +func (*WorkerToAdminMessage_ObjectPreviewResponse) isWorkerToAdminMessage_Body() {} + // AdminToWorkerMessage carries commands and lifecycle notifications from admin. type AdminToWorkerMessage struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -646,6 +678,7 @@ type AdminToWorkerMessage struct { // *AdminToWorkerMessage_ExecuteJobRequest // *AdminToWorkerMessage_CancelRequest // *AdminToWorkerMessage_Shutdown + // *AdminToWorkerMessage_RequestObjectPreview Body isAdminToWorkerMessage_Body `protobuf_oneof:"body"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -756,6 +789,15 @@ func (x *AdminToWorkerMessage) GetShutdown() *AdminShutdown { return nil } +func (x *AdminToWorkerMessage) GetRequestObjectPreview() *RequestObjectPreview { + if x != nil { + if x, ok := x.Body.(*AdminToWorkerMessage_RequestObjectPreview); ok { + return x.RequestObjectPreview + } + } + return nil +} + type isAdminToWorkerMessage_Body interface { isAdminToWorkerMessage_Body() } @@ -784,6 +826,10 @@ type AdminToWorkerMessage_Shutdown struct { Shutdown *AdminShutdown `protobuf:"bytes,15,opt,name=shutdown,proto3,oneof"` } +type AdminToWorkerMessage_RequestObjectPreview struct { + RequestObjectPreview *RequestObjectPreview `protobuf:"bytes,16,opt,name=request_object_preview,json=requestObjectPreview,proto3,oneof"` +} + func (*AdminToWorkerMessage_Hello) isAdminToWorkerMessage_Body() {} func (*AdminToWorkerMessage_RequestConfigSchema) isAdminToWorkerMessage_Body() {} @@ -796,6 +842,8 @@ func (*AdminToWorkerMessage_CancelRequest) isAdminToWorkerMessage_Body() {} func (*AdminToWorkerMessage_Shutdown) isAdminToWorkerMessage_Body() {} +func (*AdminToWorkerMessage_RequestObjectPreview) isAdminToWorkerMessage_Body() {} + type WorkerHello struct { state protoimpl.MessageState `protogen:"open.v1"` WorkerId string `protobuf:"bytes,1,opt,name=worker_id,json=workerId,proto3" json:"worker_id,omitempty"` @@ -3928,11 +3976,353 @@ func (x *PersistedJobTypeConfig) GetUpdatedBy() string { return "" } +// WorkerObservations reports what a worker learned about the objects it +// inspected. A worker opens a table to decide whether it needs a job; what it +// saw is worth keeping either way, because for a format the cluster cannot +// read, the worker is the only thing that can describe it. +// +// Observations are not work. Admin caches them and serves them back, so a +// worker that goes away leaves its last ones standing rather than blanking a +// page. +type WorkerObservations struct { + state protoimpl.MessageState `protogen:"open.v1"` + JobType string `protobuf:"bytes,1,opt,name=job_type,json=jobType,proto3" json:"job_type,omitempty"` + Observations []*ObjectObservation `protobuf:"bytes,2,rep,name=observations,proto3" json:"observations,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *WorkerObservations) Reset() { + *x = WorkerObservations{} + mi := &file_plugin_proto_msgTypes[39] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *WorkerObservations) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*WorkerObservations) ProtoMessage() {} + +func (x *WorkerObservations) ProtoReflect() protoreflect.Message { + mi := &file_plugin_proto_msgTypes[39] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use WorkerObservations.ProtoReflect.Descriptor instead. +func (*WorkerObservations) Descriptor() ([]byte, []int) { + return file_plugin_proto_rawDescGZIP(), []int{39} +} + +func (x *WorkerObservations) GetJobType() string { + if x != nil { + return x.JobType + } + return "" +} + +func (x *WorkerObservations) GetObservations() []*ObjectObservation { + if x != nil { + return x.Observations + } + return nil +} + +// ObjectObservation is one object as a worker last saw it. +type ObjectObservation struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Identifier of the object, e.g. ["bucket", "namespace", "table"]. + ObjectId []string `protobuf:"bytes,1,rep,name=object_id,json=objectId,proto3" json:"object_id,omitempty"` + // What kind of object this is, e.g. "table". + ObjectKind string `protobuf:"bytes,2,opt,name=object_kind,json=objectKind,proto3" json:"object_kind,omitempty"` + // The object's format, e.g. "LANCE", so a reader can tell whose observation + // this is without parsing the attributes. + Format string `protobuf:"bytes,3,opt,name=format,proto3" json:"format,omitempty"` + // Whatever the worker can cheaply say: schema, row count, fragment count. + // The keys are the worker's to choose. + Attributes map[string]*ConfigValue `protobuf:"bytes,4,rep,name=attributes,proto3" json:"attributes,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` + ObservedAt *timestamppb.Timestamp `protobuf:"bytes,5,opt,name=observed_at,json=observedAt,proto3" json:"observed_at,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ObjectObservation) Reset() { + *x = ObjectObservation{} + mi := &file_plugin_proto_msgTypes[40] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ObjectObservation) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ObjectObservation) ProtoMessage() {} + +func (x *ObjectObservation) ProtoReflect() protoreflect.Message { + mi := &file_plugin_proto_msgTypes[40] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ObjectObservation.ProtoReflect.Descriptor instead. +func (*ObjectObservation) Descriptor() ([]byte, []int) { + return file_plugin_proto_rawDescGZIP(), []int{40} +} + +func (x *ObjectObservation) GetObjectId() []string { + if x != nil { + return x.ObjectId + } + return nil +} + +func (x *ObjectObservation) GetObjectKind() string { + if x != nil { + return x.ObjectKind + } + return "" +} + +func (x *ObjectObservation) GetFormat() string { + if x != nil { + return x.Format + } + return "" +} + +func (x *ObjectObservation) GetAttributes() map[string]*ConfigValue { + if x != nil { + return x.Attributes + } + return nil +} + +func (x *ObjectObservation) GetObservedAt() *timestamppb.Timestamp { + if x != nil { + return x.ObservedAt + } + return nil +} + +// RequestObjectPreview asks a worker for sample rows of an object admin cannot +// read itself. Unlike an observation this is not cached: it is fetched when +// someone opens the page, because rows are the object's data rather than a +// description of it, and holding a copy in admin is neither fresh nor its +// business. +type RequestObjectPreview struct { + state protoimpl.MessageState `protogen:"open.v1"` + ObjectId []string `protobuf:"bytes,1,rep,name=object_id,json=objectId,proto3" json:"object_id,omitempty"` + // The format admin believes this object is, so a worker that does not own it + // can decline instead of guessing. + Format string `protobuf:"bytes,2,opt,name=format,proto3" json:"format,omitempty"` + RowLimit int32 `protobuf:"varint,3,opt,name=row_limit,json=rowLimit,proto3" json:"row_limit,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *RequestObjectPreview) Reset() { + *x = RequestObjectPreview{} + mi := &file_plugin_proto_msgTypes[41] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *RequestObjectPreview) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*RequestObjectPreview) ProtoMessage() {} + +func (x *RequestObjectPreview) ProtoReflect() protoreflect.Message { + mi := &file_plugin_proto_msgTypes[41] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use RequestObjectPreview.ProtoReflect.Descriptor instead. +func (*RequestObjectPreview) Descriptor() ([]byte, []int) { + return file_plugin_proto_rawDescGZIP(), []int{41} +} + +func (x *RequestObjectPreview) GetObjectId() []string { + if x != nil { + return x.ObjectId + } + return nil +} + +func (x *RequestObjectPreview) GetFormat() string { + if x != nil { + return x.Format + } + return "" +} + +func (x *RequestObjectPreview) GetRowLimit() int32 { + if x != nil { + return x.RowLimit + } + return 0 +} + +// ObjectPreviewResponse carries the sample back, already rendered as text. The +// worker is the only thing that knows the object's types, so it formats them; +// admin displays what it is given. +type ObjectPreviewResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + RequestId string `protobuf:"bytes,6,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` + Success bool `protobuf:"varint,1,opt,name=success,proto3" json:"success,omitempty"` + ErrorMessage string `protobuf:"bytes,2,opt,name=error_message,json=errorMessage,proto3" json:"error_message,omitempty"` + Columns []string `protobuf:"bytes,3,rep,name=columns,proto3" json:"columns,omitempty"` + Rows []*PreviewRow `protobuf:"bytes,4,rep,name=rows,proto3" json:"rows,omitempty"` + // Rows in the object, which is not the number sampled. + TotalRows int64 `protobuf:"varint,5,opt,name=total_rows,json=totalRows,proto3" json:"total_rows,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ObjectPreviewResponse) Reset() { + *x = ObjectPreviewResponse{} + mi := &file_plugin_proto_msgTypes[42] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ObjectPreviewResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ObjectPreviewResponse) ProtoMessage() {} + +func (x *ObjectPreviewResponse) ProtoReflect() protoreflect.Message { + mi := &file_plugin_proto_msgTypes[42] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ObjectPreviewResponse.ProtoReflect.Descriptor instead. +func (*ObjectPreviewResponse) Descriptor() ([]byte, []int) { + return file_plugin_proto_rawDescGZIP(), []int{42} +} + +func (x *ObjectPreviewResponse) GetRequestId() string { + if x != nil { + return x.RequestId + } + return "" +} + +func (x *ObjectPreviewResponse) GetSuccess() bool { + if x != nil { + return x.Success + } + return false +} + +func (x *ObjectPreviewResponse) GetErrorMessage() string { + if x != nil { + return x.ErrorMessage + } + return "" +} + +func (x *ObjectPreviewResponse) GetColumns() []string { + if x != nil { + return x.Columns + } + return nil +} + +func (x *ObjectPreviewResponse) GetRows() []*PreviewRow { + if x != nil { + return x.Rows + } + return nil +} + +func (x *ObjectPreviewResponse) GetTotalRows() int64 { + if x != nil { + return x.TotalRows + } + return 0 +} + +type PreviewRow struct { + state protoimpl.MessageState `protogen:"open.v1"` + Values []string `protobuf:"bytes,1,rep,name=values,proto3" json:"values,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *PreviewRow) Reset() { + *x = PreviewRow{} + mi := &file_plugin_proto_msgTypes[43] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *PreviewRow) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*PreviewRow) ProtoMessage() {} + +func (x *PreviewRow) ProtoReflect() protoreflect.Message { + mi := &file_plugin_proto_msgTypes[43] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use PreviewRow.ProtoReflect.Descriptor instead. +func (*PreviewRow) Descriptor() ([]byte, []int) { + return file_plugin_proto_rawDescGZIP(), []int{43} +} + +func (x *PreviewRow) GetValues() []string { + if x != nil { + return x.Values + } + return nil +} + var File_plugin_proto protoreflect.FileDescriptor const file_plugin_proto_rawDesc = "" + "\n" + - "\fplugin.proto\x12\x06plugin\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"\x90\x05\n" + + "\fplugin.proto\x12\x06plugin\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"\xab\x06\n" + "\x14WorkerToAdminMessage\x12\x1b\n" + "\tworker_id\x18\x01 \x01(\tR\bworkerId\x123\n" + "\asent_at\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\x06sentAt\x12+\n" + @@ -3944,8 +4334,10 @@ const file_plugin_proto_rawDesc = "" + "\x13detection_proposals\x18\x0e \x01(\v2\x1a.plugin.DetectionProposalsH\x00R\x12detectionProposals\x12J\n" + "\x12detection_complete\x18\x0f \x01(\v2\x19.plugin.DetectionCompleteH\x00R\x11detectionComplete\x12K\n" + "\x13job_progress_update\x18\x10 \x01(\v2\x19.plugin.JobProgressUpdateH\x00R\x11jobProgressUpdate\x12;\n" + - "\rjob_completed\x18\x11 \x01(\v2\x14.plugin.JobCompletedH\x00R\fjobCompletedB\x06\n" + - "\x04body\"\x86\x04\n" + + "\rjob_completed\x18\x11 \x01(\v2\x14.plugin.JobCompletedH\x00R\fjobCompleted\x12@\n" + + "\fobservations\x18\x12 \x01(\v2\x1a.plugin.WorkerObservationsH\x00R\fobservations\x12W\n" + + "\x17object_preview_response\x18\x13 \x01(\v2\x1d.plugin.ObjectPreviewResponseH\x00R\x15objectPreviewResponseB\x06\n" + + "\x04body\"\xdc\x04\n" + "\x14AdminToWorkerMessage\x12\x1d\n" + "\n" + "request_id\x18\x01 \x01(\tR\trequestId\x123\n" + @@ -3956,7 +4348,8 @@ const file_plugin_proto_rawDesc = "" + "\x15run_detection_request\x18\f \x01(\v2\x1b.plugin.RunDetectionRequestH\x00R\x13runDetectionRequest\x12K\n" + "\x13execute_job_request\x18\r \x01(\v2\x19.plugin.ExecuteJobRequestH\x00R\x11executeJobRequest\x12>\n" + "\x0ecancel_request\x18\x0e \x01(\v2\x15.plugin.CancelRequestH\x00R\rcancelRequest\x123\n" + - "\bshutdown\x18\x0f \x01(\v2\x15.plugin.AdminShutdownH\x00R\bshutdownB\x06\n" + + "\bshutdown\x18\x0f \x01(\v2\x15.plugin.AdminShutdownH\x00R\bshutdown\x12T\n" + + "\x16request_object_preview\x18\x10 \x01(\v2\x1c.plugin.RequestObjectPreviewH\x00R\x14requestObjectPreviewB\x06\n" + "\x04body\"\xff\x02\n" + "\vWorkerHello\x12\x1b\n" + "\tworker_id\x18\x01 \x01(\tR\bworkerId\x12,\n" + @@ -4319,7 +4712,39 @@ const file_plugin_proto_rawDesc = "" + "\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01\x1aZ\n" + "\x17WorkerConfigValuesEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12)\n" + - "\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01*W\n" + + "\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01\"n\n" + + "\x12WorkerObservations\x12\x19\n" + + "\bjob_type\x18\x01 \x01(\tR\ajobType\x12=\n" + + "\fobservations\x18\x02 \x03(\v2\x19.plugin.ObjectObservationR\fobservations\"\xc5\x02\n" + + "\x11ObjectObservation\x12\x1b\n" + + "\tobject_id\x18\x01 \x03(\tR\bobjectId\x12\x1f\n" + + "\vobject_kind\x18\x02 \x01(\tR\n" + + "objectKind\x12\x16\n" + + "\x06format\x18\x03 \x01(\tR\x06format\x12I\n" + + "\n" + + "attributes\x18\x04 \x03(\v2).plugin.ObjectObservation.AttributesEntryR\n" + + "attributes\x12;\n" + + "\vobserved_at\x18\x05 \x01(\v2\x1a.google.protobuf.TimestampR\n" + + "observedAt\x1aR\n" + + "\x0fAttributesEntry\x12\x10\n" + + "\x03key\x18\x01 \x01(\tR\x03key\x12)\n" + + "\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01\"h\n" + + "\x14RequestObjectPreview\x12\x1b\n" + + "\tobject_id\x18\x01 \x03(\tR\bobjectId\x12\x16\n" + + "\x06format\x18\x02 \x01(\tR\x06format\x12\x1b\n" + + "\trow_limit\x18\x03 \x01(\x05R\browLimit\"\xd6\x01\n" + + "\x15ObjectPreviewResponse\x12\x1d\n" + + "\n" + + "request_id\x18\x06 \x01(\tR\trequestId\x12\x18\n" + + "\asuccess\x18\x01 \x01(\bR\asuccess\x12#\n" + + "\rerror_message\x18\x02 \x01(\tR\ferrorMessage\x12\x18\n" + + "\acolumns\x18\x03 \x03(\tR\acolumns\x12&\n" + + "\x04rows\x18\x04 \x03(\v2\x12.plugin.PreviewRowR\x04rows\x12\x1d\n" + + "\n" + + "total_rows\x18\x05 \x01(\x03R\ttotalRows\"$\n" + + "\n" + + "PreviewRow\x12\x16\n" + + "\x06values\x18\x01 \x03(\tR\x06values*W\n" + "\bWorkKind\x12\x19\n" + "\x15WORK_KIND_UNSPECIFIED\x10\x00\x12\x17\n" + "\x13WORK_KIND_DETECTION\x10\x01\x12\x17\n" + @@ -4388,7 +4813,7 @@ func file_plugin_proto_rawDescGZIP() []byte { } var file_plugin_proto_enumTypes = make([]protoimpl.EnumInfo, 7) -var file_plugin_proto_msgTypes = make([]protoimpl.MessageInfo, 59) +var file_plugin_proto_msgTypes = make([]protoimpl.MessageInfo, 65) var file_plugin_proto_goTypes = []any{ (WorkKind)(0), // 0: plugin.WorkKind (JobPriority)(0), // 1: plugin.JobPriority @@ -4436,31 +4861,37 @@ var file_plugin_proto_goTypes = []any{ (*CancelRequest)(nil), // 43: plugin.CancelRequest (*AdminShutdown)(nil), // 44: plugin.AdminShutdown (*PersistedJobTypeConfig)(nil), // 45: plugin.PersistedJobTypeConfig - nil, // 46: plugin.WorkerHello.MetadataEntry - nil, // 47: plugin.WorkerHeartbeat.QueuedJobsByTypeEntry - nil, // 48: plugin.WorkerHeartbeat.MetadataEntry - nil, // 49: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry - nil, // 50: plugin.ConfigForm.DefaultValuesEntry - nil, // 51: plugin.ValueMap.FieldsEntry - nil, // 52: plugin.RunDetectionRequest.AdminConfigValuesEntry - nil, // 53: plugin.RunDetectionRequest.WorkerConfigValuesEntry - nil, // 54: plugin.JobProposal.ParametersEntry - nil, // 55: plugin.JobProposal.LabelsEntry - nil, // 56: plugin.ExecuteJobRequest.AdminConfigValuesEntry - nil, // 57: plugin.ExecuteJobRequest.WorkerConfigValuesEntry - nil, // 58: plugin.JobSpec.ParametersEntry - nil, // 59: plugin.JobSpec.LabelsEntry - nil, // 60: plugin.JobProgressUpdate.MetricsEntry - nil, // 61: plugin.JobResult.OutputValuesEntry - nil, // 62: plugin.ClusterContext.MetadataEntry - nil, // 63: plugin.ActivityEvent.DetailsEntry - nil, // 64: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry - nil, // 65: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry - (*timestamppb.Timestamp)(nil), // 66: google.protobuf.Timestamp - (*durationpb.Duration)(nil), // 67: google.protobuf.Duration + (*WorkerObservations)(nil), // 46: plugin.WorkerObservations + (*ObjectObservation)(nil), // 47: plugin.ObjectObservation + (*RequestObjectPreview)(nil), // 48: plugin.RequestObjectPreview + (*ObjectPreviewResponse)(nil), // 49: plugin.ObjectPreviewResponse + (*PreviewRow)(nil), // 50: plugin.PreviewRow + nil, // 51: plugin.WorkerHello.MetadataEntry + nil, // 52: plugin.WorkerHeartbeat.QueuedJobsByTypeEntry + nil, // 53: plugin.WorkerHeartbeat.MetadataEntry + nil, // 54: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry + nil, // 55: plugin.ConfigForm.DefaultValuesEntry + nil, // 56: plugin.ValueMap.FieldsEntry + nil, // 57: plugin.RunDetectionRequest.AdminConfigValuesEntry + nil, // 58: plugin.RunDetectionRequest.WorkerConfigValuesEntry + nil, // 59: plugin.JobProposal.ParametersEntry + nil, // 60: plugin.JobProposal.LabelsEntry + nil, // 61: plugin.ExecuteJobRequest.AdminConfigValuesEntry + nil, // 62: plugin.ExecuteJobRequest.WorkerConfigValuesEntry + nil, // 63: plugin.JobSpec.ParametersEntry + nil, // 64: plugin.JobSpec.LabelsEntry + nil, // 65: plugin.JobProgressUpdate.MetricsEntry + nil, // 66: plugin.JobResult.OutputValuesEntry + nil, // 67: plugin.ClusterContext.MetadataEntry + nil, // 68: plugin.ActivityEvent.DetailsEntry + nil, // 69: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry + nil, // 70: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry + nil, // 71: plugin.ObjectObservation.AttributesEntry + (*timestamppb.Timestamp)(nil), // 72: google.protobuf.Timestamp + (*durationpb.Duration)(nil), // 73: google.protobuf.Duration } var file_plugin_proto_depIdxs = []int32{ - 66, // 0: plugin.WorkerToAdminMessage.sent_at:type_name -> google.protobuf.Timestamp + 72, // 0: plugin.WorkerToAdminMessage.sent_at:type_name -> google.protobuf.Timestamp 9, // 1: plugin.WorkerToAdminMessage.hello:type_name -> plugin.WorkerHello 11, // 2: plugin.WorkerToAdminMessage.heartbeat:type_name -> plugin.WorkerHeartbeat 12, // 3: plugin.WorkerToAdminMessage.acknowledge:type_name -> plugin.WorkerAcknowledge @@ -4469,104 +4900,112 @@ var file_plugin_proto_depIdxs = []int32{ 34, // 6: plugin.WorkerToAdminMessage.detection_complete:type_name -> plugin.DetectionComplete 38, // 7: plugin.WorkerToAdminMessage.job_progress_update:type_name -> plugin.JobProgressUpdate 39, // 8: plugin.WorkerToAdminMessage.job_completed:type_name -> plugin.JobCompleted - 66, // 9: plugin.AdminToWorkerMessage.sent_at:type_name -> google.protobuf.Timestamp - 10, // 10: plugin.AdminToWorkerMessage.hello:type_name -> plugin.AdminHello - 15, // 11: plugin.AdminToWorkerMessage.request_config_schema:type_name -> plugin.RequestConfigSchema - 32, // 12: plugin.AdminToWorkerMessage.run_detection_request:type_name -> plugin.RunDetectionRequest - 36, // 13: plugin.AdminToWorkerMessage.execute_job_request:type_name -> plugin.ExecuteJobRequest - 43, // 14: plugin.AdminToWorkerMessage.cancel_request:type_name -> plugin.CancelRequest - 44, // 15: plugin.AdminToWorkerMessage.shutdown:type_name -> plugin.AdminShutdown - 14, // 16: plugin.WorkerHello.capabilities:type_name -> plugin.JobTypeCapability - 46, // 17: plugin.WorkerHello.metadata:type_name -> plugin.WorkerHello.MetadataEntry - 13, // 18: plugin.WorkerHeartbeat.running_work:type_name -> plugin.RunningWork - 47, // 19: plugin.WorkerHeartbeat.queued_jobs_by_type:type_name -> plugin.WorkerHeartbeat.QueuedJobsByTypeEntry - 48, // 20: plugin.WorkerHeartbeat.metadata:type_name -> plugin.WorkerHeartbeat.MetadataEntry - 0, // 21: plugin.RunningWork.kind:type_name -> plugin.WorkKind - 2, // 22: plugin.RunningWork.state:type_name -> plugin.JobState - 17, // 23: plugin.ConfigSchemaResponse.job_type_descriptor:type_name -> plugin.JobTypeDescriptor - 18, // 24: plugin.JobTypeDescriptor.admin_config_form:type_name -> plugin.ConfigForm - 18, // 25: plugin.JobTypeDescriptor.worker_config_form:type_name -> plugin.ConfigForm - 30, // 26: plugin.JobTypeDescriptor.admin_runtime_defaults:type_name -> plugin.AdminRuntimeDefaults - 49, // 27: plugin.JobTypeDescriptor.worker_default_values:type_name -> plugin.JobTypeDescriptor.WorkerDefaultValuesEntry - 19, // 28: plugin.ConfigForm.sections:type_name -> plugin.ConfigSection - 50, // 29: plugin.ConfigForm.default_values:type_name -> plugin.ConfigForm.DefaultValuesEntry - 20, // 30: plugin.ConfigSection.fields:type_name -> plugin.ConfigField - 3, // 31: plugin.ConfigField.field_type:type_name -> plugin.ConfigFieldType - 4, // 32: plugin.ConfigField.widget:type_name -> plugin.ConfigWidget - 23, // 33: plugin.ConfigField.min_value:type_name -> plugin.ConfigValue - 23, // 34: plugin.ConfigField.max_value:type_name -> plugin.ConfigValue - 21, // 35: plugin.ConfigField.options:type_name -> plugin.ConfigOption - 22, // 36: plugin.ConfigField.validation_rules:type_name -> plugin.ValidationRule - 23, // 37: plugin.ConfigField.visible_when_equals:type_name -> plugin.ConfigValue - 5, // 38: plugin.ValidationRule.type:type_name -> plugin.ValidationRuleType - 67, // 39: plugin.ConfigValue.duration_value:type_name -> google.protobuf.Duration - 24, // 40: plugin.ConfigValue.string_list:type_name -> plugin.StringList - 25, // 41: plugin.ConfigValue.int64_list:type_name -> plugin.Int64List - 26, // 42: plugin.ConfigValue.double_list:type_name -> plugin.DoubleList - 27, // 43: plugin.ConfigValue.bool_list:type_name -> plugin.BoolList - 28, // 44: plugin.ConfigValue.list_value:type_name -> plugin.ValueList - 29, // 45: plugin.ConfigValue.map_value:type_name -> plugin.ValueMap - 23, // 46: plugin.ValueList.values:type_name -> plugin.ConfigValue - 51, // 47: plugin.ValueMap.fields:type_name -> plugin.ValueMap.FieldsEntry - 31, // 48: plugin.RunDetectionRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig - 52, // 49: plugin.RunDetectionRequest.admin_config_values:type_name -> plugin.RunDetectionRequest.AdminConfigValuesEntry - 53, // 50: plugin.RunDetectionRequest.worker_config_values:type_name -> plugin.RunDetectionRequest.WorkerConfigValuesEntry - 41, // 51: plugin.RunDetectionRequest.cluster_context:type_name -> plugin.ClusterContext - 66, // 52: plugin.RunDetectionRequest.last_successful_run:type_name -> google.protobuf.Timestamp - 35, // 53: plugin.DetectionProposals.proposals:type_name -> plugin.JobProposal - 1, // 54: plugin.JobProposal.priority:type_name -> plugin.JobPriority - 54, // 55: plugin.JobProposal.parameters:type_name -> plugin.JobProposal.ParametersEntry - 55, // 56: plugin.JobProposal.labels:type_name -> plugin.JobProposal.LabelsEntry - 66, // 57: plugin.JobProposal.not_before:type_name -> google.protobuf.Timestamp - 66, // 58: plugin.JobProposal.expires_at:type_name -> google.protobuf.Timestamp - 37, // 59: plugin.ExecuteJobRequest.job:type_name -> plugin.JobSpec - 31, // 60: plugin.ExecuteJobRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig - 56, // 61: plugin.ExecuteJobRequest.admin_config_values:type_name -> plugin.ExecuteJobRequest.AdminConfigValuesEntry - 57, // 62: plugin.ExecuteJobRequest.worker_config_values:type_name -> plugin.ExecuteJobRequest.WorkerConfigValuesEntry - 41, // 63: plugin.ExecuteJobRequest.cluster_context:type_name -> plugin.ClusterContext - 1, // 64: plugin.JobSpec.priority:type_name -> plugin.JobPriority - 58, // 65: plugin.JobSpec.parameters:type_name -> plugin.JobSpec.ParametersEntry - 59, // 66: plugin.JobSpec.labels:type_name -> plugin.JobSpec.LabelsEntry - 66, // 67: plugin.JobSpec.created_at:type_name -> google.protobuf.Timestamp - 66, // 68: plugin.JobSpec.scheduled_at:type_name -> google.protobuf.Timestamp - 2, // 69: plugin.JobProgressUpdate.state:type_name -> plugin.JobState - 60, // 70: plugin.JobProgressUpdate.metrics:type_name -> plugin.JobProgressUpdate.MetricsEntry - 42, // 71: plugin.JobProgressUpdate.activities:type_name -> plugin.ActivityEvent - 66, // 72: plugin.JobProgressUpdate.updated_at:type_name -> google.protobuf.Timestamp - 40, // 73: plugin.JobCompleted.result:type_name -> plugin.JobResult - 42, // 74: plugin.JobCompleted.activities:type_name -> plugin.ActivityEvent - 66, // 75: plugin.JobCompleted.completed_at:type_name -> google.protobuf.Timestamp - 61, // 76: plugin.JobResult.output_values:type_name -> plugin.JobResult.OutputValuesEntry - 62, // 77: plugin.ClusterContext.metadata:type_name -> plugin.ClusterContext.MetadataEntry - 6, // 78: plugin.ActivityEvent.source:type_name -> plugin.ActivitySource - 63, // 79: plugin.ActivityEvent.details:type_name -> plugin.ActivityEvent.DetailsEntry - 66, // 80: plugin.ActivityEvent.created_at:type_name -> google.protobuf.Timestamp - 0, // 81: plugin.CancelRequest.target_kind:type_name -> plugin.WorkKind - 64, // 82: plugin.PersistedJobTypeConfig.admin_config_values:type_name -> plugin.PersistedJobTypeConfig.AdminConfigValuesEntry - 65, // 83: plugin.PersistedJobTypeConfig.worker_config_values:type_name -> plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry - 31, // 84: plugin.PersistedJobTypeConfig.admin_runtime:type_name -> plugin.AdminRuntimeConfig - 66, // 85: plugin.PersistedJobTypeConfig.updated_at:type_name -> google.protobuf.Timestamp - 23, // 86: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 87: plugin.ConfigForm.DefaultValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 88: plugin.ValueMap.FieldsEntry.value:type_name -> plugin.ConfigValue - 23, // 89: plugin.RunDetectionRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 90: plugin.RunDetectionRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 91: plugin.JobProposal.ParametersEntry.value:type_name -> plugin.ConfigValue - 23, // 92: plugin.ExecuteJobRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 93: plugin.ExecuteJobRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 94: plugin.JobSpec.ParametersEntry.value:type_name -> plugin.ConfigValue - 23, // 95: plugin.JobProgressUpdate.MetricsEntry.value:type_name -> plugin.ConfigValue - 23, // 96: plugin.JobResult.OutputValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 97: plugin.ActivityEvent.DetailsEntry.value:type_name -> plugin.ConfigValue - 23, // 98: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue - 23, // 99: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue - 7, // 100: plugin.PluginControlService.WorkerStream:input_type -> plugin.WorkerToAdminMessage - 8, // 101: plugin.PluginControlService.WorkerStream:output_type -> plugin.AdminToWorkerMessage - 101, // [101:102] is the sub-list for method output_type - 100, // [100:101] is the sub-list for method input_type - 100, // [100:100] is the sub-list for extension type_name - 100, // [100:100] is the sub-list for extension extendee - 0, // [0:100] is the sub-list for field type_name + 46, // 9: plugin.WorkerToAdminMessage.observations:type_name -> plugin.WorkerObservations + 49, // 10: plugin.WorkerToAdminMessage.object_preview_response:type_name -> plugin.ObjectPreviewResponse + 72, // 11: plugin.AdminToWorkerMessage.sent_at:type_name -> google.protobuf.Timestamp + 10, // 12: plugin.AdminToWorkerMessage.hello:type_name -> plugin.AdminHello + 15, // 13: plugin.AdminToWorkerMessage.request_config_schema:type_name -> plugin.RequestConfigSchema + 32, // 14: plugin.AdminToWorkerMessage.run_detection_request:type_name -> plugin.RunDetectionRequest + 36, // 15: plugin.AdminToWorkerMessage.execute_job_request:type_name -> plugin.ExecuteJobRequest + 43, // 16: plugin.AdminToWorkerMessage.cancel_request:type_name -> plugin.CancelRequest + 44, // 17: plugin.AdminToWorkerMessage.shutdown:type_name -> plugin.AdminShutdown + 48, // 18: plugin.AdminToWorkerMessage.request_object_preview:type_name -> plugin.RequestObjectPreview + 14, // 19: plugin.WorkerHello.capabilities:type_name -> plugin.JobTypeCapability + 51, // 20: plugin.WorkerHello.metadata:type_name -> plugin.WorkerHello.MetadataEntry + 13, // 21: plugin.WorkerHeartbeat.running_work:type_name -> plugin.RunningWork + 52, // 22: plugin.WorkerHeartbeat.queued_jobs_by_type:type_name -> plugin.WorkerHeartbeat.QueuedJobsByTypeEntry + 53, // 23: plugin.WorkerHeartbeat.metadata:type_name -> plugin.WorkerHeartbeat.MetadataEntry + 0, // 24: plugin.RunningWork.kind:type_name -> plugin.WorkKind + 2, // 25: plugin.RunningWork.state:type_name -> plugin.JobState + 17, // 26: plugin.ConfigSchemaResponse.job_type_descriptor:type_name -> plugin.JobTypeDescriptor + 18, // 27: plugin.JobTypeDescriptor.admin_config_form:type_name -> plugin.ConfigForm + 18, // 28: plugin.JobTypeDescriptor.worker_config_form:type_name -> plugin.ConfigForm + 30, // 29: plugin.JobTypeDescriptor.admin_runtime_defaults:type_name -> plugin.AdminRuntimeDefaults + 54, // 30: plugin.JobTypeDescriptor.worker_default_values:type_name -> plugin.JobTypeDescriptor.WorkerDefaultValuesEntry + 19, // 31: plugin.ConfigForm.sections:type_name -> plugin.ConfigSection + 55, // 32: plugin.ConfigForm.default_values:type_name -> plugin.ConfigForm.DefaultValuesEntry + 20, // 33: plugin.ConfigSection.fields:type_name -> plugin.ConfigField + 3, // 34: plugin.ConfigField.field_type:type_name -> plugin.ConfigFieldType + 4, // 35: plugin.ConfigField.widget:type_name -> plugin.ConfigWidget + 23, // 36: plugin.ConfigField.min_value:type_name -> plugin.ConfigValue + 23, // 37: plugin.ConfigField.max_value:type_name -> plugin.ConfigValue + 21, // 38: plugin.ConfigField.options:type_name -> plugin.ConfigOption + 22, // 39: plugin.ConfigField.validation_rules:type_name -> plugin.ValidationRule + 23, // 40: plugin.ConfigField.visible_when_equals:type_name -> plugin.ConfigValue + 5, // 41: plugin.ValidationRule.type:type_name -> plugin.ValidationRuleType + 73, // 42: plugin.ConfigValue.duration_value:type_name -> google.protobuf.Duration + 24, // 43: plugin.ConfigValue.string_list:type_name -> plugin.StringList + 25, // 44: plugin.ConfigValue.int64_list:type_name -> plugin.Int64List + 26, // 45: plugin.ConfigValue.double_list:type_name -> plugin.DoubleList + 27, // 46: plugin.ConfigValue.bool_list:type_name -> plugin.BoolList + 28, // 47: plugin.ConfigValue.list_value:type_name -> plugin.ValueList + 29, // 48: plugin.ConfigValue.map_value:type_name -> plugin.ValueMap + 23, // 49: plugin.ValueList.values:type_name -> plugin.ConfigValue + 56, // 50: plugin.ValueMap.fields:type_name -> plugin.ValueMap.FieldsEntry + 31, // 51: plugin.RunDetectionRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig + 57, // 52: plugin.RunDetectionRequest.admin_config_values:type_name -> plugin.RunDetectionRequest.AdminConfigValuesEntry + 58, // 53: plugin.RunDetectionRequest.worker_config_values:type_name -> plugin.RunDetectionRequest.WorkerConfigValuesEntry + 41, // 54: plugin.RunDetectionRequest.cluster_context:type_name -> plugin.ClusterContext + 72, // 55: plugin.RunDetectionRequest.last_successful_run:type_name -> google.protobuf.Timestamp + 35, // 56: plugin.DetectionProposals.proposals:type_name -> plugin.JobProposal + 1, // 57: plugin.JobProposal.priority:type_name -> plugin.JobPriority + 59, // 58: plugin.JobProposal.parameters:type_name -> plugin.JobProposal.ParametersEntry + 60, // 59: plugin.JobProposal.labels:type_name -> plugin.JobProposal.LabelsEntry + 72, // 60: plugin.JobProposal.not_before:type_name -> google.protobuf.Timestamp + 72, // 61: plugin.JobProposal.expires_at:type_name -> google.protobuf.Timestamp + 37, // 62: plugin.ExecuteJobRequest.job:type_name -> plugin.JobSpec + 31, // 63: plugin.ExecuteJobRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig + 61, // 64: plugin.ExecuteJobRequest.admin_config_values:type_name -> plugin.ExecuteJobRequest.AdminConfigValuesEntry + 62, // 65: plugin.ExecuteJobRequest.worker_config_values:type_name -> plugin.ExecuteJobRequest.WorkerConfigValuesEntry + 41, // 66: plugin.ExecuteJobRequest.cluster_context:type_name -> plugin.ClusterContext + 1, // 67: plugin.JobSpec.priority:type_name -> plugin.JobPriority + 63, // 68: plugin.JobSpec.parameters:type_name -> plugin.JobSpec.ParametersEntry + 64, // 69: plugin.JobSpec.labels:type_name -> plugin.JobSpec.LabelsEntry + 72, // 70: plugin.JobSpec.created_at:type_name -> google.protobuf.Timestamp + 72, // 71: plugin.JobSpec.scheduled_at:type_name -> google.protobuf.Timestamp + 2, // 72: plugin.JobProgressUpdate.state:type_name -> plugin.JobState + 65, // 73: plugin.JobProgressUpdate.metrics:type_name -> plugin.JobProgressUpdate.MetricsEntry + 42, // 74: plugin.JobProgressUpdate.activities:type_name -> plugin.ActivityEvent + 72, // 75: plugin.JobProgressUpdate.updated_at:type_name -> google.protobuf.Timestamp + 40, // 76: plugin.JobCompleted.result:type_name -> plugin.JobResult + 42, // 77: plugin.JobCompleted.activities:type_name -> plugin.ActivityEvent + 72, // 78: plugin.JobCompleted.completed_at:type_name -> google.protobuf.Timestamp + 66, // 79: plugin.JobResult.output_values:type_name -> plugin.JobResult.OutputValuesEntry + 67, // 80: plugin.ClusterContext.metadata:type_name -> plugin.ClusterContext.MetadataEntry + 6, // 81: plugin.ActivityEvent.source:type_name -> plugin.ActivitySource + 68, // 82: plugin.ActivityEvent.details:type_name -> plugin.ActivityEvent.DetailsEntry + 72, // 83: plugin.ActivityEvent.created_at:type_name -> google.protobuf.Timestamp + 0, // 84: plugin.CancelRequest.target_kind:type_name -> plugin.WorkKind + 69, // 85: plugin.PersistedJobTypeConfig.admin_config_values:type_name -> plugin.PersistedJobTypeConfig.AdminConfigValuesEntry + 70, // 86: plugin.PersistedJobTypeConfig.worker_config_values:type_name -> plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry + 31, // 87: plugin.PersistedJobTypeConfig.admin_runtime:type_name -> plugin.AdminRuntimeConfig + 72, // 88: plugin.PersistedJobTypeConfig.updated_at:type_name -> google.protobuf.Timestamp + 47, // 89: plugin.WorkerObservations.observations:type_name -> plugin.ObjectObservation + 71, // 90: plugin.ObjectObservation.attributes:type_name -> plugin.ObjectObservation.AttributesEntry + 72, // 91: plugin.ObjectObservation.observed_at:type_name -> google.protobuf.Timestamp + 50, // 92: plugin.ObjectPreviewResponse.rows:type_name -> plugin.PreviewRow + 23, // 93: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 94: plugin.ConfigForm.DefaultValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 95: plugin.ValueMap.FieldsEntry.value:type_name -> plugin.ConfigValue + 23, // 96: plugin.RunDetectionRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 97: plugin.RunDetectionRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 98: plugin.JobProposal.ParametersEntry.value:type_name -> plugin.ConfigValue + 23, // 99: plugin.ExecuteJobRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 100: plugin.ExecuteJobRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 101: plugin.JobSpec.ParametersEntry.value:type_name -> plugin.ConfigValue + 23, // 102: plugin.JobProgressUpdate.MetricsEntry.value:type_name -> plugin.ConfigValue + 23, // 103: plugin.JobResult.OutputValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 104: plugin.ActivityEvent.DetailsEntry.value:type_name -> plugin.ConfigValue + 23, // 105: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 106: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue + 23, // 107: plugin.ObjectObservation.AttributesEntry.value:type_name -> plugin.ConfigValue + 7, // 108: plugin.PluginControlService.WorkerStream:input_type -> plugin.WorkerToAdminMessage + 8, // 109: plugin.PluginControlService.WorkerStream:output_type -> plugin.AdminToWorkerMessage + 109, // [109:110] is the sub-list for method output_type + 108, // [108:109] is the sub-list for method input_type + 108, // [108:108] is the sub-list for extension type_name + 108, // [108:108] is the sub-list for extension extendee + 0, // [0:108] is the sub-list for field type_name } func init() { file_plugin_proto_init() } @@ -4583,6 +5022,8 @@ func file_plugin_proto_init() { (*WorkerToAdminMessage_DetectionComplete)(nil), (*WorkerToAdminMessage_JobProgressUpdate)(nil), (*WorkerToAdminMessage_JobCompleted)(nil), + (*WorkerToAdminMessage_Observations)(nil), + (*WorkerToAdminMessage_ObjectPreviewResponse)(nil), } file_plugin_proto_msgTypes[1].OneofWrappers = []any{ (*AdminToWorkerMessage_Hello)(nil), @@ -4591,6 +5032,7 @@ func file_plugin_proto_init() { (*AdminToWorkerMessage_ExecuteJobRequest)(nil), (*AdminToWorkerMessage_CancelRequest)(nil), (*AdminToWorkerMessage_Shutdown)(nil), + (*AdminToWorkerMessage_RequestObjectPreview)(nil), } file_plugin_proto_msgTypes[16].OneofWrappers = []any{ (*ConfigValue_BoolValue)(nil), @@ -4612,7 +5054,7 @@ func file_plugin_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_plugin_proto_rawDesc), len(file_plugin_proto_rawDesc)), NumEnums: 7, - NumMessages: 59, + NumMessages: 65, NumExtensions: 0, NumServices: 1, }, diff --git a/weed/s3api/bucket_paths.go b/weed/s3api/bucket_paths.go index f3febef4f..c143e213e 100644 --- a/weed/s3api/bucket_paths.go +++ b/weed/s3api/bucket_paths.go @@ -69,6 +69,11 @@ func (s3a *S3ApiServer) validateTableBucketObjectPath(bucket, object string) err return err } parts := strings.SplitN(cleanObject, "/", 4) + // A table's marker files sit at namespace/table/, one level above + // everything else, so they are three parts rather than four. + if len(parts) == 3 && s3tables.IsTableMarkerFile(parts[2]) { + return nil + } if len(parts) < 4 { return &s3tables.IcebergLayoutError{ Code: s3tables.ErrCodeInvalidIcebergLayout, diff --git a/weed/s3api/iceberg/utils.go b/weed/s3api/iceberg/utils.go index 266d975e3..3014a8ac3 100644 --- a/weed/s3api/iceberg/utils.go +++ b/weed/s3api/iceberg/utils.go @@ -169,14 +169,38 @@ func writeManagerError(w http.ResponseWriter, err error) { writeError(w, http.StatusBadRequest, "BadRequestException", err.Error()) return } - // A missing table bucket means the catalog the client selected does not - // exist, not a server fault. The storage-layer message names the resolved - // bucket, which for a client that sent no warehouse at all is the default - // one it never asked for, so say how to select a real table bucket. + // Storage-layer failures are mostly the client's, not the server's. Reporting + // a missing namespace or a name conflict as a 500 makes the catalog look + // broken and gives the client nothing to act on. var tableErr *s3tables.S3TablesError - if errors.As(err, &tableErr) && tableErr.Type == s3tables.ErrCodeNoSuchBucket { - writeError(w, http.StatusNotFound, "NoSuchNamespaceException", - fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3:/// or /v1//", tableErr.Message)) + if errors.As(err, &tableErr) { + switch tableErr.Type { + case s3tables.ErrCodeNoSuchBucket: + // The storage-layer message names the resolved bucket, which for a + // client that sent no warehouse at all is the default one it never + // asked for, so say how to select a real table bucket. + writeError(w, http.StatusNotFound, "NoSuchNamespaceException", + fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3:/// or /v1//", tableErr.Message)) + case s3tables.ErrCodeNoSuchNamespace: + writeError(w, http.StatusNotFound, "NoSuchNamespaceException", tableErr.Message) + case s3tables.ErrCodeNoSuchTable: + writeError(w, http.StatusNotFound, "NoSuchTableException", tableErr.Message) + case s3tables.ErrCodeNoSuchView: + writeError(w, http.StatusNotFound, "NoSuchViewException", tableErr.Message) + case s3tables.ErrCodeNamespaceAlreadyExists, s3tables.ErrCodeBucketAlreadyExists, + s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists: + writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message) + case s3tables.ErrCodeNamespaceNotEmpty, s3tables.ErrCodeBucketNotEmpty: + writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message) + case s3tables.ErrCodeConflict: + writeError(w, http.StatusConflict, "CommitFailedException", tableErr.Message) + case s3tables.ErrCodeAccessDenied: + writeError(w, http.StatusForbidden, "ForbiddenException", tableErr.Message) + case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout: + writeError(w, http.StatusBadRequest, "BadRequestException", tableErr.Message) + default: + writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error()) + } return } writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error()) diff --git a/weed/s3api/lance/errors.go b/weed/s3api/lance/errors.go new file mode 100644 index 000000000..f40062869 --- /dev/null +++ b/weed/s3api/lance/errors.go @@ -0,0 +1,100 @@ +package lance + +import ( + "encoding/json" + "errors" + "net/http" + + "github.com/seaweedfs/seaweedfs/weed/glog" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// Lance Namespace error codes. The spec numbers them, so a client switches on +// the code rather than parsing the message. +const ( + codeUnsupported = 0 + codeNamespaceNotFound = 1 + codeNamespaceAlreadyExists = 2 + codeNamespaceNotEmpty = 3 + codeTableNotFound = 4 + codeTableAlreadyExists = 5 + codeTableVersionNotFound = 11 + codeInvalidInput = 13 + codeConcurrentModification = 14 + codePermissionDenied = 15 + codeUnauthenticated = 16 + codeInternal = 18 +) + +type errorResponse struct { + Error string `json:"error,omitempty"` + Code int `json:"code"` + Detail string `json:"detail,omitempty"` + Instance string `json:"instance,omitempty"` +} + +func writeJSON(w http.ResponseWriter, status int, body interface{}) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + if body == nil { + return + } + if err := json.NewEncoder(w).Encode(body); err != nil { + glog.Warningf("lance: failed to encode response: %v", err) + } +} + +func writeError(w http.ResponseWriter, r *http.Request, status, code int, message string) { + instance := "" + if r != nil { + instance = r.URL.Path + } + writeJSON(w, status, errorResponse{Error: message, Code: code, Instance: instance}) +} + +// writeStorageError translates an S3 Tables storage error into the Lance error +// model. A table bucket is the first namespace level here, so a missing bucket +// is a missing namespace rather than a missing catalog. +func writeStorageError(w http.ResponseWriter, r *http.Request, err error) { + var storageErr *s3tables.S3TablesError + if !errors.As(err, &storageErr) { + writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error()) + return + } + + status, code := http.StatusInternalServerError, codeInternal + switch storageErr.Type { + case s3tables.ErrCodeNoSuchBucket, s3tables.ErrCodeNoSuchNamespace: + status, code = http.StatusNotFound, codeNamespaceNotFound + case s3tables.ErrCodeNoSuchTable, s3tables.ErrCodeNoSuchView: + status, code = http.StatusNotFound, codeTableNotFound + case s3tables.ErrCodeBucketAlreadyExists, s3tables.ErrCodeNamespaceAlreadyExists: + status, code = http.StatusConflict, codeNamespaceAlreadyExists + case s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists: + status, code = http.StatusConflict, codeTableAlreadyExists + case s3tables.ErrCodeBucketNotEmpty, s3tables.ErrCodeNamespaceNotEmpty: + status, code = http.StatusConflict, codeNamespaceNotEmpty + case s3tables.ErrCodeConflict: + status, code = http.StatusConflict, codeConcurrentModification + case s3tables.ErrCodeAccessDenied: + status, code = http.StatusForbidden, codePermissionDenied + case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout: + status, code = http.StatusBadRequest, codeInvalidInput + } + writeError(w, r, status, code, storageErr.Message) +} + +// isNotFound reports whether a storage error means the object is absent, so +// exists-style handlers can answer without a second lookup. +func isNotFound(err error) bool { + var storageErr *s3tables.S3TablesError + if !errors.As(err, &storageErr) { + return false + } + switch storageErr.Type { + case s3tables.ErrCodeNoSuchBucket, s3tables.ErrCodeNoSuchNamespace, + s3tables.ErrCodeNoSuchTable, s3tables.ErrCodeNoSuchView: + return true + } + return false +} diff --git a/weed/s3api/lance/handlers_namespace.go b/weed/s3api/lance/handlers_namespace.go new file mode 100644 index 000000000..9d3a51b2d --- /dev/null +++ b/weed/s3api/lance/handlers_namespace.go @@ -0,0 +1,341 @@ +package lance + +import ( + "net/http" + "strings" + + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// handleCreateNamespace creates a table bucket for a one-part identifier and a +// namespace inside one for anything deeper. The root cannot be created. +func (s *Server) handleCreateNamespace(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + + var req CreateNamespaceRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + if len(id) == 0 { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "the root namespace always exists and cannot be created") + return + } + + mode := normalizeMode(req.Mode, modeCreate) + switch mode { + case modeCreate, modeExistOk, modeOverwrite: + default: + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "mode must be Create, ExistOk or Overwrite") + return + } + + bucket, ns := id.namespace() + exists, err := s.namespaceExists(r, bucket, ns) + if err != nil { + writeStorageError(w, r, err) + return + } + if exists { + switch mode { + case modeExistOk: + writeJSON(w, http.StatusOK, CreateNamespaceResponse{Properties: req.Properties}) + return + case modeCreate: + writeError(w, r, http.StatusConflict, codeNamespaceAlreadyExists, "namespace already exists") + return + } + // Overwrite replaces the namespace with an empty one, so the drop has to + // succeed first. A namespace holding tables refuses, which is the point. + if err := s.dropNamespace(r, bucket, ns); err != nil { + writeStorageError(w, r, err) + return + } + } + + if err := s.createNamespace(r, bucket, ns, req.Properties); err != nil { + writeStorageError(w, r, err) + return + } + writeJSON(w, http.StatusOK, CreateNamespaceResponse{Properties: normalizeProperties(req.Properties)}) +} + +// handleListNamespaces lists the children of a namespace: table buckets at the +// root, and the next path component below that. +func (s *Server) handleListNamespaces(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + bucket, ns := id.namespace() + + if bucket == "" { + var resp s3tables.ListTableBucketsResponse + req := &s3tables.ListTableBucketsRequest{ + ContinuationToken: r.URL.Query().Get("page_token"), + MaxBuckets: pageSize(r), + } + if err := s.execute(r, "ListTableBuckets", req, &resp); err != nil { + writeStorageError(w, r, err) + return + } + names := make([]string, 0, len(resp.TableBuckets)) + for _, b := range resp.TableBuckets { + names = append(names, b.Name) + } + writeJSON(w, http.StatusOK, ListNamespacesResponse{Namespaces: names, PageToken: resp.ContinuationToken}) + return + } + + var resp s3tables.ListNamespacesResponse + req := &s3tables.ListNamespacesRequest{ + TableBucketARN: bucketARN(bucket), + ContinuationToken: r.URL.Query().Get("page_token"), + MaxNamespaces: pageSize(r), + } + if len(ns) > 0 { + req.Prefix = strings.Join(ns, ".") + "." + } + if err := s.execute(r, "ListNamespaces", req, &resp); err != nil { + writeStorageError(w, r, err) + return + } + + // Storage namespaces are full paths; the spec wants the child name relative + // to the parent, so take the next component and drop repeats. + children := make([]string, 0, len(resp.Namespaces)) + seen := make(map[string]struct{}, len(resp.Namespaces)) + for _, summary := range resp.Namespaces { + if len(summary.Namespace) <= len(ns) { + continue + } + child := summary.Namespace[len(ns)] + if _, done := seen[child]; done { + continue + } + seen[child] = struct{}{} + children = append(children, child) + } + writeJSON(w, http.StatusOK, ListNamespacesResponse{Namespaces: children, PageToken: resp.ContinuationToken}) +} + +// handleDescribeNamespace returns a namespace's properties. +func (s *Server) handleDescribeNamespace(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req DescribeNamespaceRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + + bucket, ns := id.namespace() + if bucket == "" { + writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: map[string]string{}}) + return + } + if len(ns) == 0 { + var resp s3tables.GetTableBucketResponse + if err := s.execute(r, "GetTableBucket", &s3tables.GetTableBucketRequest{TableBucketARN: bucketARN(bucket)}, &resp); err != nil { + writeStorageError(w, r, err) + return + } + writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: map[string]string{}}) + return + } + + var resp s3tables.GetNamespaceResponse + req2 := &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns} + if err := s.execute(r, "GetNamespace", req2, &resp); err != nil { + writeStorageError(w, r, err) + return + } + writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: normalizeProperties(resp.Properties)}) +} + +// handleDropNamespace removes a namespace or table bucket. +func (s *Server) handleDropNamespace(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req DropNamespaceRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + if len(id) == 0 { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "the root namespace cannot be dropped") + return + } + + if normalizeMode(req.Behavior, behaviorRestrict) == behaviorCascade { + writeError(w, r, http.StatusNotImplemented, codeUnsupported, + "cascade drop is not supported; drop the tables in the namespace first") + return + } + + bucket, ns := id.namespace() + err := s.dropNamespace(r, bucket, ns) + if err == nil { + writeJSON(w, http.StatusOK, DropNamespaceResponse{}) + return + } + if isNotFound(err) { + // Skip reports success on a missing namespace; Fail reports 400 rather + // than 404, which is what the spec asks for on this operation alone. + if normalizeMode(req.Mode, modeFail) == modeSkip { + w.WriteHeader(http.StatusNoContent) + return + } + writeError(w, r, http.StatusBadRequest, codeNamespaceNotFound, "namespace does not exist") + return + } + writeStorageError(w, r, err) +} + +// handleNamespaceExists answers with the status code and no body. +func (s *Server) handleNamespaceExists(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req NamespaceExistsRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + + bucket, ns := id.namespace() + if bucket == "" { + w.WriteHeader(http.StatusOK) + return + } + exists, err := s.namespaceExists(r, bucket, ns) + if err != nil { + writeStorageError(w, r, err) + return + } + if !exists { + writeError(w, r, http.StatusNotFound, codeNamespaceNotFound, "namespace does not exist") + return + } + w.WriteHeader(http.StatusOK) +} + +func (s *Server) namespaceExists(r *http.Request, bucket string, ns []string) (bool, error) { + var err error + if len(ns) == 0 { + var resp s3tables.GetTableBucketResponse + err = s.execute(r, "GetTableBucket", &s3tables.GetTableBucketRequest{TableBucketARN: bucketARN(bucket)}, &resp) + } else { + var resp s3tables.GetNamespaceResponse + err = s.execute(r, "GetNamespace", &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}, &resp) + } + if err == nil { + return true, nil + } + if isNotFound(err) { + return false, nil + } + return false, err +} + +func (s *Server) createNamespace(r *http.Request, bucket string, ns []string, properties map[string]string) error { + if len(ns) == 0 { + // A bucket made through this surface holds Lance tables. Saying so is + // what stops it from being described to a client as an Iceberg catalog. + var resp s3tables.CreateTableBucketResponse + return s.execute(r, "CreateTableBucket", &s3tables.CreateTableBucketRequest{ + Name: bucket, + Format: s3tables.FormatLance, + }, &resp) + } + // A table bucket is a tenant resource with its own policy and lifecycle, so + // it is created deliberately, never as a side effect of naming a namespace + // inside it. + // The immediate parent has to exist too. Storage keeps a namespace's parts + // flattened, so creating "a.b" without "a" leaves an intermediate that + // listing derives from the name and describe then denies exists. The spec + // asks for NamespaceNotFound here, which also keeps the two consistent. + if len(ns) > 1 { + parent := ns[:len(ns)-1] + if exists, err := s.namespaceExists(r, bucket, parent); err != nil { + return err + } else if !exists { + return &s3tables.S3TablesError{ + Type: s3tables.ErrCodeNoSuchNamespace, + Message: "parent namespace " + strings.Join(parent, ".") + " does not exist", + } + } + } + if exists, err := s.namespaceExists(r, bucket, nil); err != nil { + return err + } else if !exists { + return &s3tables.S3TablesError{ + Type: s3tables.ErrCodeNoSuchBucket, + Message: "table bucket " + bucket + " does not exist", + } + } + var resp s3tables.CreateNamespaceResponse + req := &s3tables.CreateNamespaceRequest{ + TableBucketARN: bucketARN(bucket), + Namespace: ns, + Properties: properties, + } + return s.execute(r, "CreateNamespace", req, &resp) +} + +func (s *Server) dropNamespace(r *http.Request, bucket string, ns []string) error { + if len(ns) == 0 { + return s.execute(r, "DeleteTableBucket", &s3tables.DeleteTableBucketRequest{TableBucketARN: bucketARN(bucket)}, nil) + } + req := &s3tables.DeleteNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns} + return s.execute(r, "DeleteNamespace", req, nil) +} + +// normalizeMode folds a spec mode or behavior value, which is case-insensitive +// and spelled either PascalCase or snake_case, onto its lowercase form. +func normalizeMode(value, fallback string) string { + value = strings.ToLower(strings.ReplaceAll(strings.TrimSpace(value), "_", "")) + if value == "" { + return fallback + } + return value +} + +func normalizeProperties(properties map[string]string) map[string]string { + if properties == nil { + return map[string]string{} + } + return properties +} diff --git a/weed/s3api/lance/handlers_table.go b/weed/s3api/lance/handlers_table.go new file mode 100644 index 000000000..909d6f05f --- /dev/null +++ b/weed/s3api/lance/handlers_table.go @@ -0,0 +1,566 @@ +package lance + +import ( + "net/http" + "strings" + + "github.com/seaweedfs/seaweedfs/weed/glog" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// handleListTables lists the Lance tables under a namespace. The spec asks for +// full string identifiers, not bare names, so a recursive listing stays +// unambiguous. +func (s *Server) handleListTables(w http.ResponseWriter, r *http.Request) { + id, delimiter, ok := routeIdentifier(w, r) + if !ok { + return + } + bucket, ns := id.namespace() + if bucket == "" { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "listing tables needs a namespace, not the root") + return + } + + req := &s3tables.ListTablesRequest{ + TableBucketARN: bucketARN(bucket), + Namespace: ns, + ContinuationToken: r.URL.Query().Get("page_token"), + MaxTables: pageSize(r), + } + var resp s3tables.ListTablesResponse + if err := s.execute(r, "ListTables", req, &resp); err != nil { + writeStorageError(w, r, err) + return + } + + includeDeclared := true + if raw := r.URL.Query().Get("include_declared"); raw != "" { + includeDeclared = boolQuery(r, "include_declared") + } + + writeJSON(w, http.StatusOK, ListTablesResponse{ + Tables: s.lanceTableIDs(r, bucket, resp.Tables, delimiter, includeDeclared), + PageToken: resp.ContinuationToken, + }) +} + +// handleListAllTables lists every Lance table the caller can see, across every +// table bucket. +func (s *Server) handleListAllTables(w http.ResponseWriter, r *http.Request) { + delimiter := requestDelimiter(r) + + var buckets s3tables.ListTableBucketsResponse + if err := s.execute(r, "ListTableBuckets", &s3tables.ListTableBucketsRequest{MaxBuckets: pageSize(r)}, &buckets); err != nil { + writeStorageError(w, r, err) + return + } + + // The spec marks `tables` required, so an empty catalog answers with an + // empty list rather than null. + all := []string{} + for _, bucket := range buckets.TableBuckets { + var tables s3tables.ListTablesResponse + req := &s3tables.ListTablesRequest{TableBucketARN: bucketARN(bucket.Name), MaxTables: pageSize(r)} + if err := s.execute(r, "ListTables", req, &tables); err != nil { + // One unreadable bucket must not hide the rest; a caller with access + // to some buckets still gets those. + glog.V(2).Infof("lance: skipping bucket %s in ListAllTables: %v", bucket.Name, err) + continue + } + all = append(all, s.lanceTableIDs(r, bucket.Name, tables.Tables, delimiter, true)...) + } + writeJSON(w, http.StatusOK, ListTablesResponse{Tables: all}) +} + +// lanceTableIDs keeps the Lance tables out of a listing that also carries +// Iceberg tables, and drops deregistered ones because they are meant to be +// invisible until re-registered. +func (s *Server) lanceTableIDs(r *http.Request, bucket string, summaries []s3tables.TableSummary, delimiter string, includeDeclared bool) []string { + ids := make([]string, 0, len(summaries)) + for _, summary := range summaries { + if summary.Format != s3tables.FormatLance { + continue + } + location := summary.MetadataLocation + if location == "" { + location = tableLocation(bucket, summary.Namespace, summary.Name) + } + deregistered, hasData, err := s.datasetState(r, location) + if err != nil { + glog.V(2).Infof("lance: cannot read dataset state for %s: %v", location, err) + continue + } + if deregistered { + continue + } + if !hasData && !includeDeclared { + continue + } + parts := append([]string{bucket}, summary.Namespace...) + ids = append(ids, identifier(append(parts, summary.Name)).String(delimiter)) + } + return ids +} + +// handleDeclareTable records a table that does not exist on storage yet. This +// is what a Lance client calls on CREATE TABLE, before it writes any data. +func (s *Server) handleDeclareTable(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req DeclareTableRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + + location := strings.TrimSuffix(req.Location, "/") + if location == "" { + location = tableLocation(bucket, ns, name) + } + + if err := s.createTable(r, bucket, ns, name, location); err != nil { + writeStorageError(w, r, err) + return + } + if err := s.writeMarker(r, location, reservedMarker); err != nil { + // The catalog entry is the authority; the marker only mirrors it for + // clients that read the storage prefix directly. + glog.V(1).Infof("lance: could not write %s for %s: %v", reservedMarker, location, err) + } + // Declaring a name that was deregistered brings it back, the same way + // registering it does. + if err := s.removeMarker(r, location, deregisteredMarker); err != nil { + glog.V(2).Infof("lance: could not clear %s for %s: %v", deregisteredMarker, location, err) + } + + options, err := s.storageOptions(r, bucket, location, wants(r, "vend_credentials", req.VendCredentials)) + if err != nil { + writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error()) + return + } + // Properties are not persisted for a table, so neither response carries + // them: null says "this catalog does not keep them", where echoing the + // request back or answering {} would say they were stored and are empty. + writeJSON(w, http.StatusOK, DeclareTableResponse{ + Location: location, + StorageOptions: options, + }) +} + +// handleDescribeTable resolves a table to a location, and to credentials when +// the caller asks for them. +func (s *Server) handleDescribeTable(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req DescribeTableRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if req.Version != nil || req.Tag != "" || req.Branch != "" { + writeError(w, r, http.StatusNotImplemented, codeUnsupported, + "this namespace does not resolve versions, tags or branches; the dataset owns them") + return + } + + table, err := s.loadLanceTable(r, bucket, ns, name) + if err != nil { + writeStorageError(w, r, err) + return + } + location := table.location + + deregistered, hasData, err := s.datasetState(r, location) + if err != nil { + writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error()) + return + } + if deregistered { + writeError(w, r, http.StatusNotFound, codeTableNotFound, "table is deregistered") + return + } + + options, err := s.storageOptions(r, bucket, location, wants(r, "vend_credentials", req.VendCredentials)) + if err != nil { + writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error()) + return + } + + resp := DescribeTableResponse{ + Location: location, + } + if len(options) > 0 { + resp.StorageOptions = options + } + if wants(r, "with_table_uri", req.WithTableURI) { + resp.TableURI = location + } + if wants(r, "check_declared", req.CheckDeclared) { + onlyDeclared := !hasData + resp.IsOnlyDeclared = &onlyDeclared + } + if wants(r, "load_detailed_metadata", req.LoadDetailedMetadata) { + resp.Table = name + resp.Namespace = ns + } + writeJSON(w, http.StatusOK, resp) +} + +// handleTableExists answers with the status code and no body. +func (s *Server) handleTableExists(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req TableExistsRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + + table, err := s.loadLanceTable(r, bucket, ns, name) + if err != nil { + if isNotFound(err) { + writeError(w, r, http.StatusNotFound, codeTableNotFound, "table does not exist") + return + } + writeStorageError(w, r, err) + return + } + deregistered, _, err := s.datasetState(r, table.location) + if err != nil { + writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error()) + return + } + if deregistered { + writeError(w, r, http.StatusNotFound, codeTableNotFound, "table is deregistered") + return + } + w.WriteHeader(http.StatusOK) +} + +// handleRegisterTable points a table name at an existing dataset. +func (s *Server) handleRegisterTable(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req RegisterTableRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + location := strings.TrimSuffix(req.Location, "/") + if location == "" { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "location is required") + return + } + + mode := normalizeMode(req.Mode, modeCreate) + if mode != modeCreate && mode != modeOverwrite { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "mode must be Create or Overwrite") + return + } + + existing, err := s.loadLanceTable(r, bucket, ns, name) + switch { + case err != nil && !isNotFound(err): + writeStorageError(w, r, err) + return + case err == nil: + // A deregistered table is absent as far as the spec is concerned, so + // registering over it is a re-registration rather than a conflict. + deregistered, _, stateErr := s.datasetState(r, existing.location) + if stateErr != nil { + writeError(w, r, http.StatusInternalServerError, codeInternal, stateErr.Error()) + return + } + if !deregistered && mode == modeCreate { + writeError(w, r, http.StatusConflict, codeTableAlreadyExists, "table already exists") + return + } + if existing.location != location { + // Repointing the name at another dataset is an update. Dropping and + // recreating the entry would take the old dataset's files with it, + // because the entry is the directory holding them. + if err := s.repointTable(r, bucket, ns, name, location, existing.versionToken); err != nil { + writeStorageError(w, r, err) + return + } + } + default: + if err := s.createTable(r, bucket, ns, name, location); err != nil { + writeStorageError(w, r, err) + return + } + } + + // Registering a deregistered dataset brings it back. + if err := s.removeMarker(r, location, deregisteredMarker); err != nil { + glog.V(2).Infof("lance: could not clear %s for %s: %v", deregisteredMarker, location, err) + } + + writeJSON(w, http.StatusOK, RegisterTableResponse{ + Location: location, + Properties: normalizeProperties(req.Properties), + }) +} + +// handleDeregisterTable forgets a table without touching its data. +func (s *Server) handleDeregisterTable(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req DeregisterTableRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + + table, err := s.loadLanceTable(r, bucket, ns, name) + if err != nil { + writeStorageError(w, r, err) + return + } + + // Deregistering is a state, not a deletion. Dropping the catalog entry would + // take the dataset with it, because the entry is the dataset directory and + // DeleteTable purges what it holds. The marker is what hides the table, and + // it is also what a directory-catalog client reads. + if err := s.writeMarker(r, table.location, deregisteredMarker); err != nil { + writeError(w, r, http.StatusInternalServerError, codeInternal, + "could not mark the table deregistered: "+err.Error()) + return + } + + writeJSON(w, http.StatusOK, DeregisterTableResponse{ + ID: append(append([]string{bucket}, ns...), name), + Location: table.location, + Properties: map[string]string{}, + }) +} + +// handleDropTable removes the table and its data. +func (s *Server) handleDropTable(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req DropTableRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + + table, err := s.loadLanceTable(r, bucket, ns, name) + if err != nil { + writeStorageError(w, r, err) + return + } + if err := s.dropTableEntry(r, bucket, ns, name); err != nil { + writeStorageError(w, r, err) + return + } + + writeJSON(w, http.StatusOK, DropTableResponse{ + ID: append(append([]string{bucket}, ns...), name), + Location: table.location, + Properties: map[string]string{}, + }) +} + +// handleRenameTable moves a table's catalog entry. The dataset stays put, which +// is what the storage layer already does for a renamed Iceberg table. +func (s *Server) handleRenameTable(w http.ResponseWriter, r *http.Request) { + id, _, ok := routeIdentifier(w, r) + if !ok { + return + } + var req RenameTableRequest + if err := decodeBody(r, &req); err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + if len(id) == 0 && len(req.ID) > 0 { + id = req.ID + } + if !checkBodyIdentifier(w, r, id, req.ID) { + return + } + bucket, ns, name, err := id.table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return + } + destBucket, destNS, destName, err := identifier(req.NewID).table() + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, "new_id: "+err.Error()) + return + } + if destBucket != bucket { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, + "a table cannot move between table buckets") + return + } + + if _, err := s.loadLanceTable(r, bucket, ns, name); err != nil { + writeStorageError(w, r, err) + return + } + + renameReq := &s3tables.RenameTableRequest{ + TableBucketARN: bucketARN(bucket), + SourceNamespace: ns, + SourceName: name, + DestNamespace: destNS, + DestName: destName, + } + var renameResp s3tables.RenameTableResponse + if err := s.execute(r, "RenameTable", renameReq, &renameResp); err != nil { + writeStorageError(w, r, err) + return + } + writeJSON(w, http.StatusOK, RenameTableResponse{}) +} + +// lanceTable is the catalog's record of one Lance table. +type lanceTable struct { + location string + versionToken string +} + +// loadLanceTable reads a table and refuses one that is not a Lance table, so a +// Lance client never resolves an Iceberg table's location and writes over it. +func (s *Server) loadLanceTable(r *http.Request, bucket string, ns []string, name string) (*lanceTable, error) { + req := &s3tables.GetTableRequest{ + TableBucketARN: bucketARN(bucket), + Namespace: ns, + Name: name, + } + var resp s3tables.GetTableResponse + if err := s.execute(r, "GetTable", req, &resp); err != nil { + return nil, err + } + if resp.Format != s3tables.FormatLance { + return nil, &s3tables.S3TablesError{ + Type: s3tables.ErrCodeNoSuchTable, + Message: "table " + name + " is not a lance table", + } + } + location := resp.MetadataLocation + if location == "" { + location = tableLocation(bucket, resp.Namespace, resp.Name) + } + return &lanceTable{location: location, versionToken: resp.VersionToken}, nil +} + +func (s *Server) createTable(r *http.Request, bucket string, ns []string, name, location string) error { + req := &s3tables.CreateTableRequest{ + TableBucketARN: bucketARN(bucket), + Namespace: ns, + Name: name, + Format: s3tables.FormatLance, + // A Lance table has no metadata file, so the location the catalog stores + // is the dataset root itself. + MetadataLocation: location, + } + var resp s3tables.CreateTableResponse + return s.execute(r, "CreateTable", req, &resp) +} + +// repointTable moves an existing entry to another dataset location without +// touching either dataset's files. +func (s *Server) repointTable(r *http.Request, bucket string, ns []string, name, location, versionToken string) error { + req := &s3tables.UpdateTableRequest{ + TableBucketARN: bucketARN(bucket), + Namespace: ns, + Name: name, + VersionToken: versionToken, + MetadataLocation: location, + } + var resp s3tables.UpdateTableResponse + return s.execute(r, "UpdateTable", req, &resp) +} + +// dropTableEntry removes the catalog entry and the dataset under it. Only +// DropTable wants this; deregistering and repointing must leave the files. +func (s *Server) dropTableEntry(r *http.Request, bucket string, ns []string, name string) error { + req := &s3tables.DeleteTableRequest{ + TableBucketARN: bucketARN(bucket), + Namespace: ns, + Name: name, + } + return s.execute(r, "DeleteTable", req, nil) +} diff --git a/weed/s3api/lance/handlers_test.go b/weed/s3api/lance/handlers_test.go new file mode 100644 index 000000000..8402770ce --- /dev/null +++ b/weed/s3api/lance/handlers_test.go @@ -0,0 +1,359 @@ +package lance + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/gorilla/mux" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3err" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" +) + +// openAuthenticator stands in for a gateway with no IAM configured, which is the +// mode the namespace falls open in. +type openAuthenticator struct{} + +func (openAuthenticator) AuthenticateRequest(*http.Request) (string, interface{}, s3err.ErrorCode) { + return s3tables.DefaultAccountID, nil, s3err.ErrNone +} + +func (openAuthenticator) DefaultAllow() bool { return true } + +type testHarness struct { + router *mux.Router + filer *s3tablestest.MemFiler + admin *s3tables.Manager + server *Server +} + +func newTestHarness(t *testing.T) *testHarness { + t.Helper() + filer := s3tablestest.Start(t) + server := NewServer(s3tables.NewManagerClient(filer.Client), openAuthenticator{}) + server.SetS3Endpoint("http://127.0.0.1:8333") + server.SetS3Region(s3tables.DefaultRegion) + router := mux.NewRouter().SkipClean(true) + server.RegisterRoutes(router) + + // Table buckets are created by an operator, not by a namespace client, so + // the harness seeds them the way the shell and admin console would. + admin := s3tables.NewManager() + admin.SetTrusted(true) + return &testHarness{router: router, filer: filer, admin: admin, server: server} +} + +// createBucket seeds a table bucket the Lance namespace can then be pointed at. +// An empty format strips the declaration afterwards, which is the only way to +// get the shape a bucket made before formats existed has: one that still +// accepts either format. +func (h *testHarness) createBucket(t *testing.T, name, format string) { + t.Helper() + var resp s3tables.CreateTableBucketResponse + declared := format + if declared == "" { + declared = s3tables.FormatIceberg + } + err := h.admin.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTableBucket", + &s3tables.CreateTableBucketRequest{Name: name, Format: declared}, &resp, "") + if err != nil { + t.Fatalf("create table bucket %s: %v", name, err) + } + if format == "" { + h.undeclareBucket(t, name) + } +} + +// undeclareBucket removes a bucket's format from its metadata, ageing it back to +// what the filer holds for a bucket created before the field existed. +func (h *testHarness) undeclareBucket(t *testing.T, name string) { + t.Helper() + entry := h.filer.Get(s3tables.TablesPath, name) + if entry == nil { + t.Fatalf("table bucket %s is not in the filer", name) + } + var metadata map[string]any + if err := json.Unmarshal(entry.Extended[s3tables.ExtendedKeyMetadata], &metadata); err != nil { + t.Fatalf("read bucket metadata: %v", err) + } + delete(metadata, "format") + updated, err := json.Marshal(metadata) + if err != nil { + t.Fatalf("write bucket metadata: %v", err) + } + extended := map[string][]byte{} + for key, value := range entry.Extended { + extended[key] = value + } + extended[s3tables.ExtendedKeyMetadata] = updated + h.filer.Put(s3tables.TablesPath, name, extended) +} + +func (h *testHarness) bucketARN(t *testing.T, name string) string { + t.Helper() + arn, err := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3tables.DefaultAccountID, name) + if err != nil { + t.Fatalf("build arn: %v", err) + } + return arn +} + +func (h *testHarness) do(t *testing.T, method, target, body string) *httptest.ResponseRecorder { + t.Helper() + var reader *strings.Reader + if body == "" { + reader = strings.NewReader("") + } else { + reader = strings.NewReader(body) + } + req := httptest.NewRequest(method, target, reader) + recorder := httptest.NewRecorder() + h.router.ServeHTTP(recorder, req) + return recorder +} + +func (h *testHarness) mustDo(t *testing.T, method, target, body string, want int) *httptest.ResponseRecorder { + t.Helper() + recorder := h.do(t, method, target, body) + if recorder.Code != want { + t.Fatalf("%s %s = %d (%s), want %d", method, target, recorder.Code, recorder.Body.String(), want) + } + return recorder +} + +func decode[T any](t *testing.T, recorder *httptest.ResponseRecorder) T { + t.Helper() + var out T + if err := json.Unmarshal(recorder.Body.Bytes(), &out); err != nil { + t.Fatalf("decode %s: %v", recorder.Body.String(), err) + } + return out +} + +// The lifecycle a Lance client drives: create the namespace, declare the table +// before any data exists, resolve it to a location, then deregister and bring it +// back by registering the same location. +func TestTableLifecycle(t *testing.T) { + h := newTestHarness(t) + + h.createBucket(t, "analytics", s3tables.FormatLance) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK) + + declared := decode[DeclareTableResponse](t, + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK)) + if declared.Location != "s3://analytics/sales/orders" { + t.Fatalf("declared location = %q", declared.Location) + } + // The endpoint is plaintext, so a client that does not get allow_http fails + // with what looks like a credential error. + if declared.StorageOptions["aws_endpoint"] != "http://127.0.0.1:8333" || + declared.StorageOptions["allow_http"] != "true" { + t.Fatalf("storage options = %v", declared.StorageOptions) + } + + described := decode[DescribeTableResponse](t, + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe?check_declared=true", `{}`, http.StatusOK)) + if described.Location != declared.Location { + t.Fatalf("described location = %q, want %q", described.Location, declared.Location) + } + if described.IsOnlyDeclared == nil || !*described.IsOnlyDeclared { + t.Fatalf("is_only_declared = %v, want true for a table with no data", described.IsOnlyDeclared) + } + + listed := decode[ListTablesResponse](t, + h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK)) + if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$orders" { + t.Fatalf("listed tables = %v, want the full identifier", listed.Tables) + } + + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK) + + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/deregister", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusNotFound) + afterDrop := decode[ListTablesResponse](t, + h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK)) + if len(afterDrop.Tables) != 0 { + t.Fatalf("deregistered table still listed: %v", afterDrop.Tables) + } + // Deregistering preserves the data. The catalog entry is the dataset + // directory, so dropping it would take the dataset with it. + if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil { + t.Fatal("deregister deleted the dataset directory") + } + + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register", + `{"location":"s3://analytics/sales/orders"}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK) + + // Dropping is the operation that does remove the data. + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/drop", `{}`, http.StatusOK) + if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") != nil { + t.Fatal("drop left the dataset directory behind") + } +} + +// Repointing a registered name at another dataset must not take the dataset it +// used to name with it. +func TestRegisterOverwriteKeepsTheOldDataset(t *testing.T) { + h := newTestHarness(t) + h.createBucket(t, "analytics", s3tables.FormatLance) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$archive/declare", `{}`, http.StatusOK) + + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register", + `{"location":"s3://analytics/sales/archive","mode":"Overwrite"}`, http.StatusOK) + + described := decode[DescribeTableResponse](t, + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe", `{}`, http.StatusOK)) + if described.Location != "s3://analytics/sales/archive" { + t.Fatalf("location after repointing = %q", described.Location) + } + if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil { + t.Fatal("repointing deleted the dataset the name used to hold") + } +} + +// A required list field answers empty rather than null, which a generated +// client may decode differently. +func TestListAllTablesIsNeverNull(t *testing.T) { + h := newTestHarness(t) + h.createBucket(t, "analytics", s3tables.FormatLance) + + body := h.mustDo(t, http.MethodGet, "/v1/table", "", http.StatusOK).Body.String() + if strings.Contains(body, `"tables":null`) { + t.Fatalf("ListAllTables returned null for a required field: %s", body) + } +} + +func TestNamespaceListing(t *testing.T) { + h := newTestHarness(t) + h.createBucket(t, "analytics", s3tables.FormatLance) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$finance/create", `{}`, http.StatusOK) + + roots := decode[ListNamespacesResponse](t, + h.mustDo(t, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK)) + if len(roots.Namespaces) != 1 || roots.Namespaces[0] != "analytics" { + t.Fatalf("root listing = %v, want the table buckets", roots.Namespaces) + } + + children := decode[ListNamespacesResponse](t, + h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK)) + if len(children.Namespaces) != 2 { + t.Fatalf("bucket listing = %v, want two namespaces", children.Namespaces) + } + + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/exists", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/exists", `{}`, http.StatusNotFound) + + // Fail mode reports a missing namespace as 400 on this operation, which is + // what the spec asks for; Skip reports success. + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{}`, http.StatusBadRequest) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{"mode":"Skip"}`, http.StatusNoContent) +} + +// Storage flattens a namespace's parts, so creating a$b without a would leave +// an intermediate that listing derives from the name and describe denies +// exists. The spec asks for NamespaceNotFound, which keeps them consistent. +func TestCreateNamespaceRequiresItsParent(t *testing.T) { + h := newTestHarness(t) + h.createBucket(t, "analytics", s3tables.FormatLance) + + recorder := h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$missing$child/create", `{}`, http.StatusNotFound) + if got := decode[errorResponse](t, recorder); got.Code != codeNamespaceNotFound { + t.Fatalf("error code = %d, want %d", got.Code, codeNamespaceNotFound) + } + + // With the parent in place the child is fine, and both are then listed and + // describable. + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent/create", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent$child/create", `{}`, http.StatusOK) + + listed := decode[ListNamespacesResponse](t, + h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK)) + for _, name := range listed.Namespaces { + path := "/v1/namespace/analytics$" + name + h.mustDo(t, http.MethodPost, path+"/exists", `{}`, http.StatusOK) + } +} + +func TestCreateNamespaceModes(t *testing.T) { + h := newTestHarness(t) + h.createBucket(t, "analytics", s3tables.FormatLance) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK) + + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusConflict) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"ExistOk"}`, http.StatusOK) + // snake_case is the other spelling the spec accepts. + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"exist_ok"}`, http.StatusOK) + + h.mustDo(t, http.MethodPost, "/v1/namespace/$/create", `{}`, http.StatusBadRequest) +} + +// A Lance client must never resolve an Iceberg table's location, or it would +// write a dataset over a table another engine owns. +func TestIcebergTablesAreInvisible(t *testing.T) { + h := newTestHarness(t) + // Undeclared, because a bucket that declares one format cannot hold the + // other - and mixing is exactly what this test needs to prove is hidden. + h.createBucket(t, "analytics", "") + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK) + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$vectors/declare", `{}`, http.StatusOK) + + manager := h.admin + var created s3tables.CreateTableResponse + err := manager.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTable", &s3tables.CreateTableRequest{ + TableBucketARN: h.bucketARN(t, "analytics"), + Namespace: []string{"sales"}, + Name: "ledger", + Format: s3tables.FormatIceberg, + MetadataLocation: "s3://analytics/sales/ledger/metadata/v1.metadata.json", + }, &created, "") + if err != nil { + t.Fatalf("create iceberg table: %v", err) + } + + listed := decode[ListTablesResponse](t, + h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK)) + if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$vectors" { + t.Fatalf("listing = %v, want only the lance table", listed.Tables) + } + + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/describe", `{}`, http.StatusNotFound) + h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/exists", `{}`, http.StatusNotFound) + + // Declaring over the Iceberg table must not quietly succeed and hand the + // Lance client a directory another format owns. + recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/declare", `{}`, http.StatusConflict) + if got := decode[errorResponse](t, recorder); got.Code != codeTableAlreadyExists { + t.Fatalf("error code = %d, want %d", got.Code, codeTableAlreadyExists) + } +} + +// The route and the body naming different objects is a bad request, not a silent +// preference for one of them. +func TestRouteAndBodyMustAgree(t *testing.T) { + h := newTestHarness(t) + h.createBucket(t, "analytics", s3tables.FormatLance) + h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK) + + recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", + `{"id":["analytics","sales","other"]}`, http.StatusBadRequest) + if got := decode[errorResponse](t, recorder); got.Code != codeInvalidInput { + t.Fatalf("error code = %d, want %d", got.Code, codeInvalidInput) + } +} + +// The data plane needs Lance format support that does not exist in Go, so it +// answers with the spec's own Unsupported code rather than a bare 404. +func TestDataPlaneIsUnsupported(t *testing.T) { + h := newTestHarness(t) + recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/query", `{}`, http.StatusNotImplemented) + if got := decode[errorResponse](t, recorder); got.Code != codeUnsupported { + t.Fatalf("error code = %d, want %d", got.Code, codeUnsupported) + } +} diff --git a/weed/s3api/lance/identifier.go b/weed/s3api/lance/identifier.go new file mode 100644 index 000000000..dfe8aee2a --- /dev/null +++ b/weed/s3api/lance/identifier.go @@ -0,0 +1,84 @@ +package lance + +import ( + "fmt" + "net/http" + "strings" +) + +// defaultDelimiter joins the parts of a Lance string identifier when the caller +// does not pass ?delimiter=. An id equal to the delimiter is the root namespace, +// so /v1/namespace/$/list lists the root's children. +const defaultDelimiter = "$" + +// identifier is a Lance object identifier: zero parts is the root namespace, one +// part names a table bucket, and the rest are namespace parts with a table name +// last. The mapping onto storage is bucket / namespace / table, which is the +// three-level shape Lance clients already use. +type identifier []string + +func requestDelimiter(r *http.Request) string { + if d := r.URL.Query().Get("delimiter"); d != "" { + return d + } + return defaultDelimiter +} + +// parseIdentifier decodes the {id} route variable. Empty parts are rejected +// rather than dropped so that "a$$b" cannot silently resolve to "a$b". +func parseIdentifier(encoded, delimiter string) (identifier, error) { + if encoded == "" || encoded == delimiter { + return nil, nil + } + parts := strings.Split(encoded, delimiter) + for _, part := range parts { + if part == "" { + return nil, fmt.Errorf("identifier %q has an empty part", encoded) + } + } + return parts, nil +} + +func (id identifier) String(delimiter string) string { + if len(id) == 0 { + return delimiter + } + return strings.Join(id, delimiter) +} + +// namespace splits a namespace identifier into the table bucket and the parts of +// the namespace inside it. The root and a bare bucket both return an empty +// namespace, so callers check the bucket to tell them apart. +func (id identifier) namespace() (bucket string, ns []string) { + if len(id) == 0 { + return "", nil + } + return id[0], id[1:] +} + +// table splits a table identifier. A table needs a bucket, at least one +// namespace part and a name, because storage has no unnamespaced tables. +func (id identifier) table() (bucket string, ns []string, name string, err error) { + if len(id) < 3 { + return "", nil, "", fmt.Errorf("table identifier needs a bucket, a namespace and a name") + } + return id[0], id[1 : len(id)-1], id[len(id)-1], nil +} + +// matchesBody reports whether an identifier carried in the request body agrees +// with the one in the route. The spec requires 400 when both are present and +// differ, and requires the route to win when the body omits it. +func (id identifier) matchesBody(body []string) bool { + if len(body) == 0 { + return true + } + if len(body) != len(id) { + return false + } + for i := range body { + if body[i] != id[i] { + return false + } + } + return true +} diff --git a/weed/s3api/lance/identifier_test.go b/weed/s3api/lance/identifier_test.go new file mode 100644 index 000000000..28bbf61aa --- /dev/null +++ b/weed/s3api/lance/identifier_test.go @@ -0,0 +1,104 @@ +package lance + +import "testing" + +func TestParseIdentifier(t *testing.T) { + cases := []struct { + name string + encoded string + delimiter string + want []string + wantErr bool + }{ + {name: "root is the delimiter", encoded: "$", delimiter: "$"}, + {name: "empty is the root", encoded: "", delimiter: "$"}, + {name: "bucket", encoded: "analytics", delimiter: "$", want: []string{"analytics"}}, + {name: "namespace", encoded: "analytics$sales", delimiter: "$", want: []string{"analytics", "sales"}}, + {name: "table", encoded: "analytics$sales$orders", delimiter: "$", want: []string{"analytics", "sales", "orders"}}, + {name: "custom delimiter", encoded: "a.b.c", delimiter: ".", want: []string{"a", "b", "c"}}, + {name: "empty part is rejected", encoded: "a$$b", delimiter: "$", wantErr: true}, + {name: "trailing delimiter is rejected", encoded: "a$", delimiter: "$", wantErr: true}, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got, err := parseIdentifier(c.encoded, c.delimiter) + if c.wantErr { + if err == nil { + t.Fatalf("parseIdentifier(%q) error = nil, want an error", c.encoded) + } + return + } + if err != nil { + t.Fatalf("parseIdentifier(%q) error = %v", c.encoded, err) + } + if len(got) != len(c.want) { + t.Fatalf("parseIdentifier(%q) = %v, want %v", c.encoded, got, c.want) + } + for i := range got { + if got[i] != c.want[i] { + t.Fatalf("parseIdentifier(%q) = %v, want %v", c.encoded, got, c.want) + } + } + }) + } +} + +func TestIdentifierNamespace(t *testing.T) { + bucket, ns := identifier(nil).namespace() + if bucket != "" || len(ns) != 0 { + t.Fatalf("root namespace() = %q %v, want empty", bucket, ns) + } + + bucket, ns = identifier{"analytics"}.namespace() + if bucket != "analytics" || len(ns) != 0 { + t.Fatalf("bucket namespace() = %q %v", bucket, ns) + } + + bucket, ns = identifier{"analytics", "sales", "eu"}.namespace() + if bucket != "analytics" || len(ns) != 2 || ns[0] != "sales" || ns[1] != "eu" { + t.Fatalf("nested namespace() = %q %v", bucket, ns) + } +} + +// A table always needs a bucket, a namespace and a name: storage has no +// unnamespaced tables, so a two-part identifier is a client error rather than a +// table at the top of a bucket. +func TestIdentifierTable(t *testing.T) { + if _, _, _, err := (identifier{"analytics", "orders"}).table(); err == nil { + t.Fatal("table() on a two-part identifier error = nil, want an error") + } + + bucket, ns, name, err := identifier{"analytics", "sales", "eu", "orders"}.table() + if err != nil { + t.Fatalf("table() error = %v", err) + } + if bucket != "analytics" || name != "orders" || len(ns) != 2 || ns[0] != "sales" || ns[1] != "eu" { + t.Fatalf("table() = %q %v %q", bucket, ns, name) + } +} + +func TestIdentifierMatchesBody(t *testing.T) { + id := identifier{"a", "b", "c"} + if !id.matchesBody(nil) { + t.Fatal("an absent body identifier must defer to the route") + } + if !id.matchesBody([]string{"a", "b", "c"}) { + t.Fatal("an equal body identifier must match") + } + if id.matchesBody([]string{"a", "b"}) { + t.Fatal("a shorter body identifier must not match") + } + if id.matchesBody([]string{"a", "b", "d"}) { + t.Fatal("a different body identifier must not match") + } +} + +func TestIdentifierString(t *testing.T) { + if got := identifier(nil).String("$"); got != "$" { + t.Fatalf("root String() = %q, want %q", got, "$") + } + if got := (identifier{"a", "b"}).String("$"); got != "a$b" { + t.Fatalf("String() = %q, want %q", got, "a$b") + } +} diff --git a/weed/s3api/lance/markers.go b/weed/s3api/lance/markers.go new file mode 100644 index 000000000..9b1633602 --- /dev/null +++ b/weed/s3api/lance/markers.go @@ -0,0 +1,93 @@ +package lance + +import ( + "net/http" + "time" + + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// Marker files the Lance Directory Catalog reads. Writing them beside the +// dataset keeps a client that bypasses this namespace, and lists the storage +// prefix directly, seeing the same table states the catalog reports. +const ( + reservedMarker = ".lance-reserved" + deregisteredMarker = ".lance-deregistered" + versionsDir = "_versions" +) + +// datasetDir maps a table location onto the filer directory holding it. +func datasetDir(location string) string { + return s3tables.TableDataDirFromMetadataLocation(location) +} + +func (s *Server) entryExists(r *http.Request, dir, name string) (bool, error) { + found := false + err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + _, lookupErr := filer_pb.LookupEntry(r.Context(), client, &filer_pb.LookupDirectoryEntryRequest{ + Directory: dir, + Name: name, + }) + if lookupErr == filer_pb.ErrNotFound { + return nil + } + if lookupErr != nil { + return lookupErr + } + found = true + return nil + }) + return found, err +} + +// writeMarker drops a zero-length marker into the dataset directory. The +// directory has to exist already: a table declared through this namespace has +// one, and a table registered at a location that does not yet exist has no +// storage to mark. +func (s *Server) writeMarker(r *http.Request, location, name string) error { + dir := datasetDir(location) + if dir == "" { + return nil + } + now := time.Now().Unix() + return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + return filer_pb.CreateEntry(r.Context(), client, &filer_pb.CreateEntryRequest{ + Directory: dir, + Entry: &filer_pb.Entry{ + Name: name, + Attributes: &filer_pb.FuseAttributes{ + Mtime: now, + Crtime: now, + FileMode: uint32(0644), + }, + }, + }) + }) +} + +func (s *Server) removeMarker(r *http.Request, location, name string) error { + dir := datasetDir(location) + if dir == "" { + return nil + } + return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + return filer_pb.DoRemove(r.Context(), client, dir, name, true, false, true, false, nil) + }) +} + +// datasetState reports the two things the spec asks about a table's storage: +// whether it has been deregistered, and whether it holds data yet. A dataset +// holds data once it has a version manifest, which is how a directory catalog +// decides the same question. +func (s *Server) datasetState(r *http.Request, location string) (deregistered, hasData bool, err error) { + dir := datasetDir(location) + if dir == "" { + return false, false, nil + } + if deregistered, err = s.entryExists(r, dir, deregisteredMarker); err != nil { + return false, false, err + } + hasData, err = s.entryExists(r, dir, versionsDir) + return deregistered, hasData, err +} diff --git a/weed/s3api/lance/server.go b/weed/s3api/lance/server.go new file mode 100644 index 000000000..c94121b82 --- /dev/null +++ b/weed/s3api/lance/server.go @@ -0,0 +1,188 @@ +package lance + +import ( + "context" + "net/http" + "time" + + "github.com/gorilla/mux" + "github.com/seaweedfs/seaweedfs/weed/glog" + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3err" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// FilerClient provides access to the filer for storage operations. +type FilerClient interface { + WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error +} + +type S3Authenticator interface { + AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode) + DefaultAllow() bool +} + +// VendedCredentials are short-lived S3 credentials scoped to one table. +type VendedCredentials struct { + AccessKeyID string + SecretAccessKey string + SessionToken string + Expiration time.Time +} + +// CredentialVendor mints credentials limited to a single table's prefix for a +// caller the catalog has already authenticated and authorized. A nil result with +// no error means the deployment has vending switched off. +type CredentialVendor interface { + VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*VendedCredentials, error) +} + +// Server implements the Lance Namespace REST spec. +type Server struct { + filerClient FilerClient + tablesManager *s3tables.Manager + authenticator S3Authenticator + credentialVendor CredentialVendor + s3Endpoint string + s3Region string +} + +// NewServer creates a Lance namespace server over the given filer. +func NewServer(filerClient FilerClient, authenticator S3Authenticator) *Server { + manager := s3tables.NewManager() + // Mirror the S3 port: fall open by default only when the gateway itself is + // open, so an authenticated caller still passes the normal permission check. + if authenticator != nil { + manager.SetDefaultAllow(authenticator.DefaultAllow()) + } + return &Server{ + filerClient: filerClient, + tablesManager: manager, + authenticator: authenticator, + } +} + +// SetCredentialVendor enables storage_options credential vending for clients +// that ask for it with vend_credentials. +func (s *Server) SetCredentialVendor(vendor CredentialVendor) { + s.credentialVendor = vendor +} + +// SetS3Endpoint configures the S3 endpoint advertised in storage_options so a +// client can reach the dataset without separately discovering the S3 address. +func (s *Server) SetS3Endpoint(endpoint string) { + s.s3Endpoint = endpoint +} + +// SetS3Region configures the region advertised in storage_options. +func (s *Server) SetS3Region(region string) { + s.s3Region = region +} + +// RegisterRoutes registers the Lance Namespace REST routes. +// +// The spec puts the identifier in the path rather than the body so a reverse +// proxy can route and authorize without deserializing the request. +func (s *Server) RegisterRoutes(router *mux.Router) { + router.Use(loggingMiddleware) + + router.HandleFunc("/v1/namespace/{id}/create", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost) + router.HandleFunc("/v1/namespace/{id}/list", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet) + router.HandleFunc("/v1/namespace/{id}/describe", s.Auth(s.handleDescribeNamespace)).Methods(http.MethodPost) + router.HandleFunc("/v1/namespace/{id}/drop", s.Auth(s.handleDropNamespace)).Methods(http.MethodPost) + router.HandleFunc("/v1/namespace/{id}/exists", s.Auth(s.handleNamespaceExists)).Methods(http.MethodPost) + router.HandleFunc("/v1/namespace/{id}/table/list", s.Auth(s.handleListTables)).Methods(http.MethodGet) + + router.HandleFunc("/v1/table", s.Auth(s.handleListAllTables)).Methods(http.MethodGet) + router.HandleFunc("/v1/table/{id}/declare", s.Auth(s.handleDeclareTable)).Methods(http.MethodPost) + router.HandleFunc("/v1/table/{id}/describe", s.Auth(s.handleDescribeTable)).Methods(http.MethodPost) + router.HandleFunc("/v1/table/{id}/exists", s.Auth(s.handleTableExists)).Methods(http.MethodPost) + router.HandleFunc("/v1/table/{id}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost) + router.HandleFunc("/v1/table/{id}/deregister", s.Auth(s.handleDeregisterTable)).Methods(http.MethodPost) + router.HandleFunc("/v1/table/{id}/drop", s.Auth(s.handleDropTable)).Methods(http.MethodPost) + router.HandleFunc("/v1/table/{id}/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost) + + // The data plane needs Lance format support that does not exist in Go. Say + // so with the spec's own code instead of returning a bare 404. + for _, action := range []string{"create", "insert", "merge_insert", "update", "delete", + "query", "count_rows", "explain_plan", "analyze_plan", "restore", + "add_columns", "alter_columns", "drop_columns", "backfill_column", + "create_index", "create_scalar_index", "stats", "schema_metadata/update"} { + router.HandleFunc("/v1/table/{id}/"+action, s.Auth(s.handleUnsupported)).Methods(http.MethodPost) + } + // Version ops exist in the spec for stores that cannot order commits + // themselves. Ours can: a Lance commit is a put-if-not-exists, and this S3 + // evaluates that precondition at the object's owner filer under a per-path + // lock, so the dataset keeps its own version history and the catalog stays + // out of the commit path. + for _, action := range []string{ + "version/create", "version/list", "version/describe", "version/delete", + "index/list", "tags/list", "tags/version", "tags/create", "tags/delete", "tags/update", + "branches/list", "branches/create", "branches/delete"} { + router.HandleFunc("/v1/table/{id}/"+action, s.Auth(s.handleUnsupported)).Methods(http.MethodPost) + } + + router.PathPrefix("/").HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + glog.V(2).Infof("lance: no route for %s %s", r.Method, r.RequestURI) + writeError(w, r, http.StatusNotFound, codeUnsupported, "no such operation") + }) + + glog.V(2).Infof("Registered Lance Namespace routes") +} + +func (s *Server) handleUnsupported(w http.ResponseWriter, r *http.Request) { + writeError(w, r, http.StatusNotImplemented, codeUnsupported, + "this namespace records table metadata only; run data operations through a Lance client against the table location") +} + +func loggingMiddleware(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + glog.V(2).Infof("lance request: %s %s from %s", r.Method, r.RequestURI, r.RemoteAddr) + next.ServeHTTP(w, r) + }) +} + +// Auth authenticates the caller and puts the identity in the request context. +// The Lance spec maps identity onto the same headers the S3 authenticator +// already understands, so SigV4 and bearer tokens both keep working. +func (s *Server) Auth(handler http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if s.authenticator == nil { + writeError(w, r, http.StatusUnauthorized, codeUnauthenticated, "authentication required") + return + } + + identityName, identity, errCode := s.authenticator.AuthenticateRequest(r) + if errCode != s3err.ErrNone { + if !s.authenticator.DefaultAllow() { + apiErr := s3err.GetAPIError(errCode) + code := codeInternal + switch apiErr.HTTPStatusCode { + case http.StatusForbidden: + code = codePermissionDenied + case http.StatusUnauthorized: + code = codeUnauthenticated + case http.StatusBadRequest: + code = codeInvalidInput + } + writeError(w, r, apiErr.HTTPStatusCode, code, apiErr.Description) + return + } + glog.V(2).Infof("lance: authentication failed (%v) but the gateway is open, proceeding", errCode) + } + + if identityName != "" || identity != nil { + ctx := r.Context() + if identityName != "" { + ctx = s3_constants.SetIdentityNameInContext(ctx, identityName) + } + if identity != nil { + ctx = s3_constants.SetIdentityInContext(ctx, identity) + } + r = r.WithContext(ctx) + } + + handler(w, r) + } +} diff --git a/weed/s3api/lance/storage.go b/weed/s3api/lance/storage.go new file mode 100644 index 000000000..b831913a6 --- /dev/null +++ b/weed/s3api/lance/storage.go @@ -0,0 +1,158 @@ +package lance + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "strconv" + "strings" + + "github.com/gorilla/mux" + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +const defaultPageSize = 1000 + +// execute runs one S3 Tables operation as the request's authenticated caller. +func (s *Server) execute(r *http.Request, operation string, req, resp interface{}) error { + identityName := s3_constants.GetIdentityNameFromContext(r) + return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + return s.tablesManager.Execute(r.Context(), s3tables.NewManagerClient(client), operation, req, resp, identityName) + }) +} + +func bucketARN(bucket string) string { + arn, _ := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3_constants.AccountAdminId, bucket) + return arn +} + +// maxRequestBody bounds what one call can make the catalog hold. Every request +// this surface takes is a small JSON envelope; the largest carries a set of +// properties, not data. +const maxRequestBody = 4 << 20 + +// decodeBody reads an optional JSON request body. Every Lance operation carries +// one, but the fields that matter are also in the route, so an empty body is +// not an error. +func decodeBody(r *http.Request, into interface{}) error { + body, err := io.ReadAll(io.LimitReader(r.Body, maxRequestBody+1)) + if err != nil { + return fmt.Errorf("read request body: %w", err) + } + if len(body) > maxRequestBody { + return fmt.Errorf("request body is larger than %d bytes", maxRequestBody) + } + if len(strings.TrimSpace(string(body))) == 0 { + return nil + } + if err := json.Unmarshal(body, into); err != nil { + return fmt.Errorf("invalid request body: %w", err) + } + return nil +} + +// routeIdentifier parses the {id} route variable, writing the error response +// itself when the identifier is malformed. +func routeIdentifier(w http.ResponseWriter, r *http.Request) (identifier, string, bool) { + delimiter := requestDelimiter(r) + id, err := parseIdentifier(mux.Vars(r)["id"], delimiter) + if err != nil { + writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error()) + return nil, delimiter, false + } + return id, delimiter, true +} + +// checkBodyIdentifier enforces the spec rule that a route and a body naming +// different objects is a bad request rather than a silent preference. +func checkBodyIdentifier(w http.ResponseWriter, r *http.Request, id identifier, body []string) bool { + if id.matchesBody(body) { + return true + } + writeError(w, r, http.StatusBadRequest, codeInvalidInput, + "the identifier in the request body does not match the one in the route") + return false +} + +func pageSize(r *http.Request) int { + if raw := r.URL.Query().Get("limit"); raw != "" { + if parsed, err := strconv.Atoi(raw); err == nil && parsed > 0 { + return parsed + } + } + return defaultPageSize +} + +func boolQuery(r *http.Request, name string) bool { + value, err := strconv.ParseBool(r.URL.Query().Get(name)) + return err == nil && value +} + +// wants resolves a tri-state request flag: the query parameter the REST spec +// adds, else the body field, else the implementation's own choice. +func wants(r *http.Request, name string, body *bool) bool { + if boolQuery(r, name) { + return true + } + return body != nil && *body +} + +// tableLocation is where a table's dataset lives when the caller does not name +// a location. It mirrors the Iceberg catalog's layout so both catalogs put a +// table of the same name in the same place. +func tableLocation(bucket string, ns []string, name string) string { + return fmt.Sprintf("s3://%s/%s/%s", bucket, strings.Join(ns, "."), name) +} + +// storageOptions builds the object_store settings a Lance client needs to reach +// the dataset. The key names are the aws_-prefixed forms Lance clients pass +// through to object_store. +func (s *Server) storageOptions(r *http.Request, bucket, location string, vend bool) (map[string]string, error) { + options := map[string]string{} + if s.s3Endpoint != "" { + options["aws_endpoint"] = s.s3Endpoint + if strings.HasPrefix(s.s3Endpoint, "http://") { + // object_store refuses a plaintext endpoint unless told to allow it, + // and the resulting failure reads like a credential problem. + options["allow_http"] = "true" + } + } + if s.s3Region != "" { + options["aws_region"] = s.s3Region + } + + if !vend || s.credentialVendor == nil { + return options, nil + } + + principal := s3_constants.GetIdentityNameFromContext(r) + credentials, err := s.credentialVendor.VendTableCredentials(r.Context(), principal, bucket, locationPrefix(location)) + if err != nil { + return nil, err + } + if credentials == nil { + return options, nil + } + options["aws_access_key_id"] = credentials.AccessKeyID + options["aws_secret_access_key"] = credentials.SecretAccessKey + if credentials.SessionToken != "" { + options["aws_session_token"] = credentials.SessionToken + } + if !credentials.Expiration.IsZero() { + options["expires_at_millis"] = strconv.FormatInt(credentials.Expiration.UnixMilli(), 10) + } + return options, nil +} + +// locationPrefix strips the s3://bucket/ part of a location, leaving the key +// prefix a credential is scoped to. +func locationPrefix(location string) string { + trimmed := strings.TrimPrefix(location, "s3://") + if _, prefix, found := strings.Cut(trimmed, "/"); found { + return prefix + } + return "" +} diff --git a/weed/s3api/lance/types.go b/weed/s3api/lance/types.go new file mode 100644 index 000000000..0f5ababb3 --- /dev/null +++ b/weed/s3api/lance/types.go @@ -0,0 +1,149 @@ +// Package lance serves the Lance Namespace REST spec over the same table +// buckets the Iceberg REST catalog uses. A Lance table is a catalog entry with +// format LANCE: the namespace records where the dataset lives and vends +// credentials for it, and the Lance client owns everything under that location. +package lance + +// Modes shared by create and register. The spec matches them case-insensitively +// and accepts both PascalCase and snake_case. +const ( + modeCreate = "create" + modeExistOk = "existok" + modeOverwrite = "overwrite" + modeFail = "fail" + modeSkip = "skip" + + behaviorRestrict = "restrict" + behaviorCascade = "cascade" +) + +type CreateNamespaceRequest struct { + ID []string `json:"id,omitempty"` + Mode string `json:"mode,omitempty"` + Properties map[string]string `json:"properties,omitempty"` +} + +type CreateNamespaceResponse struct { + Properties map[string]string `json:"properties"` +} + +type ListNamespacesResponse struct { + Namespaces []string `json:"namespaces"` + PageToken string `json:"page_token,omitempty"` +} + +type DescribeNamespaceRequest struct { + ID []string `json:"id,omitempty"` +} + +type DescribeNamespaceResponse struct { + Properties map[string]string `json:"properties"` +} + +type DropNamespaceRequest struct { + ID []string `json:"id,omitempty"` + Mode string `json:"mode,omitempty"` + Behavior string `json:"behavior,omitempty"` +} + +type DropNamespaceResponse struct { + Properties map[string]string `json:"properties,omitempty"` +} + +type NamespaceExistsRequest struct { + ID []string `json:"id,omitempty"` +} + +type ListTablesResponse struct { + Tables []string `json:"tables"` + PageToken string `json:"page_token,omitempty"` +} + +type DeclareTableRequest struct { + ID []string `json:"id,omitempty"` + Location string `json:"location,omitempty"` + VendCredentials *bool `json:"vend_credentials,omitempty"` + Properties map[string]string `json:"properties,omitempty"` +} + +type DeclareTableResponse struct { + Location string `json:"location"` + StorageOptions map[string]string `json:"storage_options,omitempty"` + // Properties are null rather than {}: the catalog does not keep a table's + // properties, and {} would claim it kept them and found none. + Properties map[string]string `json:"properties"` + // ManagedVersioning stays false: the dataset owns its version history, + // because this store can order commits without the catalog in the path. + ManagedVersioning bool `json:"managed_versioning"` +} + +type DescribeTableRequest struct { + ID []string `json:"id,omitempty"` + Version *int64 `json:"version,omitempty"` + Tag string `json:"tag,omitempty"` + Branch string `json:"branch,omitempty"` + // The REST spec carries these as query parameters, but clients also put + // them in the body, so honour both. + WithTableURI *bool `json:"with_table_uri,omitempty"` + LoadDetailedMetadata *bool `json:"load_detailed_metadata,omitempty"` + CheckDeclared *bool `json:"check_declared,omitempty"` + VendCredentials *bool `json:"vend_credentials,omitempty"` +} + +type DescribeTableResponse struct { + Table string `json:"table,omitempty"` + Namespace []string `json:"namespace,omitempty"` + Version *int64 `json:"version,omitempty"` + Location string `json:"location"` + TableURI string `json:"table_uri,omitempty"` + StorageOptions map[string]string `json:"storage_options,omitempty"` + // Null, not {}: this catalog does not keep a table's properties. + Properties map[string]string `json:"properties"` + // ManagedVersioning stays false: the dataset owns its version history, + // because this store can order commits without the catalog in the path. + ManagedVersioning bool `json:"managed_versioning"` + IsOnlyDeclared *bool `json:"is_only_declared,omitempty"` +} + +type TableExistsRequest struct { + ID []string `json:"id,omitempty"` +} + +type RegisterTableRequest struct { + ID []string `json:"id,omitempty"` + Location string `json:"location"` + Mode string `json:"mode,omitempty"` + Properties map[string]string `json:"properties,omitempty"` +} + +type RegisterTableResponse struct { + Location string `json:"location"` + Properties map[string]string `json:"properties"` +} + +type DeregisterTableRequest struct { + ID []string `json:"id,omitempty"` +} + +type DeregisterTableResponse struct { + ID []string `json:"id"` + Location string `json:"location"` + Properties map[string]string `json:"properties"` +} + +type DropTableRequest struct { + ID []string `json:"id,omitempty"` +} + +type DropTableResponse struct { + ID []string `json:"id"` + Location string `json:"location"` + Properties map[string]string `json:"properties"` +} + +type RenameTableRequest struct { + ID []string `json:"id,omitempty"` + NewID []string `json:"new_id"` +} + +type RenameTableResponse struct{} diff --git a/weed/s3api/s3tables/filer_ops.go b/weed/s3api/s3tables/filer_ops.go index eb991b9dd..c67b60c36 100644 --- a/weed/s3api/s3tables/filer_ops.go +++ b/weed/s3api/s3tables/filer_ops.go @@ -3,6 +3,7 @@ package s3tables import ( "bytes" "context" + "encoding/json" "errors" "fmt" "os" @@ -247,6 +248,31 @@ func (h *S3TablesHandler) getExtendedAttribute(ctx context.Context, client filer return data, nil } +// loadNamespaceMetadata resolves a namespace to its metadata. A directory that +// carries no namespace metadata is not a namespace, so a missing attribute +// reports the same absence as a missing entry and every caller tests one +// condition instead of forgetting the second. +func (h *S3TablesHandler) loadNamespaceMetadata(ctx context.Context, filerClient FilerClient, bucketName, namespaceName string) (*namespaceMetadata, error) { + var metadata namespaceMetadata + err := filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + data, err := h.getExtendedAttribute(ctx, client, GetNamespacePath(bucketName, namespaceName), ExtendedKeyMetadata) + if err != nil { + if errors.Is(err, ErrAttributeNotFound) { + return filer_pb.ErrNotFound + } + return err + } + if err := json.Unmarshal(data, &metadata); err != nil { + return fmt.Errorf("failed to unmarshal namespace metadata: %w", err) + } + return nil + }) + if err != nil { + return nil, err + } + return &metadata, nil +} + // lookupEntry returns the filer entry at the given path. func (h *S3TablesHandler) lookupEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, path string) (*filer_pb.Entry, error) { dir, name := splitPath(path) diff --git a/weed/s3api/s3tables/handler_bucket_create.go b/weed/s3api/s3tables/handler_bucket_create.go index 6adf29c6c..fd42c2ede 100644 --- a/weed/s3api/s3tables/handler_bucket_create.go +++ b/weed/s3api/s3tables/handler_bucket_create.go @@ -26,6 +26,19 @@ func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http return err } + // A bucket is a catalog, and a catalog serves one protocol. Saying which one + // at creation is what lets everything downstream - the endpoint the UI + // shows, the tables the bucket accepts - be answered without opening a table. + bucketFormat := FormatIceberg + if req.Format != "" { + normalized, ok := NormalizeFormat(req.Format) + if !ok { + h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, fmt.Sprintf("unsupported format %q", req.Format)) + return fmt.Errorf("invalid format") + } + bucketFormat = normalized + } + principal := h.getAccountID(r) identityActions := getIdentityActions(r) identityPolicyNames := getIdentityPolicyNames(r) @@ -115,6 +128,7 @@ func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http Name: req.Name, CreatedAt: now, OwnerAccountID: principal, + Format: bucketFormat, } metadataBytes, err := json.Marshal(metadata) diff --git a/weed/s3api/s3tables/handler_bucket_format_test.go b/weed/s3api/s3tables/handler_bucket_format_test.go new file mode 100644 index 000000000..b024314a7 --- /dev/null +++ b/weed/s3api/s3tables/handler_bucket_format_test.go @@ -0,0 +1,109 @@ +package s3tables + +import ( + "context" + "encoding/json" + "testing" + + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" + + "github.com/stretchr/testify/require" +) + +const formatTestBucket = "formats" + +// bucketWithFormat lays down a table bucket holding one namespace, declared as +// the given format. An empty format is a bucket from before the declaration +// existed. +func bucketWithFormat(t *testing.T, format string) (*s3tablestest.MemFiler, *Manager) { + t.Helper() + fs := s3tablestest.Start(t) + m := NewManager() + + bucketMeta, _ := json.Marshal(tableBucketMetadata{ + Name: formatTestBucket, + OwnerAccountID: DefaultAccountID, + Format: format, + }) + fs.Put(TablesPath, formatTestBucket, map[string][]byte{ + ExtendedKeyTableBucket: []byte("{}"), + ExtendedKeyMetadata: bucketMeta, + }) + + nsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"ns"}, OwnerAccountID: DefaultAccountID}) + fs.Put(GetTableBucketPath(formatTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta}) + + return fs, m +} + +func createTableOfFormat(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, name, format string) error { + t.Helper() + return m.Execute(context.Background(), NewManagerClient(fs.Client), "CreateTable", &CreateTableRequest{ + TableBucketARN: "arn:aws:s3tables:::bucket/" + formatTestBucket, + Namespace: []string{"ns"}, + Name: name, + Format: format, + }, nil, "") +} + +// The declaration is the point: a bucket that says it holds Lance cannot be +// handed an Iceberg table, because the catalog serving it would never show one. +func TestCreateTableRefusesAForeignFormat(t *testing.T) { + fs, m := bucketWithFormat(t, FormatLance) + + require.NoError(t, createTableOfFormat(t, m, fs, "vectors", FormatLance)) + + err := createTableOfFormat(t, m, fs, "events", FormatIceberg) + require.Error(t, err, "an Iceberg table in a Lance bucket should be refused") + require.Contains(t, err.Error(), "holds LANCE") +} + +func TestCreateTableRefusesLanceInAnIcebergBucket(t *testing.T) { + fs, m := bucketWithFormat(t, FormatIceberg) + + err := createTableOfFormat(t, m, fs, "vectors", FormatLance) + require.Error(t, err) + require.Contains(t, err.Error(), "holds ICEBERG") +} + +// A bucket made before formats were declared keeps taking anything. Nothing is +// migrated, so nothing that worked stops working. +func TestUndeclaredBucketAcceptsEitherFormat(t *testing.T) { + fs, m := bucketWithFormat(t, "") + + require.NoError(t, createTableOfFormat(t, m, fs, "events", FormatIceberg)) + require.NoError(t, createTableOfFormat(t, m, fs, "vectors", FormatLance)) +} + +// A view is Iceberg metadata, so it has no meaning in a bucket of another format. +func TestCreateViewRefusedInALanceBucket(t *testing.T) { + fs, m := bucketWithFormat(t, FormatLance) + + err := m.Execute(context.Background(), NewManagerClient(fs.Client), "CreateView", &CreateViewRequest{ + TableBucketARN: "arn:aws:s3tables:::bucket/" + formatTestBucket, + Namespace: []string{"ns"}, + Name: "v", + }, nil, "") + require.Error(t, err) + require.Contains(t, err.Error(), "cannot hold views") +} + +func TestNormalizeFormat(t *testing.T) { + cases := []struct { + in string + want string + ok bool + }{ + {"ICEBERG", FormatIceberg, true}, + {"iceberg", FormatIceberg, true}, + {" Lance ", FormatLance, true}, + {"delta", "", false}, + {"", "", false}, + } + for _, c := range cases { + got, ok := NormalizeFormat(c.in) + if got != c.want || ok != c.ok { + t.Errorf("NormalizeFormat(%q) = %q,%v; want %q,%v", c.in, got, ok, c.want, c.ok) + } + } +} diff --git a/weed/s3api/s3tables/handler_bucket_get_list_delete.go b/weed/s3api/s3tables/handler_bucket_get_list_delete.go index 18be86bf9..1cfdea623 100644 --- a/weed/s3api/s3tables/handler_bucket_get_list_delete.go +++ b/weed/s3api/s3tables/handler_bucket_get_list_delete.go @@ -83,6 +83,7 @@ func (h *S3TablesHandler) handleGetTableBucket(w http.ResponseWriter, r *http.Re Name: metadata.Name, OwnerAccountID: metadata.OwnerAccountID, CreatedAt: metadata.CreatedAt, + Format: metadata.Format, } h.writeJSON(w, http.StatusOK, resp) @@ -202,6 +203,7 @@ func (h *S3TablesHandler) handleListTableBuckets(w http.ResponseWriter, r *http. ARN: bucketARN, Name: entry.Entry.Name, CreatedAt: metadata.CreatedAt, + Format: metadata.Format, }) if len(buckets) >= maxBuckets { diff --git a/weed/s3api/s3tables/handler_delete_decouple_test.go b/weed/s3api/s3tables/handler_delete_decouple_test.go index 966d9b12e..b26966c86 100644 --- a/weed/s3api/s3tables/handler_delete_decouple_test.go +++ b/weed/s3api/s3tables/handler_delete_decouple_test.go @@ -5,13 +5,15 @@ import ( "encoding/json" "testing" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) -func runDeleteTable(t *testing.T, m *Manager, fs *memFilerServer, namespace, name string) error { +func runDeleteTable(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, namespace, name string) error { t.Helper() - return m.Execute(context.Background(), NewManagerClient(fs.client), "DeleteTable", &DeleteTableRequest{ + return m.Execute(context.Background(), NewManagerClient(fs.Client), "DeleteTable", &DeleteTableRequest{ TableBucketARN: mustBucketARN(t), Namespace: []string{namespace}, Name: name, @@ -33,24 +35,24 @@ func TestDeleteTableDecoupledKeepsReusedNamePath(t *testing.T) { MetadataLocation: "s3://" + renameTestBucket + "/ns/newt-x/metadata/v1.metadata.json", }) markerKeys := []string{ExtendedKeyMetadata, ExtendedKeyMetadataVersion, ExtendedKeyPolicy, ExtendedKeyTags, ExtendedKeyEntryType} - fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "newt", map[string][]byte{ + fs.Put(GetNamespacePath(renameTestBucket, "ns"), "newt", map[string][]byte{ ExtendedKeyMetadata: newtMeta, ExtendedKeyMetadataVersion: []byte("v1"), ExtendedKeyPolicy: []byte(`{"Version":"2012-10-17"}`), ExtendedKeyTags: []byte(`{"k":"v"}`), ExtendedKeyEntryType: []byte(EntryTypeTable), }) - fs.putEntry(GetTablePath(renameTestBucket, "ns", "newt"), "leftover", nil) // another table's data under the name path - fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "newt-x", nil) // this table's own (decoupled) data - fs.putEntry(GetTablePath(renameTestBucket, "ns", "newt-x"), "metadata", nil) + fs.Put(GetTablePath(renameTestBucket, "ns", "newt"), "leftover", nil) // another table's data under the name path + fs.Put(GetNamespacePath(renameTestBucket, "ns"), "newt-x", nil) // this table's own (decoupled) data + fs.Put(GetTablePath(renameTestBucket, "ns", "newt-x"), "metadata", nil) require.NoError(t, runDeleteTable(t, m, fs, "ns", "newt")) - assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "newt-x"), + assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "newt-x"), "the table's own data location must be purged") - assert.NotNil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "newt"), "leftover"), + assert.NotNil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "newt"), "leftover"), "data under the reused name path must survive") - marker := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "newt") + marker := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "newt") require.NotNil(t, marker) for _, key := range markerKeys { _, present := marker.Extended[key] @@ -72,14 +74,14 @@ func TestDeleteTableRefusesAncestorDataPath(t *testing.T) { OwnerAccountID: DefaultAccountID, MetadataLocation: "s3://" + renameTestBucket + "/ns/metadata/v1.metadata.json", }) - fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "badt", map[string][]byte{ExtendedKeyMetadata: badMeta}) + fs.Put(GetNamespacePath(renameTestBucket, "ns"), "badt", map[string][]byte{ExtendedKeyMetadata: badMeta}) require.Error(t, runDeleteTable(t, m, fs, "ns", "badt")) // The sibling table seeded by startRenameManager and its data must survive. - assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), + assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "sibling table marker must survive a refused delete") - assert.NotNil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "t"), "data"), + assert.NotNil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "t"), "data"), "sibling table data must survive a refused delete") } @@ -89,6 +91,6 @@ func TestDeleteTableColocatedRemovesData(t *testing.T) { require.NoError(t, runDeleteTable(t, m, fs, "ns", "t")) - assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "colocated table entry must be deleted") - assert.Nil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "t"), "metadata"), "colocated table data must be deleted") + assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "colocated table entry must be deleted") + assert.Nil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "t"), "metadata"), "colocated table data must be deleted") } diff --git a/weed/s3api/s3tables/handler_namespace_resolution_test.go b/weed/s3api/s3tables/handler_namespace_resolution_test.go new file mode 100644 index 000000000..be2ac9aa8 --- /dev/null +++ b/weed/s3api/s3tables/handler_namespace_resolution_test.go @@ -0,0 +1,45 @@ +package s3tables + +import ( + "context" + "errors" + "testing" + + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" +) + +// A directory under a table bucket that carries no namespace metadata is not a +// namespace. Reporting that as an internal error, which is what the three +// callers of this did while each testing only for a missing entry, turns a +// client mistake into a 500 and hides it behind "attribute not found". +func TestLoadNamespaceMetadataTreatsMissingAttributeAsAbsent(t *testing.T) { + filer := s3tablestest.Start(t) + handler := NewS3TablesHandler() + client := NewManagerClient(filer.Client) + ctx := context.Background() + + filer.Put(TablesPath, "bkt", map[string][]byte{ExtendedKeyTableBucket: []byte("{}")}) + + // A stray directory where a namespace would live, with no metadata on it. + filer.Put(GetTableBucketPath("bkt"), "stray", nil) + + if _, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "stray"); !errors.Is(err, filer_pb.ErrNotFound) { + t.Fatalf("loadNamespaceMetadata on a directory without metadata = %v, want ErrNotFound", err) + } + + if _, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "absent"); !errors.Is(err, filer_pb.ErrNotFound) { + t.Fatalf("loadNamespaceMetadata on a missing directory = %v, want ErrNotFound", err) + } + + filer.Put(GetTableBucketPath("bkt"), "real", map[string][]byte{ + ExtendedKeyMetadata: []byte(`{"namespace":["real"],"ownerAccountId":"000000000000"}`), + }) + metadata, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "real") + if err != nil { + t.Fatalf("loadNamespaceMetadata on a real namespace: %v", err) + } + if metadata.OwnerAccountID != "000000000000" { + t.Fatalf("owner = %q, want the stored one", metadata.OwnerAccountID) + } +} diff --git a/weed/s3api/s3tables/handler_rename_test.go b/weed/s3api/s3tables/handler_rename_test.go index a7f265937..88fa99a63 100644 --- a/weed/s3api/s3tables/handler_rename_test.go +++ b/weed/s3api/s3tables/handler_rename_test.go @@ -1,165 +1,18 @@ package s3tables import ( - "bytes" "context" "encoding/json" - "net" "path" - "sort" - "strings" "testing" - "time" "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" "github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "google.golang.org/grpc" - "google.golang.org/grpc/codes" - "google.golang.org/grpc/credentials/insecure" - "google.golang.org/grpc/status" ) -// memFilerServer is an in-memory filer used to drive Manager operations -// end-to-end without a live cluster. -type memFilerServer struct { - filer_pb.UnimplementedSeaweedFilerServer - entries map[string]map[string]*filer_pb.Entry // dir -> name -> entry - client filer_pb.SeaweedFilerClient - // beforeUpdate runs once, at the start of the next UpdateEntry, so a test - // can land a competing write in a handler's read-to-write window. - beforeUpdate func() -} - -func newMemFilerServer() *memFilerServer { - return &memFilerServer{entries: make(map[string]map[string]*filer_pb.Entry)} -} - -func (f *memFilerServer) getEntry(dir, name string) *filer_pb.Entry { - if d, ok := f.entries[dir]; ok { - return d[name] - } - return nil -} - -func (f *memFilerServer) putEntry(dir, name string, extended map[string][]byte) { - if _, ok := f.entries[dir]; !ok { - f.entries[dir] = make(map[string]*filer_pb.Entry) - } - f.entries[dir][name] = &filer_pb.Entry{Name: name, IsDirectory: true, Extended: extended} -} - -func (f *memFilerServer) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) { - if e := f.getEntry(req.Directory, req.Name); e != nil { - return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil - } - // Carry the sentinel text so filer_pb.LookupEntry maps it to ErrNotFound. - return nil, status.Errorf(codes.NotFound, "%s: %s/%s", filer_pb.ErrNotFound.Error(), req.Directory, req.Name) -} - -func (f *memFilerServer) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error { - d, ok := f.entries[req.Directory] - if !ok { - return nil - } - names := make([]string, 0, len(d)) - for name := range d { - names = append(names, name) - } - sort.Strings(names) - for _, name := range names { - if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: d[name]}); err != nil { - return err - } - } - return nil -} - -func (f *memFilerServer) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest) (*filer_pb.CreateEntryResponse, error) { - if _, ok := f.entries[req.Directory]; !ok { - f.entries[req.Directory] = make(map[string]*filer_pb.Entry) - } - f.entries[req.Directory][req.Entry.Name] = req.Entry - return &filer_pb.CreateEntryResponse{}, nil -} - -func (f *memFilerServer) UpdateEntry(_ context.Context, req *filer_pb.UpdateEntryRequest) (*filer_pb.UpdateEntryResponse, error) { - if hook := f.beforeUpdate; hook != nil { - f.beforeUpdate = nil - hook() - } - // The real filer validates ExpectedExtended under the per-path lock; without - // it here a lost update would look like a success. - for key, expected := range req.ExpectedExtended { - var actual []byte - if existing := f.getEntry(req.Directory, req.Entry.Name); existing != nil { - actual = existing.Extended[key] - } - if !bytes.Equal(actual, expected) { - return nil, status.Errorf(codes.FailedPrecondition, "extended attribute %q changed", key) - } - } - if _, ok := f.entries[req.Directory]; !ok { - f.entries[req.Directory] = make(map[string]*filer_pb.Entry) - } - f.entries[req.Directory][req.Entry.Name] = req.Entry - return &filer_pb.UpdateEntryResponse{}, nil -} - -func (f *memFilerServer) DeleteEntry(_ context.Context, req *filer_pb.DeleteEntryRequest) (*filer_pb.DeleteEntryResponse, error) { - if d, ok := f.entries[req.Directory]; ok { - delete(d, req.Name) - } - // Honor recursive data deletion so a regression that wipes the table directory - // also drops its metadata/ and data/ children (the data-loss this guards against). - if req.IsRecursive && req.IsDeleteData { - child := path.Join(req.Directory, req.Name) - for dir := range f.entries { - if dir == child || strings.HasPrefix(dir, child+"/") { - delete(f.entries, dir) - } - } - } - return &filer_pb.DeleteEntryResponse{}, nil -} - -func (f *memFilerServer) Ping(_ context.Context, _ *filer_pb.PingRequest) (*filer_pb.PingResponse, error) { - now := time.Now().UnixNano() - return &filer_pb.PingResponse{StartTimeNs: now, RemoteTimeNs: now, StopTimeNs: now}, nil -} - -func startMemFiler(t *testing.T) *memFilerServer { - t.Helper() - fs := newMemFilerServer() - - listener, err := net.Listen("tcp", "127.0.0.1:0") - require.NoError(t, err) - - server := grpc.NewServer() - filer_pb.RegisterSeaweedFilerServer(server, fs) - go func() { _ = server.Serve(listener) }() - t.Cleanup(server.GracefulStop) - - conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials())) - require.NoError(t, err) - t.Cleanup(func() { _ = conn.Close() }) - - fs.client = filer_pb.NewSeaweedFilerClient(conn) - deadline := time.Now().Add(5 * time.Second) - for { - pingCtx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) - _, err := fs.client.Ping(pingCtx, &filer_pb.PingRequest{}) - cancel() - if err == nil { - break - } - require.False(t, time.Now().After(deadline), "filer not ready: %v", err) - time.Sleep(10 * time.Millisecond) - } - return fs -} - const renameTestBucket = "renamebkt" func mustBucketARN(t *testing.T) string { @@ -170,18 +23,18 @@ func mustBucketARN(t *testing.T) string { } // startRenameManager seeds a bucket/namespace/table and returns a trusted Manager. -func startRenameManager(t *testing.T) (*memFilerServer, *Manager) { +func startRenameManager(t *testing.T) (*s3tablestest.MemFiler, *Manager) { t.Helper() - fs := startMemFiler(t) + fs := s3tablestest.Start(t) bucketMeta, _ := json.Marshal(tableBucketMetadata{Name: renameTestBucket, OwnerAccountID: DefaultAccountID}) - fs.putEntry(TablesPath, renameTestBucket, map[string][]byte{ + fs.Put(TablesPath, renameTestBucket, map[string][]byte{ ExtendedKeyTableBucket: []byte("{}"), ExtendedKeyMetadata: bucketMeta, }) nsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"ns"}, OwnerAccountID: DefaultAccountID}) - fs.putEntry(GetTableBucketPath(renameTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta}) + fs.Put(GetTableBucketPath(renameTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta}) tableMeta, _ := json.Marshal(tableMetadataInternal{ Name: "t", @@ -191,31 +44,31 @@ func startRenameManager(t *testing.T) (*memFilerServer, *Manager) { MetadataVersion: 3, MetadataLocation: "s3://" + renameTestBucket + "/ns/t/metadata/v3.metadata.json", }) - fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "t", map[string][]byte{ + fs.Put(GetNamespacePath(renameTestBucket, "ns"), "t", map[string][]byte{ ExtendedKeyMetadata: tableMeta, ExtendedKeyMetadataVersion: []byte("3"), }) // Physical metadata.json and data files live under the table directory. tablePath := GetTablePath(renameTestBucket, "ns", "t") - fs.putEntry(tablePath, "metadata", nil) - fs.putEntry(tablePath, "data", nil) - fs.putEntry(path.Join(tablePath, "metadata"), "v3.metadata.json", nil) + fs.Put(tablePath, "metadata", nil) + fs.Put(tablePath, "data", nil) + fs.Put(path.Join(tablePath, "metadata"), "v3.metadata.json", nil) m := NewManager() m.SetTrusted(true) return fs, m } -func runRename(t *testing.T, m *Manager, fs *memFilerServer, req *RenameTableRequest) error { +func runRename(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, req *RenameTableRequest) error { t.Helper() - return m.Execute(context.Background(), NewManagerClient(fs.client), "RenameTable", req, nil, "") + return m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameTable", req, nil, "") } -func runGetTable(t *testing.T, m *Manager, fs *memFilerServer, namespace, name string) (*GetTableResponse, error) { +func runGetTable(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, namespace, name string) (*GetTableResponse, error) { t.Helper() resp := &GetTableResponse{} - err := m.Execute(context.Background(), NewManagerClient(fs.client), "GetTable", &GetTableRequest{ + err := m.Execute(context.Background(), NewManagerClient(fs.Client), "GetTable", &GetTableRequest{ TableBucketARN: mustBucketARN(t), Namespace: []string{namespace}, Name: name, @@ -238,12 +91,12 @@ func TestRenameTablePreservesData(t *testing.T) { // The source directory and its metadata.json/data children must survive: rename // is catalog-only and the destination still points at the original location. srcPath := GetTablePath(renameTestBucket, "ns", "t") - assert.NotNil(t, fs.getEntry(srcPath, "metadata"), "source metadata dir must survive") - assert.NotNil(t, fs.getEntry(srcPath, "data"), "source data dir must survive") - assert.NotNil(t, fs.getEntry(path.Join(srcPath, "metadata"), "v3.metadata.json"), "metadata.json must survive") + assert.NotNil(t, fs.Get(srcPath, "metadata"), "source metadata dir must survive") + assert.NotNil(t, fs.Get(srcPath, "data"), "source data dir must survive") + assert.NotNil(t, fs.Get(path.Join(srcPath, "metadata"), "v3.metadata.json"), "metadata.json must survive") // Source catalog xattrs are dropped so the name stops resolving. - src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, src, "source directory must remain to hold the data children") _, hasMeta := src.Extended[ExtendedKeyMetadata] assert.False(t, hasMeta, "source table-metadata xattr must be removed") @@ -260,7 +113,7 @@ func TestRenameTablePreservesData(t *testing.T) { assert.Equal(t, "t2", got.Name) assert.Equal(t, "s3://"+renameTestBucket+"/ns/t/metadata/v3.metadata.json", got.MetadataLocation) - dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t2") + dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t2") require.NotNil(t, dest) assert.Equal(t, []byte("3"), dest.Extended[ExtendedKeyMetadataVersion]) } @@ -283,7 +136,7 @@ func TestRenameTableSourceMissing(t *testing.T) { func TestRenameTableDestExists(t *testing.T) { fs, m := startRenameManager(t) existing, _ := json.Marshal(tableMetadataInternal{Name: "t2", Namespace: "ns", OwnerAccountID: DefaultAccountID}) - fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "t2", map[string][]byte{ExtendedKeyMetadata: existing}) + fs.Put(GetNamespacePath(renameTestBucket, "ns"), "t2", map[string][]byte{ExtendedKeyMetadata: existing}) err := runRename(t, m, fs, &RenameTableRequest{ TableBucketARN: mustBucketARN(t), @@ -296,7 +149,7 @@ func TestRenameTableDestExists(t *testing.T) { var s3Err *S3TablesError require.ErrorAs(t, err, &s3Err) assert.Equal(t, ErrCodeTableAlreadyExists, s3Err.Type) - assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched on conflict") + assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched on conflict") } func TestRenameTableDestNamespaceMissing(t *testing.T) { @@ -312,7 +165,7 @@ func TestRenameTableDestNamespaceMissing(t *testing.T) { var s3Err *S3TablesError require.ErrorAs(t, err, &s3Err) assert.Equal(t, ErrCodeNoSuchNamespace, s3Err.Type) - assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched") + assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched") } // A principal allowed to rename the source must still be denied when it cannot @@ -332,16 +185,16 @@ func TestRenameTableDestNamespaceUnauthorized(t *testing.T) { "Resource": "*", }}, }) - srcEntry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + srcEntry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, srcEntry) srcEntry.Extended[ExtendedKeyPolicy] = srcPolicy destNsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"dest"}, OwnerAccountID: DefaultAccountID}) - fs.putEntry(GetTableBucketPath(renameTestBucket), "dest", map[string][]byte{ExtendedKeyMetadata: destNsMeta}) + fs.Put(GetTableBucketPath(renameTestBucket), "dest", map[string][]byte{ExtendedKeyMetadata: destNsMeta}) mover := &testIdentity{Name: "mover", Account: &testIdentityAccount{Id: "mover"}} ctx := s3_constants.SetIdentityInContext(context.Background(), mover) - err := m.Execute(ctx, NewManagerClient(fs.client), "RenameTable", &RenameTableRequest{ + err := m.Execute(ctx, NewManagerClient(fs.Client), "RenameTable", &RenameTableRequest{ TableBucketARN: mustBucketARN(t), SourceNamespace: []string{"ns"}, SourceName: "t", @@ -353,8 +206,8 @@ func TestRenameTableDestNamespaceUnauthorized(t *testing.T) { require.ErrorAs(t, err, &s3Err) assert.Equal(t, ErrCodeAccessDenied, s3Err.Type) - assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched") - assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "dest"), "t2"), "destination must not be written") + assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched") + assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "dest"), "t2"), "destination must not be written") } func TestRenameTableInvalidName(t *testing.T) { @@ -378,7 +231,7 @@ func TestRenameTableInvalidName(t *testing.T) { func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) { fs, m := startRenameManager(t) - src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, src) config := []byte(`{"icebergSnapshotManagement":{"status":"disabled"}}`) status := []byte(`{"icebergCompaction":{"status":"Successful"}}`) @@ -393,13 +246,13 @@ func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) { DestName: "t2", })) - dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t2") + dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t2") require.NotNil(t, dest) assert.Equal(t, config, dest.Extended[ExtendedKeyMaintenance], "the disable must move with the table") assert.Equal(t, status, dest.Extended[ExtendedKeyMaintenanceStatus]) // And must not linger on the old name, where a reused name would inherit it. - moved := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + moved := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, moved) _, hasConfig := moved.Extended[ExtendedKeyMaintenance] assert.False(t, hasConfig, "source maintenance configuration must be cleared") @@ -413,7 +266,7 @@ func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) { func TestRenameTableRejectsConcurrentMaintenanceWrite(t *testing.T) { fs, _ := startRenameManager(t) - src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, src) copied := []byte(`{"icebergCompaction":{"status":"enabled"}}`) src.Extended[ExtendedKeyMaintenance] = copied @@ -425,7 +278,7 @@ func TestRenameTableRejectsConcurrentMaintenanceWrite(t *testing.T) { src.Extended[ExtendedKeyMaintenance] = landedLate h := NewS3TablesHandler() - err := NewManagerClient(fs.client).WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { + err := NewManagerClient(fs.Client).WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { return h.removeExtendedAttributesIf(context.Background(), client, GetTablePath(renameTestBucket, "ns", "t"), expected, renamedTableAttributes...) }) diff --git a/weed/s3api/s3tables/handler_table.go b/weed/s3api/s3tables/handler_table.go index 4e5a357a9..987a97347 100644 --- a/weed/s3api/s3tables/handler_table.go +++ b/weed/s3api/s3tables/handler_table.go @@ -45,8 +45,8 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque } // Validate format - if req.Format != "ICEBERG" { - h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "only ICEBERG format is supported") + if req.Format != FormatIceberg && req.Format != FormatLance { + h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, fmt.Sprintf("unsupported format %q", req.Format)) return fmt.Errorf("invalid format") } @@ -65,18 +65,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque // Check if namespace exists namespacePath := GetNamespacePath(bucketName, namespaceName) - var namespaceMetadata namespaceMetadata - err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { - data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata) - if err != nil { - return err - } - if err := json.Unmarshal(data, &namespaceMetadata); err != nil { - return fmt.Errorf("failed to unmarshal namespace metadata: %w", err) - } - return nil - }) - + namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName) if err != nil { if errors.Is(err, filer_pb.ErrNotFound) { h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName)) @@ -135,6 +124,15 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque return err } + // A bucket declares the format it holds, and a table of another format would + // be invisible to the catalog serving it. A bucket made before the + // declaration existed has none, and keeps taking anything. + if bucketMetadata.Format != "" && bucketMetadata.Format != req.Format { + message := fmt.Sprintf("table bucket %s holds %s tables", bucketName, bucketMetadata.Format) + h.writeError(w, http.StatusConflict, ErrCodeConflict, message) + return fmt.Errorf("%s", message) + } + bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName) identityActions := getIdentityActions(r) nsAllowed := CheckPermissionWithContext("CreateTable", accountID, namespaceMetadata.OwnerAccountID, namespacePolicy, bucketARN, &PolicyContext{ @@ -174,7 +172,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque if err != nil { return err } - if entryType(entry.Extended) == EntryTypeView { + if EntryType(entry.Extended) == EntryTypeView { existingIsView = true return nil } @@ -193,6 +191,14 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque h.writeError(w, http.StatusConflict, ErrCodeTableAlreadyExists, fmt.Sprintf("a view named %s already exists", tableName)) return fmt.Errorf("view name conflict: %s", tableName) } + // Creating a table that already exists is idempotent, but only for the + // same format. Handing a Lance client an Iceberg table's location, or the + // reverse, has it write one format's files into the other's directory. + if existingMetadata.Format != "" && existingMetadata.Format != req.Format { + h.writeError(w, http.StatusConflict, ErrCodeTableAlreadyExists, + fmt.Sprintf("a %s table named %s already exists", existingMetadata.Format, tableName)) + return fmt.Errorf("format conflict: %s", tableName) + } tableARN := h.generateTableARN(existingMetadata.OwnerAccountID, bucketName, namespaceName+"/"+tableName) h.writeJSON(w, http.StatusOK, &CreateTableResponse{ TableARN: tableARN, @@ -357,17 +363,7 @@ func (h *S3TablesHandler) handleRegisterTable(w http.ResponseWriter, r *http.Req // Namespace must exist. namespacePath := GetNamespacePath(bucketName, namespaceName) - var namespaceMetadata namespaceMetadata - err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { - data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata) - if err != nil { - return err - } - if err := json.Unmarshal(data, &namespaceMetadata); err != nil { - return fmt.Errorf("failed to unmarshal namespace metadata: %w", err) - } - return nil - }) + namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName) if err != nil { if errors.Is(err, filer_pb.ErrNotFound) { h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName)) @@ -468,7 +464,7 @@ func (h *S3TablesHandler) handleRegisterTable(w http.ResponseWriter, r *http.Req metadata := &tableMetadataInternal{ Name: tableName, Namespace: namespaceName, - Format: "ICEBERG", + Format: FormatIceberg, CreatedAt: now, ModifiedAt: now, OwnerAccountID: namespaceMetadata.OwnerAccountID, @@ -551,7 +547,7 @@ func (h *S3TablesHandler) handleGetTable(w http.ResponseWriter, r *http.Request, if err != nil { return err } - if entryType(entry.Extended) == EntryTypeView { + if EntryType(entry.Extended) == EntryTypeView { return filer_pb.ErrNotFound } data, ok := entry.Extended[ExtendedKeyMetadata] @@ -923,7 +919,7 @@ func (h *S3TablesHandler) listTablesWithClient(r *http.Request, client filer_pb. } // Views share the table layout; exclude them from table listings. - if entryType(entry.Entry.Extended) == EntryTypeView { + if EntryType(entry.Entry.Extended) == EntryTypeView { continue } @@ -945,11 +941,13 @@ func (h *S3TablesHandler) listTablesWithClient(r *http.Request, client filer_pb. tableARN := h.generateTableARN(metadata.OwnerAccountID, bucketName, namespaceName+"/"+entry.Entry.Name) tables = append(tables, TableSummary{ - Name: entry.Entry.Name, - TableARN: tableARN, - Namespace: expandNamespace(namespaceName), - CreatedAt: metadata.CreatedAt, - ModifiedAt: metadata.ModifiedAt, + Name: entry.Entry.Name, + TableARN: tableARN, + Namespace: expandNamespace(namespaceName), + Format: metadata.Format, + CreatedAt: metadata.CreatedAt, + ModifiedAt: metadata.ModifiedAt, + MetadataLocation: metadata.MetadataLocation, }) if len(tables) >= maxTables { @@ -1435,7 +1433,7 @@ func (h *S3TablesHandler) renameCatalogEntry(w http.ResponseWriter, r *http.Requ // Tables and views share the namespace directory, so a rename must not pick // up the other kind under the same name. - if entryType(srcExtended) != kind.entryType { + if EntryType(srcExtended) != kind.entryType { h.writeError(w, http.StatusNotFound, kind.notFoundCode, fmt.Sprintf("%s %s not found", kind.noun, srcName)) return fmt.Errorf("%s %s not found", kind.noun, srcName) } diff --git a/weed/s3api/s3tables/handler_update_cas_test.go b/weed/s3api/s3tables/handler_update_cas_test.go index ab5b3eff7..aca6539fd 100644 --- a/weed/s3api/s3tables/handler_update_cas_test.go +++ b/weed/s3api/s3tables/handler_update_cas_test.go @@ -26,7 +26,7 @@ func TestUpdateTableRejectsLostUpdate(t *testing.T) { fs, m := startRenameManager(t) winnerLocation := "s3://" + renameTestBucket + "/ns/t/metadata/v4-winner.metadata.json" - fs.beforeUpdate = func() { + fs.BeforeUpdate = func() { winner, err := json.Marshal(tableMetadataInternal{ Name: "t", Namespace: "ns", @@ -37,13 +37,13 @@ func TestUpdateTableRejectsLostUpdate(t *testing.T) { VersionToken: generateVersionToken(), }) require.NoError(t, err) - entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, entry) entry.Extended[ExtendedKeyMetadata] = winner } loserLocation := "s3://" + renameTestBucket + "/ns/t/metadata/v4-loser.metadata.json" - err := m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable", + err := m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable", updateTableRequest(t, loserLocation, 4), nil, "") require.Error(t, err) @@ -64,14 +64,14 @@ func TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges(t *testing.T) { fs, m := startRenameManager(t) policy := []byte(`{"Version":"2012-10-17","Statement":[]}`) - fs.beforeUpdate = func() { - entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + fs.BeforeUpdate = func() { + entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, entry) entry.Extended[ExtendedKeyPolicy] = policy } location := "s3://" + renameTestBucket + "/ns/t/metadata/v4.metadata.json" - err := m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable", + err := m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable", updateTableRequest(t, location, 4), nil, "") require.Error(t, err) @@ -79,7 +79,7 @@ func TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges(t *testing.T) { require.ErrorAs(t, err, &s3Err) assert.Equal(t, ErrCodeConflict, s3Err.Type) - entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t") + entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t") require.NotNil(t, entry) assert.Equal(t, policy, entry.Extended[ExtendedKeyPolicy], "the concurrent policy write must survive") } @@ -88,7 +88,7 @@ func TestUpdateTableAppliesWithoutContention(t *testing.T) { fs, m := startRenameManager(t) location := "s3://" + renameTestBucket + "/ns/t/metadata/v4.metadata.json" - require.NoError(t, m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable", + require.NoError(t, m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable", updateTableRequest(t, location, 4), nil, "")) got, err := runGetTable(t, m, fs, "ns", "t") diff --git a/weed/s3api/s3tables/handler_view.go b/weed/s3api/s3tables/handler_view.go index 79a4e8f0c..e41232204 100644 --- a/weed/s3api/s3tables/handler_view.go +++ b/weed/s3api/s3tables/handler_view.go @@ -52,14 +52,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques // Check if namespace exists namespacePath := GetNamespacePath(bucketName, namespaceName) - var namespaceMetadata namespaceMetadata - err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error { - data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata) - if err != nil { - return err - } - return json.Unmarshal(data, &namespaceMetadata) - }) + namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName) if err != nil { if errors.Is(err, filer_pb.ErrNotFound) { h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName)) @@ -77,6 +70,14 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques return err } + // A view is Iceberg metadata, so it belongs only in a bucket that holds + // Iceberg tables. + if bucketMetadata.Format != "" && bucketMetadata.Format != FormatIceberg { + message := fmt.Sprintf("table bucket %s holds %s tables and cannot hold views", bucketName, bucketMetadata.Format) + h.writeError(w, http.StatusConflict, ErrCodeConflict, message) + return fmt.Errorf("%s", message) + } + bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName) if !h.authorizeViewOp(r, "CreateView", accountID, namespaceMetadata.OwnerAccountID, bucketMetadata.OwnerAccountID, namespacePolicy, bucketPolicy, bucketARN, bucketName, namespaceName, viewName, bucketTags) { h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to create view in this namespace") @@ -93,7 +94,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques if err != nil { return err } - if entryType(entry.Extended) != EntryTypeView { + if EntryType(entry.Extended) != EntryTypeView { existingIsTable = true return nil } @@ -125,7 +126,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques metadata := &tableMetadataInternal{ Name: viewName, Namespace: namespaceName, - Format: "ICEBERG", + Format: FormatIceberg, CreatedAt: now, ModifiedAt: now, OwnerAccountID: namespaceMetadata.OwnerAccountID, @@ -185,7 +186,7 @@ func (h *S3TablesHandler) handleGetView(w http.ResponseWriter, r *http.Request, if err != nil { return err } - if entryType(entry.Extended) != EntryTypeView { + if EntryType(entry.Extended) != EntryTypeView { return filer_pb.ErrNotFound } data, ok := entry.Extended[ExtendedKeyMetadata] @@ -354,7 +355,7 @@ func (h *S3TablesHandler) listViewsInNamespace(r *http.Request, client filer_pb. continue } // Only include view entries; skip tables and untagged entries. - if entryType(entry.Entry.Extended) != EntryTypeView { + if EntryType(entry.Entry.Extended) != EntryTypeView { continue } data, ok := entry.Entry.Extended[ExtendedKeyMetadata] @@ -411,7 +412,7 @@ func (h *S3TablesHandler) handleUpdateView(w http.ResponseWriter, r *http.Reques if err != nil { return err } - if entryType(entry.Extended) != EntryTypeView { + if EntryType(entry.Extended) != EntryTypeView { return filer_pb.ErrNotFound } data, ok := entry.Extended[ExtendedKeyMetadata] @@ -516,7 +517,7 @@ func (h *S3TablesHandler) handleDeleteView(w http.ResponseWriter, r *http.Reques if err != nil { return err } - if entryType(entry.Extended) != EntryTypeView { + if EntryType(entry.Extended) != EntryTypeView { return filer_pb.ErrNotFound } data, ok := entry.Extended[ExtendedKeyMetadata] diff --git a/weed/s3api/s3tables/handler_view_rename_test.go b/weed/s3api/s3tables/handler_view_rename_test.go index 237f2f141..0afd1be11 100644 --- a/weed/s3api/s3tables/handler_view_rename_test.go +++ b/weed/s3api/s3tables/handler_view_rename_test.go @@ -6,12 +6,14 @@ import ( "path" "testing" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) // seedView adds a view alongside the table the rename harness already creates. -func seedView(t *testing.T, fs *memFilerServer, name string) { +func seedView(t *testing.T, fs *s3tablestest.MemFiler, name string) { t.Helper() viewMeta, err := json.Marshal(tableMetadataInternal{ @@ -24,19 +26,19 @@ func seedView(t *testing.T, fs *memFilerServer, name string) { }) require.NoError(t, err) - fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), name, map[string][]byte{ + fs.Put(GetNamespacePath(renameTestBucket, "ns"), name, map[string][]byte{ ExtendedKeyMetadata: viewMeta, ExtendedKeyMetadataVersion: []byte("1"), ExtendedKeyEntryType: []byte(EntryTypeView), }) viewPath := GetTablePath(renameTestBucket, "ns", name) - fs.putEntry(viewPath, "metadata", nil) - fs.putEntry(path.Join(viewPath, "metadata"), "v1.metadata.json", nil) + fs.Put(viewPath, "metadata", nil) + fs.Put(path.Join(viewPath, "metadata"), "v1.metadata.json", nil) } -func runRenameView(t *testing.T, m *Manager, fs *memFilerServer, sourceName, destName string) error { +func runRenameView(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, sourceName, destName string) error { t.Helper() - return m.Execute(context.Background(), NewManagerClient(fs.client), "RenameView", &RenameTableRequest{ + return m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameView", &RenameTableRequest{ TableBucketARN: mustBucketARN(t), SourceNamespace: []string{"ns"}, SourceName: sourceName, @@ -51,9 +53,9 @@ func TestRenameViewMovesCatalogPointer(t *testing.T) { require.NoError(t, runRenameView(t, m, fs, "v", "v2")) - dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v2") + dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v2") require.NotNil(t, dest) - assert.Equal(t, EntryTypeView, entryType(dest.Extended), "destination must stay a view") + assert.Equal(t, EntryTypeView, EntryType(dest.Extended), "destination must stay a view") var moved tableMetadataInternal require.NoError(t, json.Unmarshal(dest.Extended[ExtendedKeyMetadata], &moved)) @@ -61,11 +63,11 @@ func TestRenameViewMovesCatalogPointer(t *testing.T) { assert.Equal(t, "s3://"+renameTestBucket+"/ns/v/metadata/v1.metadata.json", moved.MetadataLocation, "rename is catalog-only, the metadata stays where it was written") - src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v") + src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v") require.NotNil(t, src) _, stillListed := src.Extended[ExtendedKeyMetadata] assert.False(t, stillListed, "source name must stop resolving") - assert.NotNil(t, fs.getEntry(path.Join(GetTablePath(renameTestBucket, "ns", "v"), "metadata"), "v1.metadata.json"), + assert.NotNil(t, fs.Get(path.Join(GetTablePath(renameTestBucket, "ns", "v"), "metadata"), "v1.metadata.json"), "the view's metadata file must survive") } @@ -110,18 +112,18 @@ func TestRenameViewAuthorizesAgainstTheViewARN(t *testing.T) { viewARN := "arn:aws:s3tables:" + DefaultRegion + ":" + DefaultAccountID + ":bucket/" + renameTestBucket + "/view/ns/v" viewPolicy := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"` + principal + `","Action":"s3tables:RenameView","Resource":"` + viewARN + `"}]}` - view := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v") + view := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v") require.NotNil(t, view) view.Extended[ExtendedKeyPolicy] = []byte(viewPolicy) // Landing in the namespace needs create permission there as well. namespacePolicy := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"` + principal + `","Action":"s3tables:CreateView","Resource":"` + mustBucketARN(t) + `"}]}` - namespace := fs.getEntry(GetTableBucketPath(renameTestBucket), "ns") + namespace := fs.Get(GetTableBucketPath(renameTestBucket), "ns") require.NotNil(t, namespace) namespace.Extended[ExtendedKeyPolicy] = []byte(namespacePolicy) - err := m.Execute(context.Background(), NewManagerClient(fs.client), "RenameView", &RenameTableRequest{ + err := m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameView", &RenameTableRequest{ TableBucketARN: mustBucketARN(t), SourceNamespace: []string{"ns"}, SourceName: "v", @@ -130,5 +132,5 @@ func TestRenameViewAuthorizesAgainstTheViewARN(t *testing.T) { }, nil, principal) require.NoError(t, err) - assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v2")) + assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v2")) } diff --git a/weed/s3api/s3tables/handler_view_test.go b/weed/s3api/s3tables/handler_view_test.go index 0a9cc0e5a..8788cf529 100644 --- a/weed/s3api/s3tables/handler_view_test.go +++ b/weed/s3api/s3tables/handler_view_test.go @@ -16,8 +16,8 @@ func TestEntryType(t *testing.T) { } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - if got := entryType(c.extended); got != c.want { - t.Fatalf("entryType() = %q, want %q", got, c.want) + if got := EntryType(c.extended); got != c.want { + t.Fatalf("EntryType() = %q, want %q", got, c.want) } }) } diff --git a/weed/s3api/s3tables/iceberg_layout.go b/weed/s3api/s3tables/iceberg_layout.go index c9083c20a..f28ddd813 100644 --- a/weed/s3api/s3tables/iceberg_layout.go +++ b/weed/s3api/s3tables/iceberg_layout.go @@ -20,12 +20,29 @@ import ( const uuidPattern = `[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}` var ( - // Allowed directories in an Iceberg table - icebergAllowedDirs = map[string]bool{ + // Allowed directories in a table. The set is the union of what every + // supported format writes, because this runs on the S3 door where the + // table's format is not in hand: metadata/ and data/ for Iceberg, + // _versions/ and _indices/ for Lance, which also puts its fragments in + // data/. + tableAllowedDirs = map[string]bool{ "metadata": true, "data": true, } + // A format's own bookkeeping lives in underscore-prefixed directories that + // the catalog does not interpret: Lance writes _versions, _transactions, + // _indices and _deletions, and enumerating them here would mean guessing at + // the next one. Iceberg writes none, so admitting them costs it nothing. + formatOwnedDirPattern = regexp.MustCompile(`^_[a-z][a-z0-9_]*$`) + + // Marker files a Lance table keeps at its root to record that it is + // reserved but not yet written, or deregistered but still on storage. + lanceMarkerFiles = map[string]bool{ + ".lance-reserved": true, + ".lance-deregistered": true, + } + // Patterns for valid metadata files. // // Note: Iceberg engines (Flink/Spark/Trino, plus different Iceberg versions) @@ -58,6 +75,7 @@ var ( regexp.MustCompile(`^[^/]+\.parquet$`), // Parquet files regexp.MustCompile(`^[^/]+\.orc$`), // ORC files regexp.MustCompile(`^[^/]+\.avro$`), // Avro files + regexp.MustCompile(`^[^/]+\.lance$`), // Lance fragments } // Data file partition path pattern (e.g., year=2024/month=01/) @@ -91,11 +109,16 @@ func (v *IcebergLayoutValidator) ValidateFilePath(relativePath string) error { topDir := parts[0] + // A Lance table's marker files sit at its root rather than in a directory. + if len(parts) == 1 && lanceMarkerFiles[topDir] { + return nil + } + // Check if top-level directory is allowed - if !icebergAllowedDirs[topDir] { + if !tableAllowedDirs[topDir] && !formatOwnedDirPattern.MatchString(topDir) { return &IcebergLayoutError{ Code: ErrCodeInvalidIcebergLayout, - Message: "files must be placed in 'metadata/' or 'data/' directories", + Message: "files must be placed in 'metadata/', 'data/' or a format-owned '_' directory", } } @@ -119,6 +142,28 @@ func (v *IcebergLayoutValidator) ValidateFilePath(relativePath string) error { return v.validateDataFile(remainingPath) } + // Everything else is a format-owned directory: the only guard is that the + // path stays inside the table. + return validateFileSegments(remainingPath) +} + +// IsTableMarkerFile reports whether a name is one of the marker files a table +// keeps at its root. +func IsTableMarkerFile(name string) bool { + return lanceMarkerFiles[name] +} + +// validateFileSegments rejects traversal and empty segments in a path whose +// contents the catalog does not otherwise interpret. +func validateFileSegments(path string) error { + for _, segment := range strings.Split(path, "/") { + if segment == "" || segment == "." || segment == ".." { + return &IcebergLayoutError{ + Code: ErrCodeInvalidIcebergLayout, + Message: "invalid path segment in: " + path, + } + } + } return nil } diff --git a/weed/s3api/s3tables/iceberg_layout_test.go b/weed/s3api/s3tables/iceberg_layout_test.go index 62e32c518..701be3532 100644 --- a/weed/s3api/s3tables/iceberg_layout_test.go +++ b/weed/s3api/s3tables/iceberg_layout_test.go @@ -100,3 +100,37 @@ func TestIcebergLayoutValidator_AcceptsRealWorldDataFiles(t *testing.T) { }) } } + +// A Lance dataset writes fragments into data/ and keeps its own bookkeeping in +// underscore-prefixed directories. The validator runs on the S3 door, where the +// table's format is not in hand, so it has to admit both formats' layouts. +func TestValidateFilePathAcceptsLanceLayout(t *testing.T) { + v := NewIcebergLayoutValidator() + allowed := []string{ + "data/01111110101110001101011164cadc43919eace9c608107bd9.lance", + "_versions/1.manifest", + "_versions/9223372036854775806.manifest", + "_transactions/0-ddb27ab7-2e5c-42c7-b4bf-265d8a3ff636.txn", + "_indices/85814508-ed9a-41f2-b939-2050bb7a0ed5-fts/index.idx", + "_deletions/_deletions-1.arrow", + ".lance-reserved", + ".lance-deregistered", + } + for _, path := range allowed { + if err := v.ValidateFilePath(path); err != nil { + t.Errorf("ValidateFilePath(%q) = %v, want nil", path, err) + } + } + + rejected := []string{ + "_versions/../../escape", + "_versions//empty", + "notadir/file.lance", + "random.txt", + } + for _, path := range rejected { + if err := v.ValidateFilePath(path); err == nil { + t.Errorf("ValidateFilePath(%q) = nil, want an error", path) + } + } +} diff --git a/weed/s3api/s3tables/s3tablestest/memfiler.go b/weed/s3api/s3tables/s3tablestest/memfiler.go new file mode 100644 index 000000000..1102439a8 --- /dev/null +++ b/weed/s3api/s3tables/s3tablestest/memfiler.go @@ -0,0 +1,234 @@ +// Package s3tablestest provides an in-memory filer for driving S3 Tables and +// Lance namespace operations end-to-end without a live cluster. +package s3tablestest + +import ( + "bytes" + "context" + "net" + "path" + "sort" + "strings" + "sync" + "testing" + "time" + + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/status" +) + +// MemFiler is an in-memory filer used to drive Manager operations +// end-to-end without a live cluster. +type MemFiler struct { + filer_pb.UnimplementedSeaweedFilerServer + // mu guards entries. The real filer serves concurrent RPCs, and a test that + // races writers against each other - the point of an exclusive create - hits + // this map from several goroutines at once. + mu sync.RWMutex + entries map[string]map[string]*filer_pb.Entry // dir -> name -> entry + Client filer_pb.SeaweedFilerClient + // BeforeUpdate runs once, at the start of the next UpdateEntry, so a test + // can land a competing write in a handler's read-to-write window. + BeforeUpdate func() +} + +func newMemFiler() *MemFiler { + return &MemFiler{entries: make(map[string]map[string]*filer_pb.Entry)} +} + +func (f *MemFiler) Get(dir, name string) *filer_pb.Entry { + f.mu.RLock() + defer f.mu.RUnlock() + if d, ok := f.entries[dir]; ok { + return d[name] + } + return nil +} + +func (f *MemFiler) Put(dir, name string, extended map[string][]byte) { + f.mu.Lock() + defer f.mu.Unlock() + if _, ok := f.entries[dir]; !ok { + f.entries[dir] = make(map[string]*filer_pb.Entry) + } + f.entries[dir][name] = &filer_pb.Entry{Name: name, IsDirectory: true, Extended: extended} +} + +// PutFile adds a file entry with an explicit modification time, which callers +// that age entries out (orphan cleanup, expiry) need in order to see them. +func (f *MemFiler) PutFile(dir, name string, mtime time.Time) { + f.mu.Lock() + defer f.mu.Unlock() + if _, ok := f.entries[dir]; !ok { + f.entries[dir] = make(map[string]*filer_pb.Entry) + } + f.entries[dir][name] = &filer_pb.Entry{ + Name: name, + Attributes: &filer_pb.FuseAttributes{Mtime: mtime.Unix(), Crtime: mtime.Unix()}, + } +} + +func (f *MemFiler) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) { + if e := f.Get(req.Directory, req.Name); e != nil { + return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil + } + // Carry the sentinel text so filer_pb.LookupEntry maps it to ErrNotFound. + return nil, status.Errorf(codes.NotFound, "%s: %s/%s", filer_pb.ErrNotFound.Error(), req.Directory, req.Name) +} + +// ListEntries honours prefix, start-from and limit the way the real filer does. +// A harness that ignores them makes a paginating caller re-read the first page +// forever, which looks like duplicated entries rather than a broken listing. +func (f *MemFiler) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error { + f.mu.RLock() + d, ok := f.entries[req.Directory] + names := make([]string, 0, len(d)) + snapshot := make(map[string]*filer_pb.Entry, len(d)) + for name, entry := range d { + names = append(names, name) + snapshot[name] = entry + } + f.mu.RUnlock() + if !ok { + return nil + } + sort.Strings(names) + + sent := uint32(0) + for _, name := range names { + if req.Prefix != "" && !strings.HasPrefix(name, req.Prefix) { + continue + } + if req.StartFromFileName != "" { + if name < req.StartFromFileName { + continue + } + if name == req.StartFromFileName && !req.InclusiveStartFrom { + continue + } + } + if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: snapshot[name]}); err != nil { + return err + } + sent++ + if req.Limit > 0 && sent >= req.Limit { + return nil + } + } + return nil +} + +func (f *MemFiler) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest) (*filer_pb.CreateEntryResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + if _, ok := f.entries[req.Directory]; !ok { + f.entries[req.Directory] = make(map[string]*filer_pb.Entry) + } + // O_EXCL is the filer's put-if-not-exists. Ignoring it here would let a test + // that races two writers see both of them win. + if _, exists := f.entries[req.Directory][req.Entry.Name]; exists && req.OExcl { + return &filer_pb.CreateEntryResponse{ErrorCode: filer_pb.FilerError_ENTRY_ALREADY_EXISTS}, nil + } + f.entries[req.Directory][req.Entry.Name] = req.Entry + return &filer_pb.CreateEntryResponse{}, nil +} + +func (f *MemFiler) UpdateEntry(_ context.Context, req *filer_pb.UpdateEntryRequest) (*filer_pb.UpdateEntryResponse, error) { + // The hook runs before the lock is taken: its whole purpose is to land a + // competing write in the read-to-write window, and that write needs the lock. + if hook := f.BeforeUpdate; hook != nil { + f.BeforeUpdate = nil + hook() + } + f.mu.Lock() + defer f.mu.Unlock() + // The real filer validates ExpectedExtended under the per-path lock; without + // it here a lost update would look like a success. + for key, expected := range req.ExpectedExtended { + var actual []byte + if d, ok := f.entries[req.Directory]; ok { + if existing := d[req.Entry.Name]; existing != nil { + actual = existing.Extended[key] + } + } + if !bytes.Equal(actual, expected) { + return nil, status.Errorf(codes.FailedPrecondition, "extended attribute %q changed", key) + } + } + if _, ok := f.entries[req.Directory]; !ok { + f.entries[req.Directory] = make(map[string]*filer_pb.Entry) + } + f.entries[req.Directory][req.Entry.Name] = req.Entry + return &filer_pb.UpdateEntryResponse{}, nil +} + +func (f *MemFiler) DeleteEntry(_ context.Context, req *filer_pb.DeleteEntryRequest) (*filer_pb.DeleteEntryResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + if d, ok := f.entries[req.Directory]; ok { + delete(d, req.Name) + } + // Honor recursive data deletion so a regression that wipes the table directory + // also drops its metadata/ and data/ children (the data-loss this guards against). + if req.IsRecursive && req.IsDeleteData { + child := path.Join(req.Directory, req.Name) + for dir := range f.entries { + if dir == child || strings.HasPrefix(dir, child+"/") { + delete(f.entries, dir) + } + } + } + return &filer_pb.DeleteEntryResponse{}, nil +} + +// GetFilerConfiguration answers with the defaults, so operations that resolve +// the buckets directory before touching an entry work against this filer. +func (f *MemFiler) GetFilerConfiguration(_ context.Context, _ *filer_pb.GetFilerConfigurationRequest) (*filer_pb.GetFilerConfigurationResponse, error) { + return &filer_pb.GetFilerConfigurationResponse{DirBuckets: s3_constants.DefaultBucketsPath}, nil +} + +func (f *MemFiler) Ping(_ context.Context, _ *filer_pb.PingRequest) (*filer_pb.PingResponse, error) { + now := time.Now().UnixNano() + return &filer_pb.PingResponse{StartTimeNs: now, RemoteTimeNs: now, StopTimeNs: now}, nil +} + +func Start(t *testing.T) *MemFiler { + t.Helper() + fs := newMemFiler() + + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("start filer: %v", err) + } + + server := grpc.NewServer() + filer_pb.RegisterSeaweedFilerServer(server, fs) + go func() { _ = server.Serve(listener) }() + t.Cleanup(server.GracefulStop) + + conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + t.Fatalf("start filer: %v", err) + } + t.Cleanup(func() { _ = conn.Close() }) + + fs.Client = filer_pb.NewSeaweedFilerClient(conn) + deadline := time.Now().Add(5 * time.Second) + for { + pingCtx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + _, err := fs.Client.Ping(pingCtx, &filer_pb.PingRequest{}) + cancel() + if err == nil { + break + } + if time.Now().After(deadline) { + t.Fatalf("filer not ready: %v", err) + } + time.Sleep(10 * time.Millisecond) + } + return fs +} diff --git a/weed/s3api/s3tables/types.go b/weed/s3api/s3tables/types.go index 767c1f1be..748abf62c 100644 --- a/weed/s3api/s3tables/types.go +++ b/weed/s3api/s3tables/types.go @@ -2,6 +2,7 @@ package s3tables import ( "encoding/json" + "strings" "time" ) @@ -12,11 +13,17 @@ type TableBucket struct { Name string `json:"name"` OwnerAccountID string `json:"ownerAccountId"` CreatedAt time.Time `json:"createdAt"` + Format string `json:"format,omitempty"` } type CreateTableBucketRequest struct { Name string `json:"name"` Tags map[string]string `json:"tags,omitempty"` + // Format is the table format this bucket holds. A bucket is a catalog and a + // catalog serves one protocol, so declaring it here is what lets a caller be + // told where to connect. Empty means ICEBERG, which is what AWS S3 Tables + // serves and therefore what an SDK that has never heard of this field means. + Format string `json:"format,omitempty"` } type CreateTableBucketResponse struct { @@ -32,6 +39,9 @@ type GetTableBucketResponse struct { Name string `json:"name"` OwnerAccountID string `json:"ownerAccountId"` CreatedAt time.Time `json:"createdAt"` + // Format is empty for a bucket created before formats were declared. Such a + // bucket accepts any format, which is what it did when it was made. + Format string `json:"format,omitempty"` } type ListTableBucketsRequest struct { @@ -44,6 +54,7 @@ type TableBucketSummary struct { ARN string `json:"arn"` Name string `json:"name"` CreatedAt time.Time `json:"createdAt"` + Format string `json:"format,omitempty"` } type ListTableBucketsResponse struct { @@ -235,9 +246,12 @@ type ListTablesRequest struct { } type TableSummary struct { - Name string `json:"name"` - TableARN string `json:"tableARN"` - Namespace []string `json:"namespace"` + Name string `json:"name"` + TableARN string `json:"tableARN"` + Namespace []string `json:"namespace"` + // Format lets a caller tell an Iceberg table from a catalog-only one without + // a GetTable per row. AWS omits it; listing a mixed catalog needs it. + Format string `json:"format,omitempty"` CreatedAt time.Time `json:"createdAt"` ModifiedAt time.Time `json:"modifiedAt"` MetadataLocation string `json:"metadataLocation,omitempty"` @@ -562,6 +576,36 @@ func (e *S3TablesError) Error() string { return e.Message } +// Table formats a catalog entry may declare. +// +// ICEBERG tables carry metadata the catalog maintains and the maintenance +// worker rewrites. LANCE is catalog-only: the entry records a name and the +// dataset root in MetadataLocation, and the Lance client owns every byte under +// it. Nothing in this package interprets a catalog-only table's files. +const ( + FormatIceberg = "ICEBERG" + FormatLance = "LANCE" +) + +// IsCatalogOnlyFormat reports whether the catalog only records where a table of +// this format lives, without understanding its files. +func IsCatalogOnlyFormat(format string) bool { + return format == FormatLance +} + +// NormalizeFormat folds a caller's spelling onto the canonical one and reports +// whether it names a format this catalog serves. +func NormalizeFormat(format string) (string, bool) { + switch strings.ToUpper(strings.TrimSpace(format)) { + case FormatIceberg: + return FormatIceberg, true + case FormatLance: + return FormatLance, true + default: + return "", false + } +} + // Error codes const ( ErrCodeBucketAlreadyExists = "BucketAlreadyExists" diff --git a/weed/s3api/s3tables/utils.go b/weed/s3api/s3tables/utils.go index 302667e0d..a0d3a6e93 100644 --- a/weed/s3api/s3tables/utils.go +++ b/weed/s3api/s3tables/utils.go @@ -142,6 +142,7 @@ type tableBucketMetadata struct { Name string `json:"name"` CreatedAt time.Time `json:"createdAt"` OwnerAccountID string `json:"ownerAccountId"` + Format string `json:"format,omitempty"` } // namespaceMetadata stores metadata for a namespace @@ -175,10 +176,10 @@ func IsTableBucketEntry(entry *filer_pb.Entry) bool { return ok } -// entryType returns the entry-type marker for a catalog entry. Tables and views +// EntryType returns the entry-type marker for a catalog entry. Tables and views // share the same on-disk layout; the marker distinguishes them. An absent marker // means table for back-compat. -func entryType(extended map[string][]byte) string { +func EntryType(extended map[string][]byte) string { if extended == nil { return EntryTypeTable } diff --git a/weed/shell/command_s3tables_bucket.go b/weed/shell/command_s3tables_bucket.go index d24a80599..26153440b 100644 --- a/weed/shell/command_s3tables_bucket.go +++ b/weed/shell/command_s3tables_bucket.go @@ -67,6 +67,7 @@ func (c *commandS3TablesBucket) Do(args []string, commandEnv *CommandEnv, writer tags := cmd.String("tags", "", "comma separated tags key=value") policyFile := cmd.String("file", "", "policy file (json)") account := cmd.String("account", "", "owner account id") + format := cmd.String("format", "", "table format the bucket holds: ICEBERG (default) or LANCE") if err := cmd.Parse(args); err != nil { return err @@ -95,6 +96,13 @@ func (c *commandS3TablesBucket) Do(args []string, commandEnv *CommandEnv, writer return err } req := &s3tables.CreateTableBucketRequest{Name: *name} + if *format != "" { + normalized, ok := s3tables.NormalizeFormat(*format) + if !ok { + return fmt.Errorf("unsupported format %q", *format) + } + req.Format = normalized + } if *tags != "" { parsed, err := parseS3TablesTags(*tags) if err != nil { diff --git a/weed/worker/tasks/iceberg/detection.go b/weed/worker/tasks/iceberg/detection.go index 5f3c1bfed..d352e204e 100644 --- a/weed/worker/tasks/iceberg/detection.go +++ b/weed/worker/tasks/iceberg/detection.go @@ -3,6 +3,7 @@ package iceberg import ( "bytes" "context" + "errors" "fmt" "path" "strings" @@ -128,7 +129,15 @@ func (h *Handler) scanTablesForMaintenance( tablePath := path.Join(nsName, tblName) state, err := parseTableMetadataEnvelope(metadataBytes, bucketName, tablePath) if err != nil { - glog.V(2).Infof("iceberg maintenance: skipping %s/%s/%s: cannot parse iceberg metadata: %v", bucketName, nsName, tblName, err) + if errors.Is(err, errForeignFormat) { + glog.V(3).Infof("iceberg maintenance: skipping %s/%s/%s: %v", bucketName, nsName, tblName, err) + } else { + glog.V(2).Infof("iceberg maintenance: skipping %s/%s/%s: cannot parse iceberg metadata: %v", bucketName, nsName, tblName, err) + } + continue + } + if !isIcebergTableEntry(tableEntry.Extended, state.Metadata) { + glog.V(2).Infof("iceberg maintenance: skipping %s/%s/%s: not an iceberg table", bucketName, nsName, tblName) continue } diff --git a/weed/worker/tasks/iceberg/exec_test.go b/weed/worker/tasks/iceberg/exec_test.go index cb2dd9eab..0e7e293c9 100644 --- a/weed/worker/tasks/iceberg/exec_test.go +++ b/weed/worker/tasks/iceberg/exec_test.go @@ -329,7 +329,7 @@ func (ts tableSetup) fileRef(elem ...string) string { func populateTable(t *testing.T, fs *fakeFilerServer, setup tableSetup) table.Metadata { t.Helper() - meta := buildTestMetadata(t, setup.Snapshots, setup.Refs, setup.Age) + meta := buildTestMetadata(t, setup.Snapshots, setup.Refs, setup.Age, nil) fullMetadataJSON, err := json.Marshal(meta) if err != nil { t.Fatalf("marshal metadata: %v", err) diff --git a/weed/worker/tasks/iceberg/filer_io.go b/weed/worker/tasks/iceberg/filer_io.go index 75fae1db0..93d914c70 100644 --- a/weed/worker/tasks/iceberg/filer_io.go +++ b/weed/worker/tasks/iceberg/filer_io.go @@ -147,7 +147,17 @@ func loadCurrentMetadata(ctx context.Context, client filer_pb.SeaweedFilerClient return nil, fmt.Errorf("no metadata xattr on table entry %s/%s", dir, name) } - return parseTableMetadataEnvelope(metadataBytes, bucketName, tablePath) + state, err := parseTableMetadataEnvelope(metadataBytes, bucketName, tablePath) + if err != nil { + return nil, err + } + // Refuse to rewrite an entry that only looks like an Iceberg table. Every + // operation here rewrites or deletes files under the table location, and a + // foreign format's files are unreferenced by Iceberg metadata by definition. + if !isIcebergTableEntry(resp.Entry.Extended, state.Metadata) { + return nil, fmt.Errorf("entry %s/%s is not an iceberg table", dir, name) + } + return state, nil } // loadFileByIcebergPath loads a file from the filer given an Iceberg-style path. diff --git a/weed/worker/tasks/iceberg/foreign_format_scan_test.go b/weed/worker/tasks/iceberg/foreign_format_scan_test.go new file mode 100644 index 000000000..a9b103197 --- /dev/null +++ b/weed/worker/tasks/iceberg/foreign_format_scan_test.go @@ -0,0 +1,113 @@ +package iceberg + +import ( + "context" + "encoding/json" + "path" + "testing" + "time" + + "github.com/apache/iceberg-go" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest" +) + +// seedAdapterRegisteredLanceTable builds what the Lance namespace's Iceberg REST +// adapter leaves behind: a real Iceberg table with a placeholder schema and +// table_type=lance, whose directory holds a Lance dataset rather than Iceberg +// data files. +func seedAdapterRegisteredLanceTable(t *testing.T, properties iceberg.Properties) (*s3tablestest.MemFiler, string) { + t.Helper() + const bucket, namespace, table = "vectors", "ml", "embeddings" + + filer := s3tablestest.Start(t) + filer.Put(s3tables.TablesPath, bucket, map[string][]byte{s3tables.ExtendedKeyTableBucket: []byte("{}")}) + + nsMeta, err := json.Marshal(map[string]any{"namespace": []string{namespace}}) + if err != nil { + t.Fatalf("marshal namespace metadata: %v", err) + } + filer.Put(s3tables.GetTableBucketPath(bucket), namespace, map[string][]byte{s3tables.ExtendedKeyMetadata: nsMeta}) + + full, err := json.Marshal(buildTestMetadata(t, nil, nil, 0, properties)) + if err != nil { + t.Fatalf("marshal iceberg metadata: %v", err) + } + envelope, err := json.Marshal(map[string]any{ + "metadataVersion": 1, + "metadataLocation": "s3://" + bucket + "/" + namespace + "/" + table + "/metadata/v1.metadata.json", + "metadata": map[string]any{"fullMetadata": json.RawMessage(full)}, + }) + if err != nil { + t.Fatalf("marshal envelope: %v", err) + } + filer.Put(s3tables.GetNamespacePath(bucket, namespace), table, + map[string][]byte{s3tables.ExtendedKeyMetadata: envelope}) + + // The Lance dataset. Its fragments live in data/, the same directory the + // orphan cleaner walks, and the Iceberg metadata references none of them. + tablePath := s3tables.GetTablePath(bucket, namespace, table) + old := time.Now().Add(-30 * 24 * time.Hour) + filer.Put(tablePath, "data", nil) + filer.PutFile(path.Join(tablePath, "data"), "01111110101110001101011164cadc43919eace9c608107bd9.lance", old) + filer.Put(tablePath, "_versions", nil) + filer.PutFile(path.Join(tablePath, "_versions"), "1.manifest", old) + filer.Put(tablePath, "metadata", nil) + filer.PutFile(path.Join(tablePath, "metadata"), "v1.metadata.json", old) + + return filer, path.Join(namespace, table) +} + +// The hazard this guards against, stated as a test: every fragment of a Lance +// dataset is unreferenced by the Iceberg metadata sitting beside it, so orphan +// cleanup would delete the whole dataset. +func TestOrphanCleanupWouldDeleteAnAdapterRegisteredLanceDataset(t *testing.T) { + filer, tablePath := seedAdapterRegisteredLanceTable(t, iceberg.Properties{tableTypeProperty: "lance"}) + + meta := buildTestMetadata(t, nil, nil, 0, iceberg.Properties{tableTypeProperty: "lance"}) + candidates, err := collectOrphanCandidates(context.Background(), filer.Client, "vectors", tablePath, + meta, "v1.metadata.json", defaultOrphanOlderThanHours) + if err != nil { + t.Fatalf("collect orphan candidates: %v", err) + } + + var doomed []string + for _, c := range candidates { + doomed = append(doomed, path.Join(c.Dir, c.Entry.Name)) + } + if len(doomed) == 0 { + t.Fatal("expected the Lance fragments to look like orphans; without that this guard is pointless") + } + for _, name := range doomed { + if path.Ext(name) == ".lance" { + return + } + } + t.Fatalf("expected a .lance fragment among the orphan candidates, got %v", doomed) +} + +// And the guard that stops it: the scan never reaches the table. +func TestScanSkipsAnAdapterRegisteredLanceTable(t *testing.T) { + handler := &Handler{} + config := normalizeDetectionConfig(Config{Operations: "all"}) + + filer, _ := seedAdapterRegisteredLanceTable(t, iceberg.Properties{tableTypeProperty: "lance"}) + tables, err := handler.scanTablesForMaintenance(context.Background(), filer.Client, config, "", "", "", 0) + if err != nil { + t.Fatalf("scan: %v", err) + } + if len(tables) != 0 { + t.Fatalf("scan picked up a lance table: %v", tables) + } + + // An ordinary Iceberg table in the same shape is still scanned, so the guard + // is not simply skipping everything. + filer, _ = seedAdapterRegisteredLanceTable(t, nil) + tables, err = handler.scanTablesForMaintenance(context.Background(), filer.Client, config, "", "", "", 0) + if err != nil { + t.Fatalf("scan: %v", err) + } + if len(tables) != 1 { + t.Fatalf("scan skipped an iceberg table: %v", tables) + } +} diff --git a/weed/worker/tasks/iceberg/foreign_format_test.go b/weed/worker/tasks/iceberg/foreign_format_test.go new file mode 100644 index 000000000..65792d88b --- /dev/null +++ b/weed/worker/tasks/iceberg/foreign_format_test.go @@ -0,0 +1,73 @@ +package iceberg + +import ( + "errors" + "testing" + + "github.com/apache/iceberg-go" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3tables" +) + +// A Lance dataset registered through the Lance namespace's Iceberg REST adapter +// arrives as an Iceberg table with a placeholder schema and table_type=lance, +// and keeps its fragments under data/ where the orphan cleaner walks. Every +// fragment is unreferenced by the Iceberg metadata, so a maintenance pass would +// delete the dataset. +func TestIsIcebergTableEntry(t *testing.T) { + cases := []struct { + name string + extended map[string][]byte + properties iceberg.Properties + want bool + }{ + { + name: "plain table", + want: true, + }, + { + name: "iceberg table_type is honoured case-insensitively", + properties: iceberg.Properties{tableTypeProperty: "ICEBERG"}, + want: true, + }, + { + name: "lance table registered through the iceberg adapter", + properties: iceberg.Properties{tableTypeProperty: "lance"}, + want: false, + }, + { + name: "view", + extended: map[string][]byte{s3tables.ExtendedKeyEntryType: []byte(s3tables.EntryTypeView)}, + want: false, + }, + { + name: "explicit table marker", + extended: map[string][]byte{s3tables.ExtendedKeyEntryType: []byte(s3tables.EntryTypeTable)}, + want: true, + }, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + meta := buildTestMetadata(t, nil, nil, 0, c.properties) + if got := isIcebergTableEntry(c.extended, meta); got != c.want { + t.Fatalf("isIcebergTableEntry() = %v, want %v", got, c.want) + } + }) + } +} + +// A table the namespace created as LANCE carries no Iceberg metadata at all. +// Without reading the catalog's own format field the parse below just fails, +// and the table gets skipped as though its metadata were damaged. +func TestParseTableMetadataEnvelopeRejectsForeignFormats(t *testing.T) { + lance := []byte(`{"name":"vectors","namespace":"ml","format":"LANCE","metadataLocation":"s3://b/ml/vectors"}`) + if _, err := parseTableMetadataEnvelope(lance, "b", "ml/vectors"); !errors.Is(err, errForeignFormat) { + t.Fatalf("parse of a LANCE entry = %v, want errForeignFormat", err) + } + + // An entry with no format recorded predates the field and is still Iceberg's. + legacy := []byte(`{"metadataVersion":1}`) + if _, err := parseTableMetadataEnvelope(legacy, "b", "ml/t"); err == nil || errors.Is(err, errForeignFormat) { + t.Fatalf("parse of a legacy entry = %v, want a plain parse failure", err) + } +} diff --git a/weed/worker/tasks/iceberg/handler_test.go b/weed/worker/tasks/iceberg/handler_test.go index f2b4818fe..e25bfc1aa 100644 --- a/weed/worker/tasks/iceberg/handler_test.go +++ b/weed/worker/tasks/iceberg/handler_test.go @@ -121,7 +121,7 @@ func TestNeedsMaintenanceNoSnapshots(t *testing.T) { MaxSnapshotsToKeep: 2, } - meta := buildTestMetadata(t, nil, nil, 0) + meta := buildTestMetadata(t, nil, nil, 0, nil) if needsMaintenance(meta, config) { t.Error("expected no maintenance for table with no snapshots") } @@ -139,7 +139,7 @@ func TestNeedsMaintenanceExceedsMaxSnapshots(t *testing.T) { {SnapshotID: 2, TimestampMs: now + 1, ManifestList: "metadata/snap-2.avro"}, {SnapshotID: 3, TimestampMs: now + 2, ManifestList: "metadata/snap-3.avro"}, } - meta := buildTestMetadata(t, snapshots, nil, 48*time.Hour) + meta := buildTestMetadata(t, snapshots, nil, 48*time.Hour, nil) if !needsMaintenance(meta, config) { t.Error("expected maintenance for table exceeding max snapshots") } @@ -159,7 +159,7 @@ func TestNeedsMaintenanceExceedsMaxSnapshotsWithinRetention(t *testing.T) { {SnapshotID: 2, TimestampMs: now + 1, ManifestList: "metadata/snap-2.avro"}, {SnapshotID: 3, TimestampMs: now + 2, ManifestList: "metadata/snap-3.avro"}, } - if needsMaintenance(buildTestMetadata(t, snapshots, nil, 0), config) { + if needsMaintenance(buildTestMetadata(t, snapshots, nil, 0, nil), config) { t.Error("expected no maintenance while every snapshot is inside the retention window") } } @@ -179,10 +179,10 @@ func TestNeedsMaintenanceSkipsRefPinnedSnapshots(t *testing.T) { refs := map[string]table.SnapshotRef{ "release": {SnapshotID: 1, SnapshotRefType: table.TagRef}, } - if needsMaintenance(buildTestMetadata(t, snapshots, refs, 0), config) { + if needsMaintenance(buildTestMetadata(t, snapshots, refs, 0, nil), config) { t.Error("expected no maintenance when the only old snapshot is tagged") } - if !needsMaintenance(buildTestMetadata(t, snapshots, nil, 0), config) { + if !needsMaintenance(buildTestMetadata(t, snapshots, nil, 0, nil), config) { t.Error("expected maintenance for the same table without the tag") } } @@ -197,7 +197,7 @@ func TestNeedsMaintenanceWithinLimits(t *testing.T) { snapshots := []table.Snapshot{ {SnapshotID: 1, TimestampMs: now, ManifestList: "metadata/snap-1.avro"}, } - meta := buildTestMetadata(t, snapshots, nil, 0) + meta := buildTestMetadata(t, snapshots, nil, 0, nil) if needsMaintenance(meta, config) { t.Error("expected no maintenance for table within limits") } @@ -215,7 +215,7 @@ func TestNeedsMaintenanceOldSnapshot(t *testing.T) { {SnapshotID: 1, TimestampMs: now, ManifestList: "metadata/snap-1.avro"}, {SnapshotID: 2, TimestampMs: now + 1, ManifestList: "metadata/snap-2.avro"}, } - meta := buildTestMetadata(t, snapshots, nil, 0) + meta := buildTestMetadata(t, snapshots, nil, 0, nil) if !needsMaintenance(meta, config) { t.Error("expected maintenance for table with expired snapshot") } @@ -233,7 +233,7 @@ func TestNeedsMaintenanceSingleSnapshot(t *testing.T) { snapshots := []table.Snapshot{ {SnapshotID: 1, TimestampMs: now, ManifestList: "metadata/snap-1.avro"}, } - if needsMaintenance(buildTestMetadata(t, snapshots, nil, 0), config) { + if needsMaintenance(buildTestMetadata(t, snapshots, nil, 0, nil), config) { t.Error("expected no maintenance for a table with only the current snapshot") } } @@ -278,7 +278,7 @@ func TestBuildMaintenanceProposal(t *testing.T) { {SnapshotID: 1, TimestampMs: now}, {SnapshotID: 2, TimestampMs: now + 1}, } - meta := buildTestMetadata(t, snapshots, nil, 0) + meta := buildTestMetadata(t, snapshots, nil, 0, nil) info := tableInfo{ BucketName: "my-bucket", @@ -1476,11 +1476,11 @@ func TestExecuteNilRequest(t *testing.T) { // are genuinely past a retention window - iceberg-go refuses to add a snapshot // stamped more than a minute before the metadata's last-updated time, so the // shift has to happen after the build. -func buildTestMetadata(t *testing.T, snapshots []table.Snapshot, refs map[string]table.SnapshotRef, age time.Duration) table.Metadata { +func buildTestMetadata(t *testing.T, snapshots []table.Snapshot, refs map[string]table.SnapshotRef, age time.Duration, properties iceberg.Properties) table.Metadata { t.Helper() schema := newTestSchema() - meta, err := table.NewMetadata(schema, iceberg.UnpartitionedSpec, table.UnsortedSortOrder, "s3://test-bucket/test-table", nil) + meta, err := table.NewMetadata(schema, iceberg.UnpartitionedSpec, table.UnsortedSortOrder, "s3://test-bucket/test-table", properties) if err != nil { t.Fatalf("failed to create test metadata: %v", err) } diff --git a/weed/worker/tasks/iceberg/planning_index.go b/weed/worker/tasks/iceberg/planning_index.go index 278bc691f..885e2817e 100644 --- a/weed/worker/tasks/iceberg/planning_index.go +++ b/weed/worker/tasks/iceberg/planning_index.go @@ -3,8 +3,10 @@ package iceberg import ( "context" "encoding/json" + "errors" "fmt" "path" + "strings" "time" "github.com/apache/iceberg-go" @@ -36,6 +38,10 @@ type planningIndexRewriteManifests struct { } type tableMetadataEnvelope struct { + // Format is the catalog's own record of what the table is. A native Lance + // table has no Iceberg metadata at all, so without this the parse below + // fails and the table is skipped as if it were corrupt. + Format string `json:"format"` MetadataVersion int `json:"metadataVersion"` MetadataLocation string `json:"metadataLocation,omitempty"` Metadata *struct { @@ -54,11 +60,37 @@ type tableState struct { PlanningIndex *planningIndex } +// tableTypeProperty is the Hive/Glue-style format marker. Catalogs that have no +// native concept of a non-Iceberg table register one as an Iceberg table with a +// placeholder schema and set this property instead; the Lance namespace's +// Iceberg REST adapter writes table_type=lance. +const tableTypeProperty = "table_type" + +// errForeignFormat means the catalog entry belongs to a format this worker does +// not maintain. It is a normal outcome of scanning a mixed catalog, not damage. +var errForeignFormat = errors.New("not an iceberg table") + +// isIcebergTableEntry reports whether a catalog entry is an Iceberg table this +// worker may rewrite. Views share the entry shape, and a foreign format +// registered through the catalog shares the location but not the file layout - +// a Lance dataset keeps its fragments under data/, where every one of them is +// unreferenced by the Iceberg metadata and so looks like an orphan. +func isIcebergTableEntry(extended map[string][]byte, meta table.Metadata) bool { + if s3tables.EntryType(extended) != s3tables.EntryTypeTable { + return false + } + tableType, ok := meta.Properties()[tableTypeProperty] + return !ok || strings.EqualFold(tableType, "iceberg") +} + func parseTableMetadataEnvelope(metadataBytes []byte, bucketName, tablePath string) (*tableState, error) { var envelope tableMetadataEnvelope if err := json.Unmarshal(metadataBytes, &envelope); err != nil { return nil, fmt.Errorf("parse metadata xattr: %w", err) } + if envelope.Format != "" && envelope.Format != s3tables.FormatIceberg { + return nil, fmt.Errorf("%w: %s", errForeignFormat, envelope.Format) + } if envelope.Metadata == nil || len(envelope.Metadata.FullMetadata) == 0 { return nil, fmt.Errorf("no fullMetadata in table xattr") } diff --git a/weed/worker/tasks/iceberg/table_config_test.go b/weed/worker/tasks/iceberg/table_config_test.go index af46d4825..a96e726be 100644 --- a/weed/worker/tasks/iceberg/table_config_test.go +++ b/weed/worker/tasks/iceberg/table_config_test.go @@ -388,7 +388,7 @@ func TestResolveCompactionRewritePlanAuto(t *testing.T) { cfg := baseTestConfig() cfg.RewriteStrategy = rewriteStrategyAuto - unsorted := buildTestMetadata(t, nil, nil, 0) + unsorted := buildTestMetadata(t, nil, nil, 0, nil) plan, err := resolveCompactionRewritePlan(cfg, unsorted) if err != nil { t.Fatalf("auto must not fail on an unsorted table: %v", err)