mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-20 14:17:07 +00:00
Lance catalog, and a Rust plugin worker to maintain it (#10841)
* iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
This commit is contained in:
Chris Lu
@@ -0,0 +1,696 @@
|
||||
# Lance Catalog for SeaweedFS
|
||||
|
||||
A second catalog surface next to the Iceberg REST catalog, speaking the Lance Namespace
|
||||
REST spec, over the same table buckets and the same filer.
|
||||
|
||||
## Why
|
||||
|
||||
Gravitino 1.1 added a Lance REST service and 1.3 ships it as a standalone server; Lakekeeper
|
||||
added Lance in the same window by a completely different route. That is the useful signal:
|
||||
two unrelated catalogs decided independently that Lance had to be first-class, not a niche.
|
||||
The client side is already there — `lance-spark` (`LanceNamespaceSparkCatalog`
|
||||
with `impl=rest`), `lance-ray`, and the generated Python/Java/Rust clients all talk the same
|
||||
OpenAPI. Implementing the spec means those engines work against SeaweedFS with no
|
||||
SeaweedFS-specific code on the client.
|
||||
|
||||
The second reason is that Gravitino's own documentation names the gap it cannot close:
|
||||
DuckDB, pandas and DataFusion "do not support Lance REST natively yet" and have to fetch a
|
||||
location from the catalog and then open the dataset directly. Gravitino cannot help there,
|
||||
because it does not own the storage. SeaweedFS does. That is the whole design opportunity
|
||||
below.
|
||||
|
||||
## Prior art: three families
|
||||
|
||||
Upstream lists twelve catalog implementations, and they fall into three shapes. Knowing
|
||||
which one we are building matters more than any individual API decision.
|
||||
|
||||
**1. Storage-native, no service.** The Lance Directory Catalog. V1 is a directory listing
|
||||
where every `<name>.lance/` child of a prefix is a table; V2 adds a `__manifest` table —
|
||||
itself a Lance table — holding `object_id`/`object_type`/`location` rows, with nested
|
||||
namespaces, hash-prefixed table directories, and optional managed versioning. No server, no
|
||||
credentials, no governance. This is the floor every other implementation has to beat.
|
||||
|
||||
**2. Protocol-native server.** Someone implements the Lance Namespace REST OpenAPI and
|
||||
clients connect with `impl=rest`. Gravitino is the only one of the twelve that does this,
|
||||
and it is what this design proposes.
|
||||
|
||||
**3. Client-side adapters onto an existing catalog.** Nine of the twelve. The Lance client
|
||||
translates namespace operations into whatever the backing catalog already speaks: Apache
|
||||
Polaris, Unity Catalog, AWS Glue, Hive Metastore v2 and v3, Google BigLake, Dataproc,
|
||||
Microsoft OneLake — and Apache Iceberg REST. Two flavors:
|
||||
|
||||
- Catalogs with a real non-Iceberg table concept mark the format directly. Polaris uses its
|
||||
Generic Table API with `format = lance`; Unity uses an `EXTERNAL` table with
|
||||
`table_type=lance` in properties and the path in `storage_location`; Glue uses
|
||||
`EXTERNAL_TABLE` plus `table_type=lance` in `Parameters`, path in
|
||||
`StorageDescriptor.Location`.
|
||||
- Catalogs with no such concept fake one. The Iceberg REST adapter registers **a regular
|
||||
Iceberg table with a dummy schema — a single nullable string column named `dummy`** —
|
||||
carrying the property `table_type=lance`, and treats the Iceberg table location as the
|
||||
Lance dataset root.
|
||||
|
||||
Every adapter in family 3 lands in the same place: `DeclareTable`/`ListTables`/
|
||||
`DescribeTable`/`DeregisterTable` only, `DropNamespace` in RESTRICT mode only,
|
||||
`load_detailed_metadata=false` only, and `managed_versioning=false`. They are a name-to-
|
||||
location map and nothing more.
|
||||
|
||||
Lakekeeper is the instructive outlier. It has the same generic-table concept Polaris has,
|
||||
but no upstream adapter exists for it — there is no `lance-namespace` reference anywhere in
|
||||
its repository and no page for it in the supported-catalogs list. So Polaris's generic tables
|
||||
are reachable from a stock Lance client and Lakekeeper's are not, despite being the same
|
||||
idea. Shipping the concept is not the same as shipping the integration.
|
||||
|
||||
## Gravitino and Lakekeeper: the two opposite bets
|
||||
|
||||
Both shipped Lance support in the same window and did not build the same thing.
|
||||
|
||||
**Gravitino implements the protocol.** Its `lance/` module serves the Lance Namespace REST
|
||||
spec on its own port (`:9101/lance`), so stock `lance-spark` and `lance-ray` connect with
|
||||
`impl=rest` and no vendor-specific client. The cost is governance: storage credentials are
|
||||
static properties on the catalog (`lance.storage.access_key_id`, `secret_access_key`,
|
||||
`endpoint`, `region`, `allow_http`), optionally overridden per table, handed to the engine
|
||||
as-is. No STS, no expiry, no per-table scoping.
|
||||
|
||||
**Lakekeeper refuses the protocol and governs the object instead.** There is no
|
||||
`lance-namespace` anywhere in the repository; Lance arrived in 0.13.0 (2026-06-30, issue
|
||||
#1673 `Generic Table API with Lance`) as one `format` string on a Lakekeeper-native Generic
|
||||
Table API:
|
||||
|
||||
```
|
||||
POST/GET/DELETE /lakekeeper/v1/{prefix}/namespaces/{ns}/generic-tables[/{table}]
|
||||
GET /lakekeeper/v1/{prefix}/namespaces/{ns}/generic-tables/{table}/credentials
|
||||
POST /lakekeeper/v1/{prefix}/generic-tables/rename
|
||||
```
|
||||
|
||||
`format` is opaque, `schema` and `statistics` are stored but never validated, and the
|
||||
catalog writes no format-specific metadata — engines go straight to the location. In
|
||||
exchange Lance tables get everything Iceberg tables get: STS-vended prefix-scoped
|
||||
credentials, OpenFGA per-action permissions (16 actions), soft-delete with undrop, a
|
||||
protection flag, rename, pagination, and name uniqueness across Iceberg tables, views and
|
||||
generic tables in one namespace. The price is that no stock Lance client can talk to it —
|
||||
you need `pylakekeeper`, which exists mainly to translate vended credentials into
|
||||
`lance_storage_options`.
|
||||
|
||||
So: protocol fidelity and weak governance, or strong governance and client lock-in. Both
|
||||
documented their limit honestly, and it is the same limit. Lakekeeper's capability table
|
||||
says it outright — "Commit coordination: the catalog does not arbitrate writes — engines
|
||||
write directly." Gravitino does not claim it either. Neither of them coordinates a Lance
|
||||
commit, which is exactly the thing a store can do and a control plane cannot.
|
||||
|
||||
We do not have to choose. Serve the Lance protocol natively the way Gravitino does, over
|
||||
the `s3tables` entries that already carry ARNs, policies, tags and maintenance config, and
|
||||
the governance comes from the layer underneath rather than from a proprietary API on top.
|
||||
That is only available to us because we are the store, which is also what makes the third
|
||||
option — arbitrating the commit — available.
|
||||
|
||||
## We are probably already a Lance catalog, and that is a problem
|
||||
|
||||
The Iceberg REST adapter does not care whose Iceberg catalog it is talking to. It needs
|
||||
`/v1/config?warehouse=`, `/v1/{prefix}/namespaces`, `/v1/{prefix}/namespaces/{ns}/tables`
|
||||
and unit-separator (`\x1F`) multi-level namespaces. We serve all of those, and
|
||||
`parseNamespace` in `weed/s3api/iceberg/utils.go:22` already splits on `\x1F`. So a stock
|
||||
Lance client pointed at our Iceberg catalog on :8181 with the Iceberg impl should already
|
||||
create, list, describe and deregister Lance tables today, with no SeaweedFS change at all.
|
||||
|
||||
That is worth testing before writing a line of the design above, for two reasons. It is a
|
||||
free baseline — and possibly a free announcement. And it is a data-loss hazard.
|
||||
|
||||
A Lance table registered this way is an Iceberg table whose metadata references no data
|
||||
files, sitting on top of a Lance dataset that uses `data/` for its fragments — the same
|
||||
subdirectory name Iceberg uses. The maintenance worker's orphan cleaner walks exactly
|
||||
`<table>/metadata` and `<table>/data`, and deletes every file not referenced by a snapshot
|
||||
and older than `orphan_older_than_hours`
|
||||
(`weed/worker/tasks/iceberg/operations.go:331`, default 72). Against an adapter-registered
|
||||
Lance table, every fragment is unreferenced by construction. Run maintenance and the
|
||||
dataset is deleted.
|
||||
|
||||
Maintenance is disabled by default (`handler.go:334`), so this is a latent hazard rather
|
||||
than a live one: it needs an operator to enable Iceberg maintenance on a bucket that also
|
||||
holds adapter-registered Lance tables. But it costs nothing to close — detection should
|
||||
skip any table carrying a non-Iceberg format marker (`table_type` property, or
|
||||
`Format != "ICEBERG"` once the format field is honest), and that guard is worth landing on
|
||||
its own regardless of whether the rest of this design ever gets built. It is the same
|
||||
"catalog-only, no maintenance" marker the generic-format question needs.
|
||||
|
||||
## Where we differ from Gravitino
|
||||
|
||||
Gravitino is a metadata service in front of somebody else's object store:
|
||||
|
||||
```
|
||||
Spark / Ray Spark / Ray / pandas / duckdb
|
||||
| |
|
||||
Lance REST Lance REST (direct S3)
|
||||
| | |
|
||||
Gravitino SeaweedFS S3 gateway ----+
|
||||
| |
|
||||
S3 keys handed out SeaweedFS filer + volumes
|
||||
|
|
||||
somebody else's S3
|
||||
```
|
||||
|
||||
It resolves a name to a location plus `lance.storage.*` credentials, and steps out of the
|
||||
way. Everything a Lance table actually is — `_versions/`, `data/`, `_indices/` — is opaque
|
||||
to it.
|
||||
|
||||
We are the store. Three things follow that Gravitino cannot do:
|
||||
|
||||
1. The catalog and a plain directory listing can be made to agree, so a client with no
|
||||
catalog at all still sees the right tables.
|
||||
2. `_versions/` is a filer directory listing, not an object-store `LIST`. Version history
|
||||
is cheap and can back the admin UI.
|
||||
3. We can offer a genuinely atomic commit reservation. Lance's commit protocol needs
|
||||
put-if-not-exists; our S3 layer does not currently provide one (see
|
||||
[Commit safety](#commit-safety)). The filer does.
|
||||
|
||||
## Placement
|
||||
|
||||
The Iceberg catalog is a thin HTTP shell over `s3tables.Manager`; the storage work lives in
|
||||
`weed/s3api/s3tables`. Table buckets live under `TablesPath = s3_constants.DefaultBucketsPath`,
|
||||
i.e. the same filer tree the S3 gateway serves, so `s3://bucket/ns/table/` is simultaneously
|
||||
a catalog entry and an S3 prefix. Catalog entries are filer directories carrying `s3tables.*`
|
||||
extended attributes. `Table.Format` already exists and is hard-checked against `"ICEBERG"`
|
||||
in `weed/s3api/s3tables/handler_table.go:48`.
|
||||
|
||||
So:
|
||||
|
||||
```
|
||||
weed/s3api/lance/ new: HTTP surface, id codec, error model
|
||||
weed/s3api/s3tables/ extended: Format "LANCE", lance state xattr, version entries
|
||||
weed/command/s3.go new: -port.lance (default 9101), startLanceServer
|
||||
```
|
||||
|
||||
`Format: "LANCE"` on the table entry is the whole storage-model change for phase 1.
|
||||
Everything else — namespaces, ARNs, policies, tags, ownership — is shared verbatim.
|
||||
|
||||
```
|
||||
s3tables.Manager (filer)
|
||||
|
|
||||
+------------------------+------------------------+
|
||||
| |
|
||||
weed/s3api/iceberg weed/s3api/lance
|
||||
Iceberg REST :8181 Lance REST :9101
|
||||
| |
|
||||
Iceberg tables Lance datasets
|
||||
\ /
|
||||
+-------------------- s3 :8333 -----------------+
|
||||
|
|
||||
SeaweedFS volumes
|
||||
```
|
||||
|
||||
## Identifier mapping
|
||||
|
||||
Lance identifiers are `["ns", ..., "table"]`, encoded in the URL as a single string joined
|
||||
by a delimiter that defaults to `$`. The delimiter alone means the root namespace, so
|
||||
`/v1/namespace/$/list` lists the root's children.
|
||||
|
||||
Iceberg had to invent a warehouse selector because its identifier is flat and every table
|
||||
bucket is a separate catalog. Lance does not need that — its identifier is already
|
||||
hierarchical, and Gravitino uses exactly three levels (`["lance_catalog", "sales", "orders"]`).
|
||||
That maps onto us without inventing anything:
|
||||
|
||||
```
|
||||
$ root -> list of table buckets
|
||||
$analytics level 1 -> a table bucket
|
||||
$analytics$sales level 2 -> a namespace in that bucket
|
||||
$analytics$sales$orders table
|
||||
```
|
||||
|
||||
`spark.sql.catalog.lance.parent = analytics` then makes `sales.orders` resolve, which is the
|
||||
same shape Gravitino's Spark example uses.
|
||||
|
||||
Levels 2..N join into one `s3tables` namespace with `.`, matching what the Iceberg catalog
|
||||
already does with `flattenNamespacePath`. The flattened form is only the directory name —
|
||||
`namespaceMetadata.Namespace []string` in the xattr keeps the authoritative parts, so the
|
||||
mapping stays invertible even though `.` is a legal character inside a namespace part.
|
||||
Reject `$` in any name part with `InvalidInput`; our charsets already exclude it, so no
|
||||
escaping scheme is needed.
|
||||
|
||||
Root-level `ListNamespaces` returning table buckets means an unauthenticated or
|
||||
broadly-scoped caller can enumerate buckets. Filter it through the same
|
||||
`s3tables/permissions.go` check `ListTableBuckets` uses, not a separate path.
|
||||
|
||||
`CreateNamespace` on a one-part identifier creates a table bucket, and it does so only if
|
||||
the caller is permitted to — the namespace never creates a bucket as a side effect of
|
||||
creating something inside it. A table bucket is a tenant resource with its own policy, ARN
|
||||
and lifecycle, and conjuring one because a client said `CREATE SCHEMA` is a privilege
|
||||
escalation dressed as a convenience. Lakekeeper draws the same line explicitly: its client
|
||||
creates tables, not warehouses.
|
||||
|
||||
## Storage layout
|
||||
|
||||
Lay tables out as:
|
||||
|
||||
```
|
||||
s3://<table-bucket>/<flattened-namespace>/<table>/
|
||||
data/
|
||||
_versions/
|
||||
_indices/
|
||||
```
|
||||
|
||||
**Built without the `.lance` suffix this design originally proposed.** The suffix would have
|
||||
made every namespace prefix a valid Lance Directory Catalog V1 root, since V1 recognises a
|
||||
table by exactly that naming. It does not survive contact with the storage layer: the
|
||||
catalog entry *is* the dataset directory, `validateTableName` excludes `.` from the charset,
|
||||
and a suffixed entry name would leak into ARNs, policy documents and the S3 Tables API,
|
||||
where the same table would answer to two different names. Making `GetTablePath` format-aware
|
||||
instead spreads an "unless it is Lance" branch through code that has no business knowing —
|
||||
the exact cross-cutting cost this design rejects family 3 for.
|
||||
|
||||
So one name, one directory. What survives is direct access by URI, which is the larger half
|
||||
of the story and needs no naming convention at all:
|
||||
|
||||
```python
|
||||
# with the catalog
|
||||
spark.sql("SELECT * FROM lance.sales.orders")
|
||||
|
||||
# without it, same bytes
|
||||
lance.dataset("s3://analytics/sales/orders")
|
||||
```
|
||||
|
||||
DuckDB, pandas and DataFusion still reach the data with no catalog running, which is the gap
|
||||
Gravitino's documentation admits to. What they no longer get for free is *enumeration* — a
|
||||
directory-catalog client pointed at the namespace prefix will not list these as tables. If
|
||||
that turns out to matter, the cheapest fix is a repair-style tool that materialises `.lance`
|
||||
aliases, not a rename of the catalog entry.
|
||||
|
||||
Note also that the directory catalog's own V2 mode puts child-namespace tables in
|
||||
`<hash>_<ns$table>` directories at the root and creates no physical subdirectories for
|
||||
namespaces, so full directory-catalog fidelity was never on offer anyway. We are a
|
||||
server-backed catalog; the human-readable prefix layout is worth more than partial V1
|
||||
lookalike behaviour.
|
||||
|
||||
## The table bucket was not a neutral container
|
||||
|
||||
This design assumed a table bucket is a place to put a table's files. It is
|
||||
not: `validateTableBucketObjectPath` runs on every S3 write into one and
|
||||
validated the path against Iceberg's layout, so a Lance client got 403 on
|
||||
`data/*.lance`, on `_versions/`, and on `_transactions/` — a directory Lance
|
||||
writes that neither the spec documentation nor this design anticipated. Nothing
|
||||
about the catalog worked end to end until that changed.
|
||||
|
||||
The layout guard now admits the union of what the supported formats write, and
|
||||
treats any underscore-prefixed top-level directory as belonging to the format,
|
||||
checking only that the path stays inside the table. Enumerating Lance's
|
||||
internal directories by name is exactly the mistake that missed
|
||||
`_transactions`. Iceberg writes none of them, so it loses nothing.
|
||||
|
||||
Found by pointing the real Python client at a running gateway, not by reading
|
||||
the spec. Worth remembering for the next format: the premise to check first is
|
||||
whether the storage layer will accept its files at all.
|
||||
|
||||
## Table lifecycle
|
||||
|
||||
Lance has three table states, and the spec pins them to marker files:
|
||||
|
||||
| State | Marker | Created by | Visible in ListTables |
|
||||
| --- | --- | --- | --- |
|
||||
| declared | `.lance-reserved` | `DeclareTable` | yes, when `include_declared=true` |
|
||||
| created | `_versions/` present | client writes, or `CreateTable` | yes |
|
||||
| deregistered | `.lance-deregistered` | `DeregisterTable` | no; data preserved |
|
||||
|
||||
Record the state in an xattr (`s3tables.lanceState`) on the catalog entry *and* write the
|
||||
marker file into the table directory. The xattr is what the catalog reads; the marker is
|
||||
what keeps a directory-catalog client honest. Dual-write is the price of the interop claim
|
||||
above, and it is one extra filer write on three rarely-called operations.
|
||||
|
||||
`DeclareTable` is the operation `lance-spark` actually calls on `CREATE TABLE` (it replaced
|
||||
the legacy `create-empty`), so it is not optional in practice even though the spec marks
|
||||
only a subset as required.
|
||||
|
||||
`DeregisterTable` preserving data is the same shape as our Iceberg rename, where the catalog
|
||||
entry moves and the data stays put — reuse `TableDataDirFromMetadataLocation`'s idea rather
|
||||
than re-deriving the data path from the catalog name.
|
||||
|
||||
## Commit safety
|
||||
|
||||
This is the part I got wrong, and the correction removed a feature rather than adding one.
|
||||
|
||||
Lance commits a version by writing `_versions/{v}.manifest` with put-if-not-exists: exactly
|
||||
one writer is supposed to win, and the loser rebases. In lance 10 that path is not optional
|
||||
and needs nothing bolted on — `commit_handler_from_url` hands every `s3://` dataset a
|
||||
`ConditionalPutCommitHandler`, which calls `put_opts` with `PutMode::Create`, which
|
||||
object_store's S3 backend sends as `If-None-Match: *`.
|
||||
|
||||
I originally read our gateway as evaluating that header check-then-act, and designed around
|
||||
it. That was already out of date. `buildWriteCondition`
|
||||
(`weed/s3api/s3api_object_routed_write.go`) reduces `If-None-Match: *` to a filer
|
||||
`WriteCondition{IF_NOT_EXISTS}`, and `putToFiler` routes the create to the object's owner
|
||||
filer, which evaluates the precondition under its per-path lock; when routing is not
|
||||
available it falls back to the object write lock, which evaluates it under the lock too.
|
||||
Either way it is atomic. Sixteen concurrent writers of the same fresh key get one 200 and
|
||||
fifteen 412s, repeatedly.
|
||||
|
||||
So the store already has the primitive Lance needs, cluster-wide, for every conditional-PUT
|
||||
client and not just this one.
|
||||
|
||||
### What that removed
|
||||
|
||||
An earlier draft of this design offered the catalog as an **external manifest store**:
|
||||
`managed_versioning: true` plus `CreateTableVersion` and friends, with the reserve step as a
|
||||
filer `CreateEntry` with `o_excl`. It was implemented, tested, and shipped behind a default-off
|
||||
flag — and it should not exist.
|
||||
|
||||
- It solves a problem this store does not have. The spec offers that path for stores that
|
||||
cannot order commits themselves.
|
||||
- It moves a table's version history out of the dataset and into the catalog, so a reader
|
||||
that does not go through this namespace no longer sees the whole picture. That is a real
|
||||
cost paid for nothing.
|
||||
- lance 10 cannot even use it past the first commit: `NamespaceManifestStore::put_if_not_exists`
|
||||
answers "put_if_not_exists is not supported for namespace-backed stores", which is exactly
|
||||
what a second `append` needs.
|
||||
|
||||
The version operations now answer `Unsupported` alongside the other operations the catalog
|
||||
does not serve, and `managed_versioning` is answered `false`. The property they were
|
||||
protecting is covered instead by a test that races eight writers at the manifest key through
|
||||
S3 and asserts one wins — testing the path Lance actually takes.
|
||||
|
||||
## Credential vending
|
||||
|
||||
Iceberg needed a header (`X-Iceberg-Access-Delegation: vended-credentials`) and a bespoke
|
||||
response shape. Lance has it in the spec: `vend_credentials: true` on the request,
|
||||
`storage_options` on the response, with `expires_at_millis` as the well-known expiry key.
|
||||
|
||||
Reuse the existing vendor interface unchanged — `iceberg.CredentialVendor` /
|
||||
`STSService.AssumeRoleForPrincipal` scoped to the table prefix (#10777) — and map its output
|
||||
to the storage options Lance passes through to `object_store`:
|
||||
|
||||
```
|
||||
aws_access_key_id, aws_secret_access_key, aws_session_token,
|
||||
aws_region, aws_endpoint, allow_http, expires_at_millis
|
||||
```
|
||||
|
||||
Those are the names `pylakekeeper` emits as `lance_storage_options`, which is the shape
|
||||
Lakekeeper's tested S3 path actually feeds to Lance. `object_store` also accepts the
|
||||
un-prefixed aliases (`endpoint`, `region`) that the directory catalog's `storage.` prefix
|
||||
strips down to and that Gravitino's `lance.storage.endpoint` resolves to, but the `aws_`
|
||||
forms are the ones with a tested integration behind them, so emit those. `aws_endpoint`
|
||||
should come from `deriveS3AdvertisedEndpoint()`, the same source the Iceberg `FileIO` config
|
||||
uses, and `allow_http` must be set when that endpoint is plain HTTP or every read fails with
|
||||
a TLS error that looks like a credential problem — Lakekeeper vends both automatically for
|
||||
exactly this reason, and calls out that there is then no per-vendor branch in client code.
|
||||
|
||||
We emit this server-side, in the `storage_options` field the Lance spec already defines,
|
||||
which is strictly better than Lakekeeper's arrangement: no client library has to translate
|
||||
anything, so vending works from any stock Lance client rather than only from theirs.
|
||||
|
||||
Guard the same way #10777 had to after review: bucket-scoped list grants need an `s3:prefix`
|
||||
condition, and a location containing `*` or `?` must be refused rather than widened into a
|
||||
resource pattern.
|
||||
|
||||
## Auth and authorization
|
||||
|
||||
Authentication reuses `S3Authenticator` and `CredentialValidator` as-is. The Lance spec maps
|
||||
identity to headers — `api_key` to `x-api-key`, `auth_token` to `Authorization: Bearer` — and
|
||||
SigV4 keeps working because it is the same authenticator the Iceberg catalog already fronts.
|
||||
|
||||
Authorization needs nothing new. A Lance table gets the same ARN shape,
|
||||
`arn:aws:s3tables:...:bucket/B/table/NS/T`, so every existing table-bucket policy covers
|
||||
Lance tables with no new policy language and no second permission model. Route it through
|
||||
`s3tables/permissions.go` and inherit the `DefaultAllow` semantics the Iceberg server already
|
||||
mirrors from the S3 port.
|
||||
|
||||
One spec quirk worth honoring: request context entries prefixed `header.` become request
|
||||
headers, and every response header comes back as a `header.`-prefixed context entry. Echoing
|
||||
`x-request-id` through it costs nothing and makes tracing work.
|
||||
|
||||
## What to take from Lakekeeper
|
||||
|
||||
Rejecting Lakekeeper's API shape does not mean rejecting what it learned building it.
|
||||
|
||||
**Deregister is soft-delete, so implement it as one.** Lakekeeper gives generic tables
|
||||
soft-deletion with undrop and a `protected` flag that makes a drop require `force=true`.
|
||||
Lance already has the concept — `DeregisterTable` preserves the data and hides the table —
|
||||
so the `.lance-deregistered` marker is a soft-delete by another name, and a re-register is
|
||||
an undrop. A protection flag on table-bucket entries is worth having regardless of Lance:
|
||||
it is a few lines against the existing xattrs and it applies to Iceberg tables too.
|
||||
|
||||
**Enforce one identifier space across entry kinds.** Lakekeeper rejects a generic table
|
||||
whose name collides with an Iceberg table or view in the same namespace. Our catalog entries
|
||||
already share one filer directory and already carry `s3tables.entryType`, so this is
|
||||
structurally true — but it has to be enforced deliberately on every path, or a Lance handler
|
||||
happily loads an Iceberg table's directory and vice versa. That is the same crossover bug
|
||||
class as the view/table rename authorization fixed in #10776; the `catalogEntryKind` pattern
|
||||
from that change is the thing to reuse rather than re-derive.
|
||||
|
||||
**A re-vend path matters more than it looks.** Lakekeeper exposes `/credentials` separately
|
||||
from load, because STS credentials expire in the middle of long jobs and re-loading the
|
||||
whole table to refresh them is wasteful. In Lance the spec's answer is another
|
||||
`DescribeTable` with `vend_credentials: true`, which is fine — but it means `DescribeTable`
|
||||
must stay cheap when `load_detailed_metadata` is false, which is another reason not to open
|
||||
the dataset on that path.
|
||||
|
||||
**Generic tables are a cheap orthogonal win.** Lakekeeper's real insight is that Delta,
|
||||
Parquet, CSV, Vortex and Paimon all get governance for free once the catalog stops caring
|
||||
what the format is. Our `Table.Format` field already exists and the only thing stopping it
|
||||
is the hard `"ICEBERG"` check in `handler_table.go:48`. Loosening that and letting the S3
|
||||
Tables API register a table with an arbitrary format and a location — no metadata, no
|
||||
commits — is a small change that makes every format cataloguable. It is independent of this
|
||||
design and probably worth doing first, since `Format: "LANCE"` is then just a value rather
|
||||
than a special case.
|
||||
|
||||
**Skip remote signing.** It is Lakekeeper's fallback for S3-compatible stores with no STS,
|
||||
and their own documentation notes that Lance will not use it — format libraries with their
|
||||
own S3 client expect static credentials and do not implement the Iceberg signer protocol. We
|
||||
have STS, so vended credentials are the path, and the signer is not worth building for a
|
||||
client that cannot consume it.
|
||||
|
||||
## Errors
|
||||
|
||||
Lance uses `{code, error, detail, instance}` with numeric codes, not Iceberg's exception-type
|
||||
strings. The mapping is mechanical:
|
||||
|
||||
| HTTP | code | when |
|
||||
| --- | --- | --- |
|
||||
| 400 | 13 InvalidInput | charset violations, malformed id, route/body mismatch |
|
||||
| 401 | 16 Unauthenticated | |
|
||||
| 403 | 15 PermissionDenied | |
|
||||
| 404 | 1 NamespaceNotFound, 4 TableNotFound, 11 TableVersionNotFound | |
|
||||
| 409 | 2/5 AlreadyExists, 3 NamespaceNotEmpty, 14 ConcurrentModification | |
|
||||
| 501 | 0 Unsupported | every phase-3 data operation |
|
||||
|
||||
Route/body mismatch is a spec requirement, not a nicety: when the identifier appears in both
|
||||
the path and the body and they disagree, the server must return 400. Cheap to get right at
|
||||
the decode step, annoying to retrofit.
|
||||
|
||||
## Route surface
|
||||
|
||||
Phase 0 is not in this table: point a stock Lance client at the existing Iceberg catalog
|
||||
with the Iceberg impl, see how far it gets, and land the maintenance guard either way. That
|
||||
tells us what the native server actually has to beat.
|
||||
|
||||
Phase 1, the whole `lance-spark` and `lance-ray` contract:
|
||||
|
||||
```
|
||||
POST /v1/namespace/{id}/create CreateNamespace mode: Create|ExistOk|Overwrite
|
||||
GET /v1/namespace/{id}/list ListNamespaces
|
||||
POST /v1/namespace/{id}/describe DescribeNamespace
|
||||
POST /v1/namespace/{id}/drop DropNamespace mode: Fail|Skip, behavior: Restrict|Cascade
|
||||
POST /v1/namespace/{id}/exists NamespaceExists
|
||||
GET /v1/namespace/{id}/table/list ListTables ?include_declared, ?page_token, ?limit
|
||||
GET /v1/table ListAllTables
|
||||
POST /v1/table/{id}/declare DeclareTable
|
||||
POST /v1/table/{id}/describe DescribeTable ?with_table_uri, ?load_detailed_metadata, ?check_declared
|
||||
POST /v1/table/{id}/exists TableExists
|
||||
POST /v1/table/{id}/register RegisterTable mode: Create|Overwrite
|
||||
POST /v1/table/{id}/deregister DeregisterTable
|
||||
POST /v1/table/{id}/drop DropTable
|
||||
POST /v1/table/{id}/rename RenameTable
|
||||
```
|
||||
|
||||
`DescribeTable` with `load_detailed_metadata=false` needs only `location`, which is the
|
||||
common case and which we can answer from xattrs alone. With `load_detailed_metadata=true`
|
||||
the spec wants `version`, `schema` and `stats`, which means reading the Lance manifest. For
|
||||
phase 1, return the fields we can derive from the filer — `version` from the highest entry in
|
||||
`_versions/`, given V2 naming is `{u64::MAX - version:020}.manifest` and V1 is
|
||||
`{version}.manifest` — and omit `schema`/`stats` rather than fabricating them. The spec
|
||||
tolerates a partial response here; it does not tolerate a wrong one.
|
||||
|
||||
Phase 2 was the five version operations plus `managed_versioning`; it was built and then
|
||||
removed, for the reasons under Commit safety.
|
||||
|
||||
Phase 3 is the data plane: `CreateTable`, `InsertIntoTable`, `MergeInsertIntoTable`,
|
||||
`UpdateTable`, `DeleteFromTable`, `QueryTable`, `CountTableRows`, and the index and tag
|
||||
operations. These exchange Arrow IPC, and more to the point they require reading and writing
|
||||
the Lance file format, for which no Go implementation exists. Return `Unsupported` (code 0)
|
||||
and say so in the docs. `arrow-go/v18` is already an indirect dependency, so Arrow framing is
|
||||
not the blocker — Lance is.
|
||||
|
||||
## Does a Lance table need maintenance?
|
||||
|
||||
Yes, and one part of it has no Iceberg equivalent. The client exposes three jobs:
|
||||
|
||||
- `optimize.compact_files()` — Lance writes a fragment per write batch, so a table fed by
|
||||
small appends accumulates small files exactly the way an Iceberg table does.
|
||||
- `optimize.optimize_indices()` — **rows written after an index was built are not covered by
|
||||
it.** A vector search against a stale index silently misses recent data. That is a
|
||||
correctness-shaped failure, not a slow query, and it is specific to what people use Lance
|
||||
for.
|
||||
- `cleanup_old_versions()` — every version is retained until something removes it. Lance can
|
||||
do this itself: `optimize.enable_auto_cleanup()` sets it on the dataset, so this one need
|
||||
not be an external job at all.
|
||||
|
||||
None of it can run in the Go worker. All three read and rewrite Lance files, which needs
|
||||
Lance format code that does not exist in Go, and there is no useful subset either: deciding
|
||||
which fragments an old version still references means parsing Lance manifests.
|
||||
|
||||
So the maintenance worker must not touch a Lance table, and it declines by reading the format
|
||||
the catalog recorded rather than by failing to parse Iceberg metadata.
|
||||
|
||||
## The worker can be Rust, and it is not a sidecar
|
||||
|
||||
The Go worker is not the only worker. `weed/pb/plugin.proto` defines `PluginControlService`,
|
||||
a language-agnostic gRPC stream that external maintenance workers connect on: the worker
|
||||
opens `WorkerStream`, sends `WorkerHello` with the job types it can `detect` and `execute`,
|
||||
answers `RequestConfigSchema` with a `JobTypeDescriptor`, replies to `RunDetectionRequest`
|
||||
with `JobProposal`s and to `ExecuteJobRequest` with `JobProgressUpdate`s and `JobCompleted`.
|
||||
`weed worker -admin=host:23646` is the Go reference implementation of exactly that contract,
|
||||
from outside the admin process.
|
||||
|
||||
Nothing in it is Go-specific, and the Rust toolchain is already in the tree.
|
||||
`seaweed-volume/build.rs` compiles protos straight out of `../weed/pb/` with `tonic_build`,
|
||||
including `filer.proto`, on tonic 0.12 and prost 0.13. A Lance worker is that same build
|
||||
with `plugin.proto` added and the `lance` crate as a dependency — the real one, no FFI and
|
||||
no Python.
|
||||
|
||||
Three job types, one per real maintenance operation:
|
||||
|
||||
| Job type | Calls | Detected from |
|
||||
| --- | --- | --- |
|
||||
| `lance_compact` | `optimize.compact_files` | fragment count and sizes |
|
||||
| `lance_optimize_indices` | `optimize.optimize_indices` | rows an index does not cover |
|
||||
| `lance_cleanup_versions` | `cleanup_old_versions` | version count and age |
|
||||
|
||||
What the existing machinery then supplies for free is the part worth noticing. Scheduling,
|
||||
retries, dedupe by `dedupe_key`, progress reporting, per-job concurrency limits and the
|
||||
admin settings page all come from the protocol: a worker that answers `RequestConfigSchema`
|
||||
with a descriptor gets its configuration form rendered in the admin UI without a line of Go
|
||||
or templ. A Rust worker is a first-class maintenance worker, not an appendage.
|
||||
|
||||
The remaining wiring is small and mostly decided already. `RunDetectionRequest` carries a
|
||||
`ClusterContext` with filer and S3 addresses plus a free-form `metadata` map, which is where
|
||||
the Lance namespace URL goes; the worker lists Lance tables from the namespace, which is the
|
||||
catalog of record and already filters by format. It gets at the data by asking
|
||||
`DescribeTable` for `storage_options` with `vend_credentials`, so the worker is just another
|
||||
client of the STS path rather than a component with its own credentials. And when it commits
|
||||
a compaction it goes through `CreateTableVersion` like any other writer, which is what
|
||||
managed versioning was for.
|
||||
|
||||
## The worker is also the only thing that can describe the table
|
||||
|
||||
Admin can render an Iceberg table because it can read Iceberg metadata. It cannot read
|
||||
Lance: it knows the dataset's location and its format string, and that is the whole of it.
|
||||
The details page showed a location and two empty panels, which is an honest answer and a
|
||||
useless one.
|
||||
|
||||
The worker already knows. Detection opens every dataset to decide whether it needs
|
||||
compacting, so at that moment it holds the schema, the row count, the fragment count and
|
||||
the version count. It just had no way to say so — every message on the stream was about
|
||||
work.
|
||||
|
||||
So `WorkerObservations` is a body on `WorkerToAdminMessage`: a repeated `ObjectObservation`
|
||||
of `object_id`, `object_kind`, `format`, and a `ConfigValue` map the worker fills with
|
||||
whatever it can cheaply say. Admin keeps the last observation per object and serves it back
|
||||
with the time it was taken and the worker that took it. Nothing schedules from it, and it is
|
||||
not authoritative — it is a cache with its staleness on the label, which is why the page
|
||||
badges it rather than presenting it as metadata it read itself.
|
||||
|
||||
The keys are the worker's to choose, which keeps the protocol out of the business of knowing
|
||||
what a Lance table is. A worker for any other format admin cannot parse describes itself the
|
||||
same way.
|
||||
|
||||
## A bucket declares its format
|
||||
|
||||
Format was recorded per table, which is enough for the storage layer and not enough for
|
||||
anything that has to answer a question about a bucket. The admin UI printed one Iceberg
|
||||
endpoint for every bucket, including the ones holding Lance datasets, where that endpoint
|
||||
serves nothing; an empty bucket had no format at all.
|
||||
|
||||
So `CreateTableBucket` takes an optional `format`, stored with the rest of the bucket
|
||||
metadata. Empty means `ICEBERG` - what AWS S3 Tables serves, and therefore what an SDK
|
||||
that has never heard of the field means. `CreateTable` refuses another format, and
|
||||
`CreateView` refuses outright outside an Iceberg bucket, a view being Iceberg metadata.
|
||||
The Lance namespace declares `LANCE` for the buckets it creates.
|
||||
|
||||
**Enforced rather than defaulted**, because the point of showing a format at all is the
|
||||
endpoint that follows from it, and that endpoint is only truthful if the bucket holds one
|
||||
format. **Buckets that already exist stay undeclared** and keep taking anything: nothing is
|
||||
migrated, and the UI shows "unset" as a fact about the bucket's age rather than a fault.
|
||||
That state is also the only way to hold both formats at once, which is what the
|
||||
Iceberg-REST adapter path produces.
|
||||
|
||||
## Sample rows are fetched, not cached
|
||||
|
||||
The same asymmetry has a second half. Admin renders an Iceberg table's rows by
|
||||
reading its Parquet files directly; for Lance it has nothing to read with, so the data
|
||||
page offered a Browse Data button that led to an empty grid.
|
||||
|
||||
`RequestObjectPreview` / `ObjectPreviewResponse` mirror the config-schema round trip
|
||||
already on the stream: admin asks, the worker scans the dataset and hands back rows it
|
||||
has already rendered as text, because it is the only side that knows the types. Admin
|
||||
picks the worker from the observation store, so the one that last described a table is
|
||||
the one asked to read it.
|
||||
|
||||
The rows are deliberately not cached, and that is the line between the two channels. An
|
||||
observation describes an object, so a copy with a timestamp on it is useful. Rows are the
|
||||
object's contents: a copy held in admin would be stale, larger, and nobody's business.
|
||||
The page fetches on load, bounded, or says why it cannot.
|
||||
|
||||
## The sidecar question
|
||||
|
||||
The data plane is a different problem, and this design previously conflated the two.
|
||||
Maintenance rides the worker protocol; `QueryTable` and `InsertIntoTable` do not, because
|
||||
they are synchronous REST operations on the namespace's own surface. Serving those means a
|
||||
Rust process that answers HTTP, either behind the Go namespace as a proxy target or in front
|
||||
of it. It would make SeaweedFS a store you can run vector search *in* rather than one you
|
||||
read vectors *out of*, which is the larger prize and the reason to keep the option open.
|
||||
|
||||
Neither should gate phase 1. Phases 1 and 2 are pure Go over the filer and are worth
|
||||
shipping on their own — they are what makes Spark and Ray work.
|
||||
|
||||
## Testing
|
||||
|
||||
Mirror the Iceberg package: `httptest` plus a fake filer client for the handler tests, in
|
||||
`weed/s3api/lance`. Then an integration suite under `test/s3tables/catalog/` next to the
|
||||
existing `pyiceberg_test.go`, driving the generated Python `lance-namespace` client against
|
||||
a live gateway. Three things that suite must cover and unit tests cannot:
|
||||
|
||||
- the storage-options key names actually work, i.e. a client that gets `storage_options` from
|
||||
`DescribeTable` can open the dataset;
|
||||
- a table created through the catalog is visible to `lance.dataset()` by URI and to a V1
|
||||
directory-catalog client rooted at the namespace prefix;
|
||||
- concurrent writers do not lose a commit, which is the phase-2 acceptance test and the
|
||||
thing that justifies the external manifest store.
|
||||
|
||||
Phase 1 is validated: `lance_namespace` 0.11.1 with `impl=rest` drives the namespace,
|
||||
`lance.write_dataset` writes to the vended location with the vended `storage_options`, and
|
||||
the rows read back. Note that this client version drops `check_declared` and
|
||||
`include_declared` on the wire, so `is_only_declared` reads null through it however the
|
||||
server behaves.
|
||||
|
||||
The commit path is validated at both levels. The mechanism: eight writers race the same
|
||||
manifest key through S3 with `If-None-Match: *`, and exactly one wins. The property that
|
||||
actually matters, which single-winner exclusivity does not by itself establish: eight
|
||||
writers append to one dataset concurrently through lance, and afterwards every batch is
|
||||
still there — the losers saw the conflict, rebased, and committed again. That second test
|
||||
is also the sequence managed versioning could not complete at all, since its store answers
|
||||
"put_if_not_exists is not supported" to the second commit.
|
||||
|
||||
One more that belongs in the Iceberg suite, not this one: a Lance dataset registered through
|
||||
the Iceberg adapter must survive a full maintenance pass. Reading the code, that test should
|
||||
fail today; it has not been run.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Root-level `ListNamespaces` enumerating table buckets is convenient and is a listing
|
||||
surface we do not have on the Iceberg side. Decide whether it is gated behind a flag.
|
||||
- Whether the `.lance` directory suffix is worth the divergence from the Iceberg layout. I
|
||||
think yes — it is what makes the catalog optional — but it means the two catalogs' tables
|
||||
do not look alike on disk, and the admin UI has to know that.
|
||||
- Names: our charsets are lowercase-only and Lance identifiers are arbitrary strings. Reject
|
||||
and document, as Iceberg does, or case-fold. Rejecting is right, but see #10734 for how
|
||||
case handling bites when only one side normalizes.
|
||||
- Whether to land generic-format registration first. Dropping the `"ICEBERG"` check and
|
||||
letting a table carry an arbitrary format plus a location is smaller than this whole
|
||||
design, gets Delta and Parquet catalogued as a side effect, and turns `Format: "LANCE"`
|
||||
into an ordinary value. The argument against is that it invites tables the maintenance
|
||||
worker cannot service, so it needs a "catalog-only, no maintenance" marker to be honest.
|
||||
@@ -0,0 +1 @@
|
||||
target/
|
||||
Generated
+6754
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,24 @@
|
||||
# Rust plugin workers for SeaweedFS.
|
||||
#
|
||||
# `core` is the plugin.proto contract and nothing else; a worker crate beside it
|
||||
# supplies job handlers and a binary. Adding a worker means adding a member here,
|
||||
# not touching the protocol.
|
||||
[workspace]
|
||||
resolver = "2"
|
||||
members = ["crates/core", "crates/lance"]
|
||||
|
||||
[workspace.package]
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
|
||||
[workspace.dependencies]
|
||||
anyhow = "1"
|
||||
async-trait = "0.1"
|
||||
prost = "0.13"
|
||||
prost-types = "0.13"
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
tokio-stream = "0.1"
|
||||
tonic = { version = "0.12", features = ["tls"] }
|
||||
tonic-build = "0.12"
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
@@ -0,0 +1,47 @@
|
||||
# SeaweedFS Rust workers
|
||||
|
||||
`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
|
||||
connects out to admin, announces the job types it can detect and execute, and
|
||||
answers requests on that one stream. `weed worker -admin=host:23646` is the Go
|
||||
implementation of it from outside the admin process. This workspace is the Rust
|
||||
one.
|
||||
|
||||
crates/core the contract: stream, handshake, heartbeat, registry, config forms
|
||||
crates/lance maintenance jobs for Lance tables, and a binary
|
||||
|
||||
`core` knows nothing about any job. A second worker is a new crate beside
|
||||
`lance` that depends on it, not a fork of the protocol.
|
||||
|
||||
## Running
|
||||
|
||||
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
|
||||
|
||||
The admin's *HTTP* address is what an operator has; the gRPC port is derived
|
||||
from it the way the Go side does. Dialling the HTTP port fails as "frame with
|
||||
invalid size", which reads like a protocol bug rather than a wrong port.
|
||||
|
||||
## Credentials
|
||||
|
||||
The worker holds none. It asks the namespace to describe a table with
|
||||
`vend_credentials` and hands the `storage_options` that come back to lance. A
|
||||
gateway without STS configured vends no credentials at all, so `--access-key`
|
||||
and `--secret-key` supply a fallback; anything the namespace does vend wins over
|
||||
them.
|
||||
|
||||
## State
|
||||
|
||||
All three jobs are implemented and tested end to end against a live gateway:
|
||||
|
||||
compaction result: 12 fragments became 1
|
||||
reindex result: 512 uncovered rows became 0
|
||||
cleanup result: removed 14 versions and 24272 bytes
|
||||
|
||||
`cargo test -p weed-lance-worker` runs them when `WEED_LANCE_NAMESPACE` names a
|
||||
live namespace and skips otherwise, the way the Go integration tests skip
|
||||
without Docker. Each test seeds the table it needs, including building a vector
|
||||
index and then appending rows outside it, so a run does not depend on what the
|
||||
previous one left behind — the first version of these did, and quietly stopped
|
||||
testing anything once it had done its job.
|
||||
|
||||
The handshake, descriptor exchange and heartbeat work against a live admin,
|
||||
which logs the worker connecting and prefetches all three descriptors.
|
||||
@@ -0,0 +1,21 @@
|
||||
[package]
|
||||
name = "seaweed-worker-core"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
description = "SeaweedFS plugin.proto worker contract"
|
||||
|
||||
[lib]
|
||||
name = "seaweed_worker_core"
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
async-trait.workspace = true
|
||||
prost.workspace = true
|
||||
prost-types.workspace = true
|
||||
tokio.workspace = true
|
||||
tokio-stream.workspace = true
|
||||
tonic.workspace = true
|
||||
tracing.workspace = true
|
||||
|
||||
[build-dependencies]
|
||||
tonic-build.workspace = true
|
||||
@@ -0,0 +1,12 @@
|
||||
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
// Compiled straight out of the Go tree, the way seaweed-volume already reads
|
||||
// filer.proto, so the contract cannot drift from a vendored copy.
|
||||
tonic_build::configure()
|
||||
// The server half is only for tests, which stand up a fake admin.
|
||||
.build_server(true)
|
||||
.build_client(true)
|
||||
.protoc_arg("--experimental_allow_proto3_optional")
|
||||
.compile_protos(&["../../../weed/pb/plugin.proto"], &["../../../weed/pb/"])?;
|
||||
println!("cargo:rerun-if-changed=../../../weed/pb/plugin.proto");
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
//! SeaweedFS addresses the way the Go tree does.
|
||||
//!
|
||||
//! An operator gives a worker the admin's HTTP address, and the gRPC port is
|
||||
//! derived from it rather than asked for separately. Dialling the HTTP port
|
||||
//! instead fails as "frame with invalid size", which reads like a protocol bug
|
||||
//! rather than a wrong port, so getting this right is worth its own module.
|
||||
//! Mirrors pb.ServerToGrpcAddress in weed/pb/grpc_client_server.go.
|
||||
|
||||
const GRPC_PORT_OFFSET: u16 = 10000;
|
||||
|
||||
/// Converts `host:port` to the gRPC address, and accepts the explicit
|
||||
/// `host:port.grpcPort` form the Go side also understands.
|
||||
pub fn server_to_grpc_address(server: &str) -> Option<String> {
|
||||
let (host, port_part) = server.rsplit_once(':')?;
|
||||
|
||||
// "port.grpcPort" states the gRPC port outright.
|
||||
if let Some((_, grpc_port)) = port_part.split_once('.') {
|
||||
if let Ok(port) = grpc_port.parse::<u16>() {
|
||||
return Some(join_host_port(host, port));
|
||||
}
|
||||
}
|
||||
|
||||
let port: u16 = port_part.parse().ok()?;
|
||||
Some(join_host_port(host, port.checked_add(GRPC_PORT_OFFSET)?))
|
||||
}
|
||||
|
||||
fn join_host_port(host: &str, port: u16) -> String {
|
||||
// An IPv6 literal has to keep its brackets or the port reads as part of it.
|
||||
if host.contains(':') && !host.starts_with('[') {
|
||||
format!("[{host}]:{port}")
|
||||
} else {
|
||||
format!("{host}:{port}")
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::server_to_grpc_address;
|
||||
|
||||
#[test]
|
||||
fn derives_the_grpc_port() {
|
||||
assert_eq!(
|
||||
server_to_grpc_address("localhost:23646").as_deref(),
|
||||
Some("localhost:33646")
|
||||
);
|
||||
assert_eq!(
|
||||
server_to_grpc_address("127.0.0.1:9333").as_deref(),
|
||||
Some("127.0.0.1:19333")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn honours_an_explicit_grpc_port() {
|
||||
assert_eq!(
|
||||
server_to_grpc_address("localhost:23646.33999").as_deref(),
|
||||
Some("localhost:33999")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn brackets_ipv6_literals() {
|
||||
assert_eq!(
|
||||
server_to_grpc_address("::1:23646").as_deref(),
|
||||
Some("[::1]:33646")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_what_it_cannot_parse() {
|
||||
assert!(server_to_grpc_address("localhost").is_none());
|
||||
assert!(server_to_grpc_address("localhost:notaport").is_none());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
use std::time::Duration;
|
||||
|
||||
/// How one worker process connects and how much work it will take on.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct WorkerOptions {
|
||||
/// Admin gRPC address, e.g. "localhost:23646".
|
||||
pub admin_address: String,
|
||||
pub worker_id: String,
|
||||
pub worker_version: String,
|
||||
/// Advertised address; empty when the worker takes no inbound connections.
|
||||
pub worker_address: String,
|
||||
pub heartbeat_interval: Duration,
|
||||
pub reconnect_delay: Duration,
|
||||
pub max_detection_concurrency: i32,
|
||||
pub max_execution_concurrency: i32,
|
||||
/// mTLS for the control stream, mirroring the Go worker's `[grpc.worker]`
|
||||
/// section of security.toml. None means plaintext, which is the default the
|
||||
/// Go worker also takes when no certificates are configured.
|
||||
pub tls: Option<TlsOptions>,
|
||||
}
|
||||
|
||||
/// Certificates for the control stream. All three are required together: the
|
||||
/// cluster's gRPC TLS is mutual, so a CA without a client identity gets refused
|
||||
/// by admin rather than falling back to one-way TLS.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct TlsOptions {
|
||||
pub ca_path: String,
|
||||
pub client_cert_path: String,
|
||||
pub client_key_path: String,
|
||||
/// Name to verify the server certificate against, when the address a worker
|
||||
/// dials is not the name the certificate carries.
|
||||
pub server_name: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for WorkerOptions {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
admin_address: "localhost:23646".to_string(),
|
||||
worker_id: String::new(),
|
||||
worker_version: env!("CARGO_PKG_VERSION").to_string(),
|
||||
worker_address: String::new(),
|
||||
heartbeat_interval: Duration::from_secs(10),
|
||||
reconnect_delay: Duration::from_secs(5),
|
||||
max_detection_concurrency: 1,
|
||||
max_execution_concurrency: 1,
|
||||
tls: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
//! Builders for the config forms a worker returns in its JobTypeDescriptor.
|
||||
//!
|
||||
//! Admin renders these into the job's settings page, so a worker written in any
|
||||
//! language gets a UI without touching Go or templ. That only holds if the
|
||||
//! field types and defaults are right, which is why they are built here rather
|
||||
//! than spelled out at each call site.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use crate::pb::{
|
||||
config_value::Kind, ConfigField, ConfigFieldType, ConfigForm, ConfigSection, ConfigValue,
|
||||
};
|
||||
|
||||
pub fn int_value(value: i64) -> ConfigValue {
|
||||
ConfigValue {
|
||||
kind: Some(Kind::Int64Value(value)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn bool_value(value: bool) -> ConfigValue {
|
||||
ConfigValue {
|
||||
kind: Some(Kind::BoolValue(value)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn string_value(value: impl Into<String>) -> ConfigValue {
|
||||
ConfigValue {
|
||||
kind: Some(Kind::StringValue(value.into())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads an integer a request carried, falling back when admin sent nothing.
|
||||
pub fn int_or(values: &HashMap<String, ConfigValue>, name: &str, fallback: i64) -> i64 {
|
||||
match values.get(name).and_then(|v| v.kind.as_ref()) {
|
||||
Some(Kind::Int64Value(value)) => *value,
|
||||
Some(Kind::DoubleValue(value)) => *value as i64,
|
||||
_ => fallback,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn bool_or(values: &HashMap<String, ConfigValue>, name: &str, fallback: bool) -> bool {
|
||||
match values.get(name).and_then(|v| v.kind.as_ref()) {
|
||||
Some(Kind::BoolValue(value)) => *value,
|
||||
_ => fallback,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn string_or(values: &HashMap<String, ConfigValue>, name: &str, fallback: &str) -> String {
|
||||
match values.get(name).and_then(|v| v.kind.as_ref()) {
|
||||
Some(Kind::StringValue(value)) if !value.is_empty() => value.clone(),
|
||||
_ => fallback.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn number_field(name: &str, label: &str, description: &str, min: i64, max: i64) -> ConfigField {
|
||||
ConfigField {
|
||||
name: name.to_string(),
|
||||
label: label.to_string(),
|
||||
description: description.to_string(),
|
||||
field_type: ConfigFieldType::Int64 as i32,
|
||||
min_value: Some(int_value(min)),
|
||||
max_value: Some(int_value(max)),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
pub fn bool_field(name: &str, label: &str, description: &str) -> ConfigField {
|
||||
ConfigField {
|
||||
name: name.to_string(),
|
||||
label: label.to_string(),
|
||||
description: description.to_string(),
|
||||
field_type: ConfigFieldType::Bool as i32,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
pub fn form(
|
||||
form_id: &str,
|
||||
title: &str,
|
||||
fields: Vec<ConfigField>,
|
||||
defaults: HashMap<String, ConfigValue>,
|
||||
) -> ConfigForm {
|
||||
ConfigForm {
|
||||
form_id: form_id.to_string(),
|
||||
title: title.to_string(),
|
||||
description: String::new(),
|
||||
sections: vec![ConfigSection {
|
||||
section_id: format!("{form_id}-main"),
|
||||
title: title.to_string(),
|
||||
description: String::new(),
|
||||
fields,
|
||||
}],
|
||||
default_values: defaults,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
//! The SeaweedFS plugin worker contract, in Rust.
|
||||
//!
|
||||
//! `weed/pb/plugin.proto` is a language-agnostic gRPC stream: a worker connects
|
||||
//! out to admin, announces the job types it can detect and execute, and then
|
||||
//! answers requests on that one stream. `weed worker -admin=host:23646` is the
|
||||
//! Go implementation of the same contract from outside the admin process; this
|
||||
//! is the Rust one, and it knows nothing about any particular job.
|
||||
|
||||
pub mod address;
|
||||
pub mod config;
|
||||
pub mod config_form;
|
||||
pub mod registry;
|
||||
pub mod senders;
|
||||
pub mod stream;
|
||||
|
||||
/// Generated plugin.proto types.
|
||||
pub mod pb {
|
||||
tonic::include_proto!("plugin");
|
||||
}
|
||||
|
||||
pub use config::{TlsOptions, WorkerOptions};
|
||||
pub use registry::{JobHandler, Preview, PreviewProvider, Registry};
|
||||
pub use senders::{DetectionSender, ExecutionSender};
|
||||
pub use stream::run;
|
||||
@@ -0,0 +1,89 @@
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::Result;
|
||||
use async_trait::async_trait;
|
||||
|
||||
use crate::pb::{ExecuteJobRequest, JobTypeCapability, JobTypeDescriptor, RunDetectionRequest};
|
||||
use crate::senders::{DetectionSender, ExecutionSender};
|
||||
|
||||
/// One job type, worker side. Mirrors the Go JobHandler interface in
|
||||
/// weed/plugin/worker/worker.go so the two stay readable against each other.
|
||||
#[async_trait]
|
||||
pub trait JobHandler: Send + Sync {
|
||||
fn capability(&self) -> JobTypeCapability;
|
||||
/// The descriptor admin renders as this job's settings page.
|
||||
fn descriptor(&self) -> JobTypeDescriptor;
|
||||
async fn detect(
|
||||
&self,
|
||||
request: &RunDetectionRequest,
|
||||
sender: &dyn DetectionSender,
|
||||
) -> Result<()>;
|
||||
async fn execute(
|
||||
&self,
|
||||
request: &ExecuteJobRequest,
|
||||
sender: &dyn ExecutionSender,
|
||||
) -> Result<()>;
|
||||
}
|
||||
|
||||
/// Sample rows of one object, already rendered as text. The worker formats
|
||||
/// them because it is the only side that knows the object's types.
|
||||
pub struct Preview {
|
||||
pub columns: Vec<String>,
|
||||
pub rows: Vec<Vec<String>>,
|
||||
/// Rows in the object, which is not the number sampled.
|
||||
pub total_rows: i64,
|
||||
}
|
||||
|
||||
/// Reads sample rows of a format admin cannot parse itself.
|
||||
///
|
||||
/// This is deliberately not a JobHandler: a preview is answered while someone
|
||||
/// waits on a page, so it neither schedules nor reports progress.
|
||||
#[async_trait]
|
||||
pub trait PreviewProvider: Send + Sync {
|
||||
/// The format this provider reads, matched case-insensitively against what
|
||||
/// the catalog recorded.
|
||||
fn format(&self) -> &str;
|
||||
async fn preview(&self, object_id: &[String], row_limit: usize) -> Result<Preview>;
|
||||
}
|
||||
|
||||
/// The handlers one worker process serves. A process may serve several job
|
||||
/// types, which is why WorkerHello carries a list of capabilities.
|
||||
#[derive(Default, Clone)]
|
||||
pub struct Registry {
|
||||
handlers: HashMap<String, Arc<dyn JobHandler>>,
|
||||
previews: HashMap<String, Arc<dyn PreviewProvider>>,
|
||||
}
|
||||
|
||||
impl Registry {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
pub fn register(mut self, handler: Arc<dyn JobHandler>) -> Self {
|
||||
self.handlers.insert(handler.capability().job_type, handler);
|
||||
self
|
||||
}
|
||||
|
||||
pub fn with_preview(mut self, provider: Arc<dyn PreviewProvider>) -> Self {
|
||||
self.previews
|
||||
.insert(provider.format().to_ascii_uppercase(), provider);
|
||||
self
|
||||
}
|
||||
|
||||
pub fn get(&self, job_type: &str) -> Option<Arc<dyn JobHandler>> {
|
||||
self.handlers.get(job_type).cloned()
|
||||
}
|
||||
|
||||
pub fn preview_provider(&self, format: &str) -> Option<Arc<dyn PreviewProvider>> {
|
||||
self.previews.get(&format.to_ascii_uppercase()).cloned()
|
||||
}
|
||||
|
||||
pub fn capabilities(&self) -> Vec<JobTypeCapability> {
|
||||
self.handlers.values().map(|h| h.capability()).collect()
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.handlers.is_empty()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
use anyhow::Result;
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
use crate::pb::{
|
||||
worker_to_admin_message::Body, ActivityEvent, DetectionComplete, DetectionProposals,
|
||||
JobCompleted, JobProgressUpdate, WorkerObservations, WorkerToAdminMessage,
|
||||
};
|
||||
|
||||
/// Replies to one detection request.
|
||||
pub trait DetectionSender: Send + Sync {
|
||||
fn send_proposals(&self, proposals: DetectionProposals) -> Result<()>;
|
||||
fn send_complete(&self, complete: DetectionComplete) -> Result<()>;
|
||||
fn send_activity(&self, activity: ActivityEvent) -> Result<()>;
|
||||
/// Reports what the worker saw while deciding. Admin caches the last one
|
||||
/// per object and serves it back for display; nothing is scheduled from it.
|
||||
fn send_observations(&self, observations: WorkerObservations) -> Result<()>;
|
||||
}
|
||||
|
||||
/// Replies to one execution request.
|
||||
pub trait ExecutionSender: Send + Sync {
|
||||
fn send_progress(&self, progress: JobProgressUpdate) -> Result<()>;
|
||||
fn send_completed(&self, completed: JobCompleted) -> Result<()>;
|
||||
}
|
||||
|
||||
/// Both senders write to the single outbound stream, so they share one channel.
|
||||
#[derive(Clone)]
|
||||
pub struct StreamSender {
|
||||
worker_id: String,
|
||||
tx: mpsc::UnboundedSender<WorkerToAdminMessage>,
|
||||
}
|
||||
|
||||
impl StreamSender {
|
||||
pub fn new(worker_id: String, tx: mpsc::UnboundedSender<WorkerToAdminMessage>) -> Self {
|
||||
Self { worker_id, tx }
|
||||
}
|
||||
|
||||
pub fn send(&self, body: Body) -> Result<()> {
|
||||
self.tx.send(WorkerToAdminMessage {
|
||||
worker_id: self.worker_id.clone(),
|
||||
sent_at: Some(std::time::SystemTime::now().into()),
|
||||
body: Some(body),
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl DetectionSender for StreamSender {
|
||||
fn send_proposals(&self, proposals: DetectionProposals) -> Result<()> {
|
||||
self.send(Body::DetectionProposals(proposals))
|
||||
}
|
||||
|
||||
fn send_complete(&self, complete: DetectionComplete) -> Result<()> {
|
||||
self.send(Body::DetectionComplete(complete))
|
||||
}
|
||||
|
||||
fn send_activity(&self, _activity: ActivityEvent) -> Result<()> {
|
||||
// Activity rides inside progress and completion messages rather than
|
||||
// being a body of its own, so there is nothing to send on its own here.
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn send_observations(&self, observations: WorkerObservations) -> Result<()> {
|
||||
self.send(Body::Observations(observations))
|
||||
}
|
||||
}
|
||||
|
||||
impl ExecutionSender for StreamSender {
|
||||
fn send_progress(&self, progress: JobProgressUpdate) -> Result<()> {
|
||||
self.send(Body::JobProgressUpdate(progress))
|
||||
}
|
||||
|
||||
fn send_completed(&self, completed: JobCompleted) -> Result<()> {
|
||||
self.send(Body::JobCompleted(completed))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,408 @@
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use tokio::sync::{mpsc, Semaphore};
|
||||
use tokio_stream::wrappers::UnboundedReceiverStream;
|
||||
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity};
|
||||
use tracing::{info, warn};
|
||||
|
||||
use crate::config::WorkerOptions;
|
||||
use crate::pb::{
|
||||
admin_to_worker_message::Body as AdminBody,
|
||||
plugin_control_service_client::PluginControlServiceClient,
|
||||
worker_to_admin_message::Body as WorkerBody, ConfigSchemaResponse, ExecuteJobRequest,
|
||||
JobCompleted, ObjectPreviewResponse, PreviewRow, RequestObjectPreview, RunDetectionRequest,
|
||||
RunningWork, WorkerHeartbeat, WorkerHello,
|
||||
};
|
||||
use crate::registry::Registry;
|
||||
use crate::senders::StreamSender;
|
||||
|
||||
/// The protocol version this worker speaks, sent in WorkerHello.
|
||||
const PROTOCOL_VERSION: &str = "1";
|
||||
|
||||
/// Connect to admin and serve the registry until the context is cancelled,
|
||||
/// reconnecting on failure. The stream is the only channel: everything admin
|
||||
/// asks for and everything the worker reports flows through it.
|
||||
pub async fn run(options: WorkerOptions, registry: Registry) -> Result<()> {
|
||||
if registry.is_empty() {
|
||||
return Err(anyhow!("no job handlers registered"));
|
||||
}
|
||||
if options.max_detection_concurrency < 1 || options.max_execution_concurrency < 1 {
|
||||
return Err(anyhow!(
|
||||
"concurrency limits must be at least 1, got detection={} execution={}",
|
||||
options.max_detection_concurrency,
|
||||
options.max_execution_concurrency
|
||||
));
|
||||
}
|
||||
let slots = Slots::new(&options);
|
||||
loop {
|
||||
match serve_once(&options, ®istry, &slots).await {
|
||||
Err(err) => warn!("worker stream ended: {err:#}"),
|
||||
// Admin asked this worker to stop, so stop. Reconnecting here would
|
||||
// make shutdown impossible: the worker would log back in.
|
||||
Ok(Outcome::ShutdownRequested) => return Ok(()),
|
||||
// Admin closing a healthy stream is not an error, but reconnecting
|
||||
// in silence hides the reason - two workers sharing an id evict
|
||||
// each other and produce nothing but a login every few seconds.
|
||||
Ok(Outcome::StreamClosed) => warn!(
|
||||
"admin closed the stream; reconnecting in {:?}. If this repeats, check for \
|
||||
another worker using the id {}",
|
||||
options.reconnect_delay, options.worker_id
|
||||
),
|
||||
}
|
||||
tokio::time::sleep(options.reconnect_delay).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a stream ended. Only one of these means "do not come back".
|
||||
enum Outcome {
|
||||
StreamClosed,
|
||||
ShutdownRequested,
|
||||
}
|
||||
|
||||
/// The capacity this worker advertises in WorkerHello. Admin schedules against
|
||||
/// those numbers, so the worker has to actually hold to them - and the heartbeat
|
||||
/// has to report what is in use, or admin is scheduling blind.
|
||||
#[derive(Clone)]
|
||||
struct Slots {
|
||||
detection: Arc<Semaphore>,
|
||||
execution: Arc<Semaphore>,
|
||||
detection_total: i32,
|
||||
execution_total: i32,
|
||||
}
|
||||
|
||||
impl Slots {
|
||||
fn new(options: &WorkerOptions) -> Self {
|
||||
Self {
|
||||
detection: Arc::new(Semaphore::new(options.max_detection_concurrency as usize)),
|
||||
execution: Arc::new(Semaphore::new(options.max_execution_concurrency as usize)),
|
||||
detection_total: options.max_detection_concurrency,
|
||||
execution_total: options.max_execution_concurrency,
|
||||
}
|
||||
}
|
||||
|
||||
fn detection_used(&self) -> i32 {
|
||||
self.detection_total - self.detection.available_permits() as i32
|
||||
}
|
||||
|
||||
fn execution_used(&self) -> i32 {
|
||||
self.execution_total - self.execution.available_permits() as i32
|
||||
}
|
||||
}
|
||||
|
||||
/// Dials admin, over mTLS when certificates are configured. Plaintext is the
|
||||
/// default and is fine over loopback; anything else carries preview rows and
|
||||
/// execution commands in the clear, and a cluster with grpc TLS on refuses the
|
||||
/// connection anyway.
|
||||
async fn connect(options: &WorkerOptions, grpc_address: &str) -> Result<Channel> {
|
||||
let Some(tls) = options.tls.as_ref() else {
|
||||
return Ok(Channel::from_shared(format!("http://{grpc_address}"))?
|
||||
.connect_timeout(Duration::from_secs(10))
|
||||
.connect()
|
||||
.await?);
|
||||
};
|
||||
|
||||
let ca = tokio::fs::read(&tls.ca_path)
|
||||
.await
|
||||
.with_context(|| format!("read CA certificate {}", tls.ca_path))?;
|
||||
let cert = tokio::fs::read(&tls.client_cert_path)
|
||||
.await
|
||||
.with_context(|| format!("read client certificate {}", tls.client_cert_path))?;
|
||||
let key = tokio::fs::read(&tls.client_key_path)
|
||||
.await
|
||||
.with_context(|| format!("read client key {}", tls.client_key_path))?;
|
||||
|
||||
let mut config = ClientTlsConfig::new()
|
||||
.ca_certificate(Certificate::from_pem(ca))
|
||||
.identity(Identity::from_pem(cert, key));
|
||||
if let Some(server_name) = tls.server_name.as_ref() {
|
||||
config = config.domain_name(server_name.clone());
|
||||
}
|
||||
|
||||
Ok(Channel::from_shared(format!("https://{grpc_address}"))?
|
||||
.tls_config(config)?
|
||||
.connect_timeout(Duration::from_secs(10))
|
||||
.connect()
|
||||
.await?)
|
||||
}
|
||||
|
||||
async fn serve_once(
|
||||
options: &WorkerOptions,
|
||||
registry: &Registry,
|
||||
slots: &Slots,
|
||||
) -> Result<Outcome> {
|
||||
// Operators give the admin's HTTP address; the gRPC port is derived, the
|
||||
// same way the Go worker does it.
|
||||
let grpc_address = crate::address::server_to_grpc_address(&options.admin_address)
|
||||
.ok_or_else(|| anyhow!("cannot parse admin address {}", options.admin_address))?;
|
||||
let channel = connect(options, &grpc_address).await?;
|
||||
let mut client = PluginControlServiceClient::new(channel);
|
||||
|
||||
let (tx, rx) = mpsc::unbounded_channel();
|
||||
let sender = StreamSender::new(options.worker_id.clone(), tx);
|
||||
|
||||
sender.send(WorkerBody::Hello(WorkerHello {
|
||||
worker_id: options.worker_id.clone(),
|
||||
worker_instance_id: options.worker_id.clone(),
|
||||
address: options.worker_address.clone(),
|
||||
worker_version: options.worker_version.clone(),
|
||||
protocol_version: PROTOCOL_VERSION.to_string(),
|
||||
capabilities: registry.capabilities(),
|
||||
metadata: Default::default(),
|
||||
}))?;
|
||||
|
||||
let mut inbound = client
|
||||
.worker_stream(UnboundedReceiverStream::new(rx))
|
||||
.await?
|
||||
.into_inner();
|
||||
|
||||
let heartbeat = spawn_heartbeat(sender.clone(), options.clone(), slots.clone());
|
||||
|
||||
while let Some(message) = inbound.message().await? {
|
||||
let request_id = message.request_id.clone();
|
||||
match message.body {
|
||||
Some(AdminBody::Hello(hello)) => {
|
||||
if !hello.accepted {
|
||||
return Err(anyhow!("admin rejected this worker: {}", hello.message));
|
||||
}
|
||||
info!(
|
||||
"connected to admin at {} ({})",
|
||||
options.admin_address, grpc_address
|
||||
);
|
||||
}
|
||||
Some(AdminBody::RequestConfigSchema(request)) => {
|
||||
let response = match registry.get(&request.job_type) {
|
||||
Some(handler) => ConfigSchemaResponse {
|
||||
request_id: request_id.clone(),
|
||||
job_type: request.job_type.clone(),
|
||||
success: true,
|
||||
error_message: String::new(),
|
||||
job_type_descriptor: Some(handler.descriptor()),
|
||||
},
|
||||
None => ConfigSchemaResponse {
|
||||
request_id: request_id.clone(),
|
||||
job_type: request.job_type.clone(),
|
||||
success: false,
|
||||
error_message: format!("unknown job type {}", request.job_type),
|
||||
job_type_descriptor: None,
|
||||
},
|
||||
};
|
||||
sender.send(WorkerBody::ConfigSchemaResponse(response))?;
|
||||
}
|
||||
Some(AdminBody::RequestObjectPreview(request)) => {
|
||||
spawn_preview(
|
||||
registry.clone(),
|
||||
sender.clone(),
|
||||
request_id.clone(),
|
||||
request,
|
||||
);
|
||||
}
|
||||
Some(AdminBody::RunDetectionRequest(request)) => {
|
||||
spawn_detection(registry.clone(), sender.clone(), slots.clone(), request);
|
||||
}
|
||||
Some(AdminBody::ExecuteJobRequest(request)) => {
|
||||
spawn_execution(registry.clone(), sender.clone(), slots.clone(), request);
|
||||
}
|
||||
Some(AdminBody::CancelRequest(request)) => {
|
||||
// Cancellation needs a per-request handle to be honoured; until
|
||||
// then say so rather than silently continuing to run the job.
|
||||
warn!(
|
||||
"cancel requested for {} ({}) but is not implemented",
|
||||
request.target_id, request.reason
|
||||
);
|
||||
}
|
||||
Some(AdminBody::Shutdown(shutdown)) => {
|
||||
info!("admin asked this worker to stop: {}", shutdown.reason);
|
||||
heartbeat.abort();
|
||||
return Ok(Outcome::ShutdownRequested);
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
}
|
||||
|
||||
heartbeat.abort();
|
||||
Ok(Outcome::StreamClosed)
|
||||
}
|
||||
|
||||
/// Answers one preview request off the stream loop. Reading rows takes as long
|
||||
/// as it takes, and the stream has heartbeats to keep up meanwhile.
|
||||
fn spawn_preview(
|
||||
registry: Registry,
|
||||
sender: StreamSender,
|
||||
request_id: String,
|
||||
request: RequestObjectPreview,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
let limit = request.row_limit.max(1) as usize;
|
||||
let response = match registry.preview_provider(&request.format) {
|
||||
None => ObjectPreviewResponse {
|
||||
request_id,
|
||||
success: false,
|
||||
error_message: format!("this worker does not read {} objects", request.format),
|
||||
..Default::default()
|
||||
},
|
||||
Some(provider) => match provider.preview(&request.object_id, limit).await {
|
||||
Ok(preview) => ObjectPreviewResponse {
|
||||
request_id,
|
||||
success: true,
|
||||
error_message: String::new(),
|
||||
columns: preview.columns,
|
||||
rows: preview
|
||||
.rows
|
||||
.into_iter()
|
||||
.map(|values| PreviewRow { values })
|
||||
.collect(),
|
||||
total_rows: preview.total_rows,
|
||||
},
|
||||
Err(err) => ObjectPreviewResponse {
|
||||
request_id,
|
||||
success: false,
|
||||
error_message: format!("{err:#}"),
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
};
|
||||
let _ = sender.send(WorkerBody::ObjectPreviewResponse(response));
|
||||
});
|
||||
}
|
||||
|
||||
fn spawn_heartbeat(
|
||||
sender: StreamSender,
|
||||
options: WorkerOptions,
|
||||
slots: Slots,
|
||||
) -> tokio::task::JoinHandle<()> {
|
||||
// The handle has to be the heartbeat's own, or aborting it aborts nothing
|
||||
// and every reconnect leaves another ticker running.
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(options.heartbeat_interval);
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
let beat = WorkerHeartbeat {
|
||||
worker_id: options.worker_id.clone(),
|
||||
running_work: Vec::<RunningWork>::new(),
|
||||
detection_slots_used: slots.detection_used(),
|
||||
detection_slots_total: slots.detection_total,
|
||||
execution_slots_used: slots.execution_used(),
|
||||
execution_slots_total: slots.execution_total,
|
||||
queued_jobs_by_type: Default::default(),
|
||||
metadata: Default::default(),
|
||||
};
|
||||
if sender.send(WorkerBody::Heartbeat(beat)).is_err() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn spawn_detection(
|
||||
registry: Registry,
|
||||
sender: StreamSender,
|
||||
slots: Slots,
|
||||
request: RunDetectionRequest,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
let Some(handler) = registry.get(&request.job_type) else {
|
||||
return;
|
||||
};
|
||||
// Held until the sweep finishes, so the worker keeps to the capacity it
|
||||
// advertised and the heartbeat reports the truth while it works.
|
||||
let _permit = slots.detection.acquire().await;
|
||||
if let Err(err) = handler.detect(&request, &sender).await {
|
||||
warn!("detection for {} failed: {err:#}", request.job_type);
|
||||
let _ = sender.send(WorkerBody::DetectionComplete(
|
||||
crate::pb::DetectionComplete {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: request.job_type.clone(),
|
||||
success: false,
|
||||
error_message: format!("{err:#}"),
|
||||
total_proposals: 0,
|
||||
},
|
||||
));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn spawn_execution(
|
||||
registry: Registry,
|
||||
sender: StreamSender,
|
||||
slots: Slots,
|
||||
request: ExecuteJobRequest,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
let _permit = slots.execution.acquire().await;
|
||||
let job_type = request
|
||||
.job
|
||||
.as_ref()
|
||||
.map(|job| job.job_type.clone())
|
||||
.unwrap_or_default();
|
||||
let job_id = request
|
||||
.job
|
||||
.as_ref()
|
||||
.map(|job| job.job_id.clone())
|
||||
.unwrap_or_default();
|
||||
let Some(handler) = registry.get(&job_type) else {
|
||||
return;
|
||||
};
|
||||
if let Err(err) = handler.execute(&request, &sender).await {
|
||||
warn!("job {job_id} failed: {err:#}");
|
||||
let _ = sender.send(WorkerBody::JobCompleted(JobCompleted {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id,
|
||||
job_type,
|
||||
success: false,
|
||||
error_message: format!("{err:#}"),
|
||||
..Default::default()
|
||||
}));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn options(detection: i32, execution: i32) -> WorkerOptions {
|
||||
WorkerOptions {
|
||||
max_detection_concurrency: detection,
|
||||
max_execution_concurrency: execution,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
// The heartbeat is admin's only view of how busy this worker is; before the
|
||||
// permits existed it reported zero however much was running.
|
||||
#[tokio::test]
|
||||
async fn slots_report_what_is_held() {
|
||||
let slots = Slots::new(&options(2, 3));
|
||||
assert_eq!(slots.detection_used(), 0);
|
||||
assert_eq!(slots.execution_used(), 0);
|
||||
|
||||
let held = slots.detection.acquire().await.unwrap();
|
||||
assert_eq!(slots.detection_used(), 1);
|
||||
assert_eq!(slots.execution_used(), 0, "the two do not share capacity");
|
||||
|
||||
drop(held);
|
||||
assert_eq!(slots.detection_used(), 0);
|
||||
}
|
||||
|
||||
// A limit of one means the second request waits, rather than running anyway
|
||||
// as it did when every request simply spawned a task.
|
||||
#[tokio::test]
|
||||
async fn a_full_lane_makes_the_next_request_wait() {
|
||||
let slots = Slots::new(&options(1, 1));
|
||||
let held = slots.execution.clone().acquire_owned().await.unwrap();
|
||||
assert_eq!(slots.execution_used(), 1);
|
||||
|
||||
let waiter = tokio::spawn({
|
||||
let execution = slots.execution.clone();
|
||||
async move { execution.acquire_owned().await.unwrap() }
|
||||
});
|
||||
tokio::task::yield_now().await;
|
||||
assert!(!waiter.is_finished(), "the second request must not start");
|
||||
|
||||
drop(held);
|
||||
let _second = waiter.await.expect("the waiter should be handed the slot");
|
||||
assert_eq!(slots.execution_used(), 1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
[package]
|
||||
name = "weed-lance-worker"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
description = "SeaweedFS maintenance worker for Lance tables"
|
||||
|
||||
[lib]
|
||||
name = "weed_lance_worker"
|
||||
|
||||
[[bin]]
|
||||
name = "weed-lance-worker"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
seaweed-worker-core = { path = "../core" }
|
||||
# Only the S3 backend: the other object stores lance enables by default are
|
||||
# build time this worker never spends.
|
||||
lance = { version = "10", default-features = false, features = ["aws"] }
|
||||
lance-index = "10"
|
||||
arrow-schema = "58"
|
||||
arrow-cast = "58"
|
||||
chrono = "0.4"
|
||||
futures = "0.3"
|
||||
anyhow.workspace = true
|
||||
async-trait.workspace = true
|
||||
clap = { version = "4", features = ["derive", "env"] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
tokio.workspace = true
|
||||
tracing.workspace = true
|
||||
tracing-subscriber.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tokio = { workspace = true }
|
||||
arrow-array = "58"
|
||||
arrow-schema = "58"
|
||||
arrow-cast = "58"
|
||||
lance-linalg = "10"
|
||||
@@ -0,0 +1,10 @@
|
||||
//! How the worker finds Lance tables and gets at their bytes.
|
||||
//!
|
||||
//! It goes through the Lance namespace rather than the filer: the namespace is
|
||||
//! the catalog of record, it already knows which tables are Lance, and asking it
|
||||
//! to describe a table with vend_credentials is how the worker gets storage
|
||||
//! credentials without holding any of its own.
|
||||
|
||||
pub mod namespace;
|
||||
|
||||
pub use namespace::{parse_id, NamespaceClient, TableDescription};
|
||||
@@ -0,0 +1,129 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use serde::Deserialize;
|
||||
|
||||
/// The delimiter the Lance namespace joins identifier parts with.
|
||||
const DELIMITER: &str = "$";
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct TableDescription {
|
||||
pub location: String,
|
||||
#[serde(default)]
|
||||
pub storage_options: HashMap<String, String>,
|
||||
#[serde(default)]
|
||||
pub managed_versioning: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct ListNamespacesResponse {
|
||||
#[serde(default)]
|
||||
namespaces: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct ListTablesResponse {
|
||||
#[serde(default)]
|
||||
tables: Vec<String>,
|
||||
}
|
||||
|
||||
/// A thin client for the operations this worker needs. It deliberately does not
|
||||
/// wrap the whole spec: a maintenance worker lists, describes, and commits.
|
||||
pub struct NamespaceClient {
|
||||
base_url: String,
|
||||
http: reqwest::Client,
|
||||
}
|
||||
|
||||
/// A namespace call that has not answered by now is not going to. Without this
|
||||
/// a gateway that accepts the connection and then goes quiet holds a detection
|
||||
/// slot open forever, and the sweep never finishes.
|
||||
const REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
|
||||
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
|
||||
impl NamespaceClient {
|
||||
pub fn new(base_url: impl Into<String>) -> Self {
|
||||
let http = reqwest::Client::builder()
|
||||
.timeout(REQUEST_TIMEOUT)
|
||||
.connect_timeout(CONNECT_TIMEOUT)
|
||||
.build()
|
||||
// The builder only fails on a bad TLS backend, which would break
|
||||
// every call anyway; a client with no timeouts is worse than a panic
|
||||
// at startup, so keep the default only as a last resort.
|
||||
.unwrap_or_else(|err| {
|
||||
tracing::warn!("falling back to an untimed HTTP client: {err}");
|
||||
reqwest::Client::new()
|
||||
});
|
||||
Self {
|
||||
base_url: base_url.into().trim_end_matches('/').to_string(),
|
||||
http,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every table the namespace holds, as delimiter-joined identifiers.
|
||||
pub async fn list_all_tables(&self) -> Result<Vec<String>> {
|
||||
let url = format!("{}/v1/table", self.base_url);
|
||||
let response: ListTablesResponse = self
|
||||
.http
|
||||
.get(&url)
|
||||
.send()
|
||||
.await
|
||||
.context("list tables")?
|
||||
.error_for_status()?
|
||||
.json()
|
||||
.await?;
|
||||
Ok(response.tables)
|
||||
}
|
||||
|
||||
/// Child namespaces of `id`; the root lists table buckets.
|
||||
pub async fn list_namespaces(&self, id: &[String]) -> Result<Vec<String>> {
|
||||
let url = format!("{}/v1/namespace/{}/list", self.base_url, encode_id(id));
|
||||
let response: ListNamespacesResponse = self
|
||||
.http
|
||||
.get(&url)
|
||||
.send()
|
||||
.await
|
||||
.context("list namespaces")?
|
||||
.error_for_status()?
|
||||
.json()
|
||||
.await?;
|
||||
Ok(response.namespaces)
|
||||
}
|
||||
|
||||
/// Resolve a table to a location and the credentials to reach it. The
|
||||
/// credentials expire, so a long compaction re-describes rather than
|
||||
/// carrying one set for the whole job.
|
||||
pub async fn describe_table(&self, id: &[String]) -> Result<TableDescription> {
|
||||
let url = format!("{}/v1/table/{}/describe", self.base_url, encode_id(id));
|
||||
let body = serde_json::json!({ "id": id, "vend_credentials": true });
|
||||
let description: TableDescription = self
|
||||
.http
|
||||
.post(&url)
|
||||
.json(&body)
|
||||
.send()
|
||||
.await
|
||||
.context("describe table")?
|
||||
.error_for_status()?
|
||||
.json()
|
||||
.await?;
|
||||
Ok(description)
|
||||
}
|
||||
}
|
||||
|
||||
fn encode_id(id: &[String]) -> String {
|
||||
if id.is_empty() {
|
||||
DELIMITER.to_string()
|
||||
} else {
|
||||
id.join(DELIMITER)
|
||||
}
|
||||
}
|
||||
|
||||
/// Splits a delimiter-joined identifier back into parts.
|
||||
pub fn parse_id(encoded: &str) -> Vec<String> {
|
||||
encoded
|
||||
.split(DELIMITER)
|
||||
.filter(|part| !part.is_empty())
|
||||
.map(|part| part.to_string())
|
||||
.collect()
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
//! Opening a Lance dataset with credentials the namespace vended.
|
||||
//!
|
||||
//! The worker holds no storage credentials of its own: it asks the namespace to
|
||||
//! describe a table with `vend_credentials`, and the `storage_options` that come
|
||||
//! back are handed to lance as-is. They expire, so a job that runs longer than
|
||||
//! their lifetime re-describes rather than carrying one set throughout.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use lance::dataset::builder::DatasetBuilder;
|
||||
use lance::dataset::Dataset;
|
||||
|
||||
use crate::catalog::{NamespaceClient, TableDescription};
|
||||
|
||||
/// A table the worker is about to work on.
|
||||
pub struct OpenTable {
|
||||
pub id: Vec<String>,
|
||||
pub location: String,
|
||||
pub dataset: Dataset,
|
||||
}
|
||||
|
||||
/// What detection needs to decide whether a table is worth a job. Reading it
|
||||
/// opens the dataset but touches no data files.
|
||||
pub struct TableStats {
|
||||
pub fragments: usize,
|
||||
pub version: u64,
|
||||
pub total_versions: usize,
|
||||
pub rows: usize,
|
||||
/// The Arrow schema as JSON, which is the only description of this table
|
||||
/// anything outside the format can produce.
|
||||
pub schema: Option<String>,
|
||||
}
|
||||
|
||||
/// Storage options an operator supplies for deployments that vend none.
|
||||
pub type FallbackOptions = HashMap<String, String>;
|
||||
|
||||
/// Resolve a table through the namespace and open it.
|
||||
pub async fn open(
|
||||
client: &NamespaceClient,
|
||||
id: &[String],
|
||||
fallback: &FallbackOptions,
|
||||
) -> Result<OpenTable> {
|
||||
let description = client.describe_table(id).await?;
|
||||
let dataset = open_at(&description, fallback).await?;
|
||||
Ok(OpenTable {
|
||||
id: id.to_vec(),
|
||||
location: description.location,
|
||||
dataset,
|
||||
})
|
||||
}
|
||||
|
||||
/// The namespace vends object_store's own option names, so they pass straight
|
||||
/// through. What it vends always wins: the fallback exists because a deployment
|
||||
/// without STS vends no credentials at all, and then the worker has no other way
|
||||
/// to reach the data.
|
||||
async fn open_at(description: &TableDescription, fallback: &FallbackOptions) -> Result<Dataset> {
|
||||
let mut options: HashMap<String, String> = fallback.clone();
|
||||
options.extend(description.storage_options.clone());
|
||||
DatasetBuilder::from_uri(&description.location)
|
||||
.with_storage_options(options)
|
||||
.load()
|
||||
.await
|
||||
.with_context(|| format!("open lance dataset at {}", description.location))
|
||||
}
|
||||
|
||||
impl OpenTable {
|
||||
pub async fn stats(&self) -> Result<TableStats> {
|
||||
let versions = self.dataset.versions().await?;
|
||||
Ok(TableStats {
|
||||
fragments: self.dataset.get_fragments().len(),
|
||||
version: self.dataset.version().version,
|
||||
total_versions: versions.len(),
|
||||
rows: self.dataset.count_rows(None).await.unwrap_or(0),
|
||||
schema: schema_json(&self.dataset),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Renders the dataset's schema as JSON. Best effort: a schema that will not
|
||||
/// serialise is not a reason to fail a maintenance sweep.
|
||||
fn schema_json(dataset: &Dataset) -> Option<String> {
|
||||
let arrow: arrow_schema::Schema = dataset.schema().into();
|
||||
serde_json::to_string(
|
||||
&arrow
|
||||
.fields()
|
||||
.iter()
|
||||
.map(|f| {
|
||||
serde_json::json!({
|
||||
"name": f.name(),
|
||||
"type": f.data_type().to_string(),
|
||||
"nullable": f.is_nullable(),
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>(),
|
||||
)
|
||||
.ok()
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use async_trait::async_trait;
|
||||
use chrono::{Duration, Utc};
|
||||
use lance::dataset::cleanup::{cleanup_old_versions, CleanupPolicy};
|
||||
use seaweed_worker_core::config_form::{form, int_or, int_value, number_field};
|
||||
use seaweed_worker_core::pb::{
|
||||
ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted,
|
||||
JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
|
||||
RunDetectionRequest,
|
||||
};
|
||||
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler};
|
||||
use tracing::warn;
|
||||
|
||||
use crate::catalog::{parse_id, NamespaceClient};
|
||||
use crate::dataset;
|
||||
use crate::jobs::{clamp, string_list, table_id};
|
||||
|
||||
pub const JOB_TYPE: &str = "lance_cleanup_versions";
|
||||
|
||||
const DEFAULT_RETAIN_HOURS: i64 = 168;
|
||||
const DEFAULT_MIN_VERSIONS: i64 = 5;
|
||||
|
||||
// The ranges the form offers. Duration::hours also panics far outside this one.
|
||||
const MAX_RETAIN_HOURS: i64 = 8760;
|
||||
const MAX_MIN_VERSIONS: i64 = 1000;
|
||||
|
||||
/// Lance keeps every version until something removes it. Lance can also do this
|
||||
/// itself through auto-cleanup, so this job is for deployments that would rather
|
||||
/// the cluster owned the policy than each writer.
|
||||
pub struct CleanupVersionsHandler {
|
||||
namespace_url: String,
|
||||
fallback: dataset::FallbackOptions,
|
||||
}
|
||||
|
||||
impl CleanupVersionsHandler {
|
||||
pub fn new(namespace_url: String) -> Self {
|
||||
Self {
|
||||
namespace_url,
|
||||
fallback: dataset::FallbackOptions::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self {
|
||||
self.fallback = fallback;
|
||||
self
|
||||
}
|
||||
|
||||
fn client(&self) -> NamespaceClient {
|
||||
NamespaceClient::new(self.namespace_url.clone())
|
||||
}
|
||||
}
|
||||
|
||||
/// The oldest version that may be removed while still leaving `min_versions`
|
||||
/// behind, or None when the table has no more than the floor.
|
||||
async fn version_floor(table: &dataset::OpenTable, min_versions: usize) -> Result<Option<u64>> {
|
||||
let mut versions: Vec<u64> = table
|
||||
.dataset
|
||||
.versions()
|
||||
.await?
|
||||
.iter()
|
||||
.map(|version| version.version)
|
||||
.collect();
|
||||
if versions.len() <= min_versions {
|
||||
return Ok(None);
|
||||
}
|
||||
versions.sort_unstable();
|
||||
Ok(Some(versions[versions.len() - min_versions]))
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl JobHandler for CleanupVersionsHandler {
|
||||
fn capability(&self) -> JobTypeCapability {
|
||||
JobTypeCapability {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
can_detect: true,
|
||||
can_execute: true,
|
||||
max_detection_concurrency: 1,
|
||||
max_execution_concurrency: 1,
|
||||
display_name: "Lance Version Cleanup".to_string(),
|
||||
description: "Remove old Lance versions and the files only they referenced".to_string(),
|
||||
weight: 10,
|
||||
}
|
||||
}
|
||||
|
||||
fn descriptor(&self) -> JobTypeDescriptor {
|
||||
let mut defaults: HashMap<String, ConfigValue> = HashMap::new();
|
||||
defaults.insert("retain_hours".to_string(), int_value(DEFAULT_RETAIN_HOURS));
|
||||
defaults.insert(
|
||||
"min_versions_to_keep".to_string(),
|
||||
int_value(DEFAULT_MIN_VERSIONS),
|
||||
);
|
||||
|
||||
JobTypeDescriptor {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
display_name: "Lance Version Cleanup".to_string(),
|
||||
description: "Age out Lance versions the table no longer needs".to_string(),
|
||||
icon: "fas fa-broom".to_string(),
|
||||
descriptor_version: 1,
|
||||
worker_config_form: Some(form(
|
||||
"lance-cleanup-worker",
|
||||
"Version cleanup",
|
||||
vec![
|
||||
number_field(
|
||||
"retain_hours",
|
||||
"Retain versions for (hours)",
|
||||
"Versions younger than this are always kept",
|
||||
1,
|
||||
8760,
|
||||
),
|
||||
number_field(
|
||||
"min_versions_to_keep",
|
||||
"Minimum versions",
|
||||
"Never leave a table with fewer versions than this, whatever their age",
|
||||
1,
|
||||
1000,
|
||||
),
|
||||
],
|
||||
defaults.clone(),
|
||||
)),
|
||||
worker_default_values: defaults,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
async fn detect(
|
||||
&self,
|
||||
request: &RunDetectionRequest,
|
||||
sender: &dyn DetectionSender,
|
||||
) -> Result<()> {
|
||||
let min_versions = clamp(
|
||||
int_or(
|
||||
&request.worker_config_values,
|
||||
"min_versions_to_keep",
|
||||
DEFAULT_MIN_VERSIONS,
|
||||
),
|
||||
1,
|
||||
MAX_MIN_VERSIONS,
|
||||
) as usize;
|
||||
let client = self.client();
|
||||
let tables = client.list_all_tables().await?;
|
||||
|
||||
let mut proposals = Vec::new();
|
||||
for encoded in &tables {
|
||||
let id = parse_id(encoded);
|
||||
let table = match dataset::open(&client, &id, &self.fallback).await {
|
||||
Ok(table) => table,
|
||||
Err(err) => {
|
||||
warn!("skipping {encoded}: {err:#}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let stats = match table.stats().await {
|
||||
Ok(stats) => stats,
|
||||
Err(err) => {
|
||||
warn!("skipping {encoded}: reading its stats failed: {err:#}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
// Age is decided at execution against the retention window; a table
|
||||
// at or under the floor cannot lose a version whatever its age, so
|
||||
// proposing one would only produce a job with nothing to do.
|
||||
if stats.total_versions <= min_versions {
|
||||
continue;
|
||||
}
|
||||
let mut parameters: HashMap<String, ConfigValue> = HashMap::new();
|
||||
parameters.insert("table_id".to_string(), string_list(&id));
|
||||
proposals.push(JobProposal {
|
||||
proposal_id: format!("{JOB_TYPE}:{encoded}"),
|
||||
dedupe_key: format!("{JOB_TYPE}:{encoded}"),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
summary: format!("Clean up {encoded} ({} versions)", stats.total_versions),
|
||||
detail: format!(
|
||||
"{} versions retained, above the {min_versions} floor; \
|
||||
those outside the retention window can go",
|
||||
stats.total_versions
|
||||
),
|
||||
parameters,
|
||||
..Default::default()
|
||||
});
|
||||
}
|
||||
|
||||
let total = proposals.len() as i32;
|
||||
sender.send_proposals(DetectionProposals {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
proposals,
|
||||
has_more: false,
|
||||
})?;
|
||||
sender.send_complete(DetectionComplete {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
error_message: String::new(),
|
||||
total_proposals: total,
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn execute(
|
||||
&self,
|
||||
request: &ExecuteJobRequest,
|
||||
sender: &dyn ExecutionSender,
|
||||
) -> Result<()> {
|
||||
let job = request
|
||||
.job
|
||||
.as_ref()
|
||||
.ok_or_else(|| anyhow!("execute request carried no job"))?;
|
||||
let id = table_id(&job.parameters)
|
||||
.ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?;
|
||||
let retain_hours = clamp(
|
||||
int_or(
|
||||
&request.worker_config_values,
|
||||
"retain_hours",
|
||||
DEFAULT_RETAIN_HOURS,
|
||||
),
|
||||
0,
|
||||
MAX_RETAIN_HOURS,
|
||||
);
|
||||
let min_versions = clamp(
|
||||
int_or(
|
||||
&request.worker_config_values,
|
||||
"min_versions_to_keep",
|
||||
DEFAULT_MIN_VERSIONS,
|
||||
),
|
||||
1,
|
||||
MAX_MIN_VERSIONS,
|
||||
) as usize;
|
||||
|
||||
let client = self.client();
|
||||
let table = dataset::open(&client, &id, &self.fallback).await?;
|
||||
let before = table.stats().await?;
|
||||
|
||||
// The floor is a promise about how much history survives, so it has to
|
||||
// be applied here and not only when the job was proposed: by the time it
|
||||
// runs, versions may have aged past the retention window, and age alone
|
||||
// would take the table below what the operator asked to keep.
|
||||
let Some(floor) = version_floor(&table, min_versions).await? else {
|
||||
sender.send_completed(JobCompleted {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
result: Some(JobResult {
|
||||
summary: format!(
|
||||
"kept all {} versions; the {min_versions} version floor leaves none to remove",
|
||||
before.total_versions
|
||||
),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
})?;
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
sender.send_progress(JobProgressUpdate {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
progress_percent: 10.0,
|
||||
stage: format!("cleaning up {} versions", before.total_versions),
|
||||
..Default::default()
|
||||
})?;
|
||||
|
||||
let policy = CleanupPolicy {
|
||||
before_timestamp: Some(Utc::now() - Duration::hours(retain_hours)),
|
||||
// should_clean() ANDs its clauses, so a version has to be both older
|
||||
// than the window and below the floor to go.
|
||||
before_version: Some(floor),
|
||||
// Files this dataset cannot account for are left alone: they may
|
||||
// belong to a writer that has not committed yet, and deleting them
|
||||
// would corrupt a commit in flight.
|
||||
delete_unverified: false,
|
||||
// A tagged version is pinned on purpose, so refuse rather than
|
||||
// silently dropping what someone named.
|
||||
error_if_tagged_old_versions: true,
|
||||
..Default::default()
|
||||
};
|
||||
let stats = cleanup_old_versions(&table.dataset, policy)
|
||||
.await
|
||||
.with_context(|| format!("clean up versions of {}", table.location))?;
|
||||
|
||||
let mut output: HashMap<String, ConfigValue> = HashMap::new();
|
||||
output.insert(
|
||||
"old_versions_removed".to_string(),
|
||||
int_value(stats.old_versions as i64),
|
||||
);
|
||||
output.insert(
|
||||
"bytes_removed".to_string(),
|
||||
int_value(stats.bytes_removed as i64),
|
||||
);
|
||||
output.insert(
|
||||
"data_files_removed".to_string(),
|
||||
int_value(stats.data_files_removed as i64),
|
||||
);
|
||||
|
||||
sender.send_completed(JobCompleted {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
result: Some(JobResult {
|
||||
output_values: output,
|
||||
summary: format!(
|
||||
"removed {} versions and {} bytes",
|
||||
stats.old_versions, stats.bytes_removed
|
||||
),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,296 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use async_trait::async_trait;
|
||||
use lance::dataset::optimize::{compact_files, CompactionOptions};
|
||||
use seaweed_worker_core::config_form::{form, int_or, int_value, number_field, string_value};
|
||||
use seaweed_worker_core::pb::{
|
||||
ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted,
|
||||
JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
|
||||
RunDetectionRequest, WorkerObservations,
|
||||
};
|
||||
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler};
|
||||
use tracing::warn;
|
||||
|
||||
use crate::catalog::{parse_id, NamespaceClient};
|
||||
use crate::dataset;
|
||||
use crate::jobs::{clamp, observation, string_list, table_id, FORMAT};
|
||||
|
||||
pub const JOB_TYPE: &str = "lance_compact";
|
||||
|
||||
const DEFAULT_TARGET_ROWS: i64 = 1_048_576;
|
||||
const DEFAULT_MIN_FRAGMENTS: i64 = 8;
|
||||
|
||||
// The ranges the descriptor's form offers. These are also the values that stay
|
||||
// meaningful: a table needs two fragments before merging them means anything,
|
||||
// and a fragment target below a thousand rows defeats the purpose of the job.
|
||||
// Both are cast to usize, where a negative would arrive as an enormous number.
|
||||
const TARGET_ROWS_FLOOR: i64 = 1024;
|
||||
const TARGET_ROWS_CEILING: i64 = 16_777_216;
|
||||
const MIN_FRAGMENTS_FLOOR: i64 = 2;
|
||||
const MIN_FRAGMENTS_CEILING: i64 = 4096;
|
||||
|
||||
/// Lance writes one fragment per write batch, so a table fed by small appends
|
||||
/// accumulates small files the same way an Iceberg table does.
|
||||
pub struct CompactHandler {
|
||||
namespace_url: String,
|
||||
fallback: dataset::FallbackOptions,
|
||||
}
|
||||
|
||||
impl CompactHandler {
|
||||
pub fn new(namespace_url: String) -> Self {
|
||||
Self {
|
||||
namespace_url,
|
||||
fallback: dataset::FallbackOptions::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Storage options to use where the namespace vends none.
|
||||
pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self {
|
||||
self.fallback = fallback;
|
||||
self
|
||||
}
|
||||
|
||||
fn client(&self) -> NamespaceClient {
|
||||
NamespaceClient::new(self.namespace_url.clone())
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl JobHandler for CompactHandler {
|
||||
fn capability(&self) -> JobTypeCapability {
|
||||
JobTypeCapability {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
can_detect: true,
|
||||
can_execute: true,
|
||||
max_detection_concurrency: 1,
|
||||
max_execution_concurrency: 1,
|
||||
display_name: "Lance Compaction".to_string(),
|
||||
description: "Merge small Lance fragments into fewer, larger ones".to_string(),
|
||||
weight: 20,
|
||||
}
|
||||
}
|
||||
|
||||
fn descriptor(&self) -> JobTypeDescriptor {
|
||||
let mut defaults: HashMap<String, ConfigValue> = HashMap::new();
|
||||
defaults.insert(
|
||||
"target_rows_per_fragment".to_string(),
|
||||
int_value(DEFAULT_TARGET_ROWS),
|
||||
);
|
||||
defaults.insert(
|
||||
"min_fragments".to_string(),
|
||||
int_value(DEFAULT_MIN_FRAGMENTS),
|
||||
);
|
||||
|
||||
JobTypeDescriptor {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
display_name: "Lance Compaction".to_string(),
|
||||
description: "Compact fragments of Lance tables".to_string(),
|
||||
icon: "fas fa-compress".to_string(),
|
||||
descriptor_version: 1,
|
||||
worker_config_form: Some(form(
|
||||
"lance-compact-worker",
|
||||
"Compaction",
|
||||
vec![
|
||||
number_field(
|
||||
"target_rows_per_fragment",
|
||||
"Target rows per fragment",
|
||||
"Rows to aim for when rewriting fragments",
|
||||
1024,
|
||||
16_777_216,
|
||||
),
|
||||
number_field(
|
||||
"min_fragments",
|
||||
"Minimum fragments",
|
||||
"Leave a table alone until it has at least this many fragments",
|
||||
2,
|
||||
4096,
|
||||
),
|
||||
],
|
||||
defaults.clone(),
|
||||
)),
|
||||
worker_default_values: defaults,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Propose a job for every table with more fragments than the operator is
|
||||
/// willing to leave alone. Opening a dataset reads its manifest, not its
|
||||
/// data, so this stays cheap across a catalog.
|
||||
async fn detect(
|
||||
&self,
|
||||
request: &RunDetectionRequest,
|
||||
sender: &dyn DetectionSender,
|
||||
) -> Result<()> {
|
||||
let min_fragments = clamp(
|
||||
int_or(
|
||||
&request.worker_config_values,
|
||||
"min_fragments",
|
||||
DEFAULT_MIN_FRAGMENTS,
|
||||
),
|
||||
MIN_FRAGMENTS_FLOOR,
|
||||
MIN_FRAGMENTS_CEILING,
|
||||
) as usize;
|
||||
let client = self.client();
|
||||
let tables = client.list_all_tables().await?;
|
||||
|
||||
let mut proposals = Vec::new();
|
||||
let mut observations = Vec::new();
|
||||
for encoded in &tables {
|
||||
let id = parse_id(encoded);
|
||||
let table = match dataset::open(&client, &id, &self.fallback).await {
|
||||
Ok(table) => table,
|
||||
Err(err) => {
|
||||
// A table that cannot be opened is the next run's problem,
|
||||
// not a reason to abandon the whole sweep.
|
||||
warn!("skipping {encoded}: {err:#}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
// One unreadable table must not end the sweep: the tables already
|
||||
// read would lose their proposals, and admin would get no
|
||||
// completion for this request at all.
|
||||
let stats = match table.stats().await {
|
||||
Ok(stats) => stats,
|
||||
Err(err) => {
|
||||
warn!("skipping {encoded}: reading its stats failed: {err:#}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
// Logged because "detection proposed nothing" is otherwise
|
||||
// indistinguishable from a table the worker could not read.
|
||||
tracing::info!(
|
||||
"compaction detection: {encoded} has {} fragments, threshold {min_fragments}",
|
||||
stats.fragments
|
||||
);
|
||||
|
||||
let mut attributes: HashMap<String, ConfigValue> = HashMap::new();
|
||||
attributes.insert("fragments".to_string(), int_value(stats.fragments as i64));
|
||||
attributes.insert("version".to_string(), int_value(stats.version as i64));
|
||||
attributes.insert(
|
||||
"versions".to_string(),
|
||||
int_value(stats.total_versions as i64),
|
||||
);
|
||||
attributes.insert("rows".to_string(), int_value(stats.rows as i64));
|
||||
if let Some(schema) = stats.schema.clone() {
|
||||
attributes.insert("schema".to_string(), string_value(schema));
|
||||
}
|
||||
observations.push(observation(&id, FORMAT, attributes));
|
||||
|
||||
if stats.fragments < min_fragments {
|
||||
continue;
|
||||
}
|
||||
let mut parameters: HashMap<String, ConfigValue> = HashMap::new();
|
||||
parameters.insert("table_id".to_string(), string_list(&id));
|
||||
proposals.push(JobProposal {
|
||||
proposal_id: format!("{JOB_TYPE}:{encoded}"),
|
||||
dedupe_key: format!("{JOB_TYPE}:{encoded}"),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
summary: format!("Compact {encoded} ({} fragments)", stats.fragments),
|
||||
detail: format!(
|
||||
"{} fragments at version {}, above the {min_fragments} the policy allows",
|
||||
stats.fragments, stats.version
|
||||
),
|
||||
parameters,
|
||||
..Default::default()
|
||||
});
|
||||
}
|
||||
|
||||
if !observations.is_empty() {
|
||||
sender.send_observations(WorkerObservations {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
observations,
|
||||
})?;
|
||||
}
|
||||
|
||||
let total = proposals.len() as i32;
|
||||
sender.send_proposals(DetectionProposals {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
proposals,
|
||||
has_more: false,
|
||||
})?;
|
||||
sender.send_complete(DetectionComplete {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
error_message: String::new(),
|
||||
total_proposals: total,
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn execute(
|
||||
&self,
|
||||
request: &ExecuteJobRequest,
|
||||
sender: &dyn ExecutionSender,
|
||||
) -> Result<()> {
|
||||
let job = request
|
||||
.job
|
||||
.as_ref()
|
||||
.ok_or_else(|| anyhow!("execute request carried no job"))?;
|
||||
let id = table_id(&job.parameters)
|
||||
.ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?;
|
||||
let target_rows = clamp(
|
||||
int_or(
|
||||
&request.worker_config_values,
|
||||
"target_rows_per_fragment",
|
||||
DEFAULT_TARGET_ROWS,
|
||||
),
|
||||
TARGET_ROWS_FLOOR,
|
||||
TARGET_ROWS_CEILING,
|
||||
) as usize;
|
||||
|
||||
let client = self.client();
|
||||
// Re-resolve rather than trusting the location detection saw: the table
|
||||
// may have been repointed, and the vended credentials have expired.
|
||||
let mut table = dataset::open(&client, &id, &self.fallback).await?;
|
||||
let before = table.stats().await?;
|
||||
|
||||
sender.send_progress(JobProgressUpdate {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
progress_percent: 10.0,
|
||||
stage: format!("compacting {} fragments", before.fragments),
|
||||
..Default::default()
|
||||
})?;
|
||||
|
||||
let options = CompactionOptions {
|
||||
target_rows_per_fragment: target_rows,
|
||||
..Default::default()
|
||||
};
|
||||
let metrics = compact_files(&mut table.dataset, options, None)
|
||||
.await
|
||||
.with_context(|| format!("compact {}", table.location))?;
|
||||
|
||||
let after = table.stats().await?;
|
||||
let mut output: HashMap<String, ConfigValue> = HashMap::new();
|
||||
output.insert(
|
||||
"fragments_removed".to_string(),
|
||||
int_value(metrics.fragments_removed as i64),
|
||||
);
|
||||
output.insert(
|
||||
"fragments_added".to_string(),
|
||||
int_value(metrics.fragments_added as i64),
|
||||
);
|
||||
output.insert(
|
||||
"files_removed".to_string(),
|
||||
int_value(metrics.files_removed as i64),
|
||||
);
|
||||
|
||||
sender.send_completed(JobCompleted {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
result: Some(JobResult {
|
||||
output_values: output,
|
||||
summary: format!("{} fragments became {}", before.fragments, after.fragments),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use async_trait::async_trait;
|
||||
use lance::index::DatasetIndexExt;
|
||||
use lance_index::optimize::OptimizeOptions;
|
||||
use seaweed_worker_core::config_form::{form, int_or, int_value, number_field};
|
||||
use seaweed_worker_core::pb::{
|
||||
ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest, JobCompleted,
|
||||
JobProgressUpdate, JobProposal, JobResult, JobTypeCapability, JobTypeDescriptor,
|
||||
RunDetectionRequest,
|
||||
};
|
||||
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler};
|
||||
use tracing::warn;
|
||||
|
||||
use crate::catalog::{parse_id, NamespaceClient};
|
||||
use crate::dataset::{self, OpenTable};
|
||||
use crate::jobs::{clamp, string_list, table_id};
|
||||
|
||||
pub const JOB_TYPE: &str = "lance_optimize_indices";
|
||||
|
||||
const DEFAULT_MAX_UNINDEXED_ROWS: i64 = 100_000;
|
||||
|
||||
// Zero is a real setting - reindex as soon as any row is uncovered - so the
|
||||
// floor is what stays meaningful rather than what the form offers. The ceiling
|
||||
// is the form's, and the point of both is that this is cast to u64: a negative
|
||||
// would arrive as an enormous budget and mean "never reindex".
|
||||
const MAX_UNINDEXED_FLOOR: i64 = 0;
|
||||
const MAX_UNINDEXED_CEILING: i64 = 100_000_000;
|
||||
|
||||
/// Rows written after an index was built are not covered by it, so a vector
|
||||
/// search quietly misses them. This is the job with no Iceberg equivalent, and
|
||||
/// the reason a neglected Lance table is a correctness problem rather than a
|
||||
/// slow one.
|
||||
pub struct OptimizeIndicesHandler {
|
||||
namespace_url: String,
|
||||
fallback: dataset::FallbackOptions,
|
||||
}
|
||||
|
||||
impl OptimizeIndicesHandler {
|
||||
pub fn new(namespace_url: String) -> Self {
|
||||
Self {
|
||||
namespace_url,
|
||||
fallback: dataset::FallbackOptions::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_fallback(mut self, fallback: dataset::FallbackOptions) -> Self {
|
||||
self.fallback = fallback;
|
||||
self
|
||||
}
|
||||
|
||||
fn client(&self) -> NamespaceClient {
|
||||
NamespaceClient::new(self.namespace_url.clone())
|
||||
}
|
||||
}
|
||||
|
||||
/// Rows no index covers, summed across a table's indices. A table with no
|
||||
/// indices at all has nothing to optimize, which is different from a table whose
|
||||
/// indices have fallen behind.
|
||||
async fn unindexed_rows(table: &OpenTable) -> Result<Option<u64>> {
|
||||
let indices = table.dataset.load_indices().await?;
|
||||
if indices.is_empty() {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let mut worst = 0u64;
|
||||
let mut names: Vec<String> = indices.iter().map(|index| index.name.clone()).collect();
|
||||
names.sort();
|
||||
names.dedup();
|
||||
for name in names {
|
||||
let raw = table.dataset.index_statistics(&name).await?;
|
||||
let stats: serde_json::Value = serde_json::from_str(&raw)
|
||||
.with_context(|| format!("parse index statistics for {name}"))?;
|
||||
let unindexed = stats
|
||||
.get("num_unindexed_rows")
|
||||
.and_then(|value| value.as_u64())
|
||||
.unwrap_or(0);
|
||||
worst = worst.max(unindexed);
|
||||
}
|
||||
Ok(Some(worst))
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl JobHandler for OptimizeIndicesHandler {
|
||||
fn capability(&self) -> JobTypeCapability {
|
||||
JobTypeCapability {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
can_detect: true,
|
||||
can_execute: true,
|
||||
max_detection_concurrency: 1,
|
||||
max_execution_concurrency: 1,
|
||||
display_name: "Lance Index Optimization".to_string(),
|
||||
description: "Extend indices to cover rows written since they were built".to_string(),
|
||||
weight: 30,
|
||||
}
|
||||
}
|
||||
|
||||
fn descriptor(&self) -> JobTypeDescriptor {
|
||||
let mut defaults: HashMap<String, ConfigValue> = HashMap::new();
|
||||
defaults.insert(
|
||||
"max_unindexed_rows".to_string(),
|
||||
int_value(DEFAULT_MAX_UNINDEXED_ROWS),
|
||||
);
|
||||
|
||||
JobTypeDescriptor {
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
display_name: "Lance Index Optimization".to_string(),
|
||||
description: "Keep vector and scalar indices covering the whole table".to_string(),
|
||||
icon: "fas fa-magnifying-glass-chart".to_string(),
|
||||
descriptor_version: 1,
|
||||
worker_config_form: Some(form(
|
||||
"lance-indices-worker",
|
||||
"Index optimization",
|
||||
vec![number_field(
|
||||
"max_unindexed_rows",
|
||||
"Unindexed row budget",
|
||||
"Reindex once a table has more rows than this outside its indices",
|
||||
1_000,
|
||||
100_000_000,
|
||||
)],
|
||||
defaults.clone(),
|
||||
)),
|
||||
worker_default_values: defaults,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
async fn detect(
|
||||
&self,
|
||||
request: &RunDetectionRequest,
|
||||
sender: &dyn DetectionSender,
|
||||
) -> Result<()> {
|
||||
let budget = clamp(
|
||||
int_or(
|
||||
&request.worker_config_values,
|
||||
"max_unindexed_rows",
|
||||
DEFAULT_MAX_UNINDEXED_ROWS,
|
||||
),
|
||||
MAX_UNINDEXED_FLOOR,
|
||||
MAX_UNINDEXED_CEILING,
|
||||
) as u64;
|
||||
let client = self.client();
|
||||
let tables = client.list_all_tables().await?;
|
||||
|
||||
let mut proposals = Vec::new();
|
||||
for encoded in &tables {
|
||||
let id = parse_id(encoded);
|
||||
let table = match dataset::open(&client, &id, &self.fallback).await {
|
||||
Ok(table) => table,
|
||||
Err(err) => {
|
||||
warn!("skipping {encoded}: {err:#}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let unindexed = match unindexed_rows(&table).await {
|
||||
Ok(Some(unindexed)) => unindexed,
|
||||
Ok(None) => continue,
|
||||
Err(err) => {
|
||||
warn!("skipping {encoded}: reading its index stats failed: {err:#}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if unindexed <= budget {
|
||||
continue;
|
||||
}
|
||||
let mut parameters: HashMap<String, ConfigValue> = HashMap::new();
|
||||
parameters.insert("table_id".to_string(), string_list(&id));
|
||||
proposals.push(JobProposal {
|
||||
proposal_id: format!("{JOB_TYPE}:{encoded}"),
|
||||
dedupe_key: format!("{JOB_TYPE}:{encoded}"),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
summary: format!("Reindex {encoded} ({unindexed} rows uncovered)"),
|
||||
detail: format!(
|
||||
"{unindexed} rows sit outside the indices, above the {budget} the policy allows; \
|
||||
a search of this table misses them"
|
||||
),
|
||||
parameters,
|
||||
..Default::default()
|
||||
});
|
||||
}
|
||||
|
||||
let total = proposals.len() as i32;
|
||||
sender.send_proposals(DetectionProposals {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
proposals,
|
||||
has_more: false,
|
||||
})?;
|
||||
sender.send_complete(DetectionComplete {
|
||||
request_id: request.request_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
error_message: String::new(),
|
||||
total_proposals: total,
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn execute(
|
||||
&self,
|
||||
request: &ExecuteJobRequest,
|
||||
sender: &dyn ExecutionSender,
|
||||
) -> Result<()> {
|
||||
let job = request
|
||||
.job
|
||||
.as_ref()
|
||||
.ok_or_else(|| anyhow!("execute request carried no job"))?;
|
||||
let id = table_id(&job.parameters)
|
||||
.ok_or_else(|| anyhow!("job {} carried no table_id", job.job_id))?;
|
||||
|
||||
let client = self.client();
|
||||
let mut table = dataset::open(&client, &id, &self.fallback).await?;
|
||||
let before = unindexed_rows(&table).await?.unwrap_or(0);
|
||||
|
||||
sender.send_progress(JobProgressUpdate {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
progress_percent: 10.0,
|
||||
stage: format!("indexing {before} uncovered rows"),
|
||||
..Default::default()
|
||||
})?;
|
||||
|
||||
// Merging every delta keeps read latency from drifting as the table is
|
||||
// reindexed again and again; leaving them unmerged is how an index ends
|
||||
// up fast to write and slow to search.
|
||||
table
|
||||
.dataset
|
||||
.optimize_indices(&OptimizeOptions::default())
|
||||
.await
|
||||
.with_context(|| format!("optimize indices of {}", table.location))?;
|
||||
|
||||
let after = unindexed_rows(&table).await?.unwrap_or(0);
|
||||
let mut output: HashMap<String, ConfigValue> = HashMap::new();
|
||||
output.insert(
|
||||
"unindexed_rows_before".to_string(),
|
||||
int_value(before as i64),
|
||||
);
|
||||
output.insert("unindexed_rows_after".to_string(), int_value(after as i64));
|
||||
|
||||
sender.send_completed(JobCompleted {
|
||||
request_id: request.request_id.clone(),
|
||||
job_id: job.job_id.clone(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
success: true,
|
||||
result: Some(JobResult {
|
||||
output_values: output,
|
||||
summary: format!("{before} uncovered rows became {after}"),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
//! One module per job type. Each declares its capability and the settings form
|
||||
//! admin renders for it, then does the work.
|
||||
//!
|
||||
//! Detection opens each table and reads its manifest rather than its data, so a
|
||||
//! sweep across a catalog stays cheap. Execution re-resolves the table instead
|
||||
//! of trusting what detection saw: it may have been repointed, and the vended
|
||||
//! credentials expire.
|
||||
|
||||
pub mod cleanup;
|
||||
pub mod compact;
|
||||
pub mod indices;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use seaweed_worker_core::pb::{config_value::Kind, ConfigValue, ObjectObservation, StringList};
|
||||
use seaweed_worker_core::JobHandler;
|
||||
|
||||
use crate::catalog::parse_id;
|
||||
|
||||
/// A table identifier travels in a proposal's parameters and comes back on the
|
||||
/// job, so both sides agree on one encoding.
|
||||
pub(crate) fn string_list(parts: &[String]) -> ConfigValue {
|
||||
ConfigValue {
|
||||
kind: Some(Kind::StringList(StringList {
|
||||
values: parts.to_vec(),
|
||||
})),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn table_id(parameters: &HashMap<String, ConfigValue>) -> Option<Vec<String>> {
|
||||
match parameters.get("table_id")?.kind.as_ref()? {
|
||||
Kind::StringList(list) => Some(list.values.clone()),
|
||||
Kind::StringValue(encoded) => Some(parse_id(encoded)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every handler this worker serves. A worker process may serve several job
|
||||
/// types, which is why WorkerHello carries a list.
|
||||
pub fn handlers(
|
||||
namespace_url: String,
|
||||
fallback: crate::dataset::FallbackOptions,
|
||||
) -> Vec<Arc<dyn JobHandler>> {
|
||||
vec![
|
||||
Arc::new(
|
||||
compact::CompactHandler::new(namespace_url.clone()).with_fallback(fallback.clone()),
|
||||
),
|
||||
Arc::new(
|
||||
indices::OptimizeIndicesHandler::new(namespace_url.clone())
|
||||
.with_fallback(fallback.clone()),
|
||||
),
|
||||
Arc::new(cleanup::CleanupVersionsHandler::new(namespace_url).with_fallback(fallback)),
|
||||
]
|
||||
}
|
||||
|
||||
/// Holds a configured value to the range its form offers. A value from outside
|
||||
/// it is one the UI could not have produced, and every one of these is cast to
|
||||
/// an unsigned type: a negative arrives as an enormous number, which silently
|
||||
/// turns a threshold into "never" rather than failing loudly.
|
||||
pub(crate) fn clamp(value: i64, low: i64, high: i64) -> i64 {
|
||||
value.max(low).min(high)
|
||||
}
|
||||
|
||||
/// The format the catalog records for the tables this worker maintains.
|
||||
pub const FORMAT: &str = "LANCE";
|
||||
|
||||
/// Builds the observation a detection sweep reports for one table. Detection
|
||||
/// has already opened the dataset to decide whether it needs work, so saying
|
||||
/// what it saw costs nothing, and for a format the cluster cannot read this is
|
||||
/// the only description of the table anything can produce.
|
||||
pub(crate) fn observation(
|
||||
id: &[String],
|
||||
format: &str,
|
||||
attributes: HashMap<String, ConfigValue>,
|
||||
) -> ObjectObservation {
|
||||
ObjectObservation {
|
||||
object_id: id.to_vec(),
|
||||
object_kind: "table".to_string(),
|
||||
format: format.to_string(),
|
||||
attributes,
|
||||
observed_at: Some(std::time::SystemTime::now().into()),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// Every configured threshold is cast to an unsigned type before use. A
|
||||
// negative one would arrive as an enormous number and quietly mean "never",
|
||||
// which looks exactly like a worker with nothing to do.
|
||||
#[test]
|
||||
fn clamp_keeps_a_negative_from_wrapping() {
|
||||
assert_eq!(clamp(-1, 2, 4096) as usize, 2);
|
||||
assert_eq!(clamp(i64::MIN, 0, 100_000_000) as u64, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clamp_holds_the_ceiling_and_passes_the_middle() {
|
||||
assert_eq!(clamp(i64::MAX, 0, 8760), 8760);
|
||||
assert_eq!(clamp(168, 0, 8760), 168);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
//! Maintenance jobs for Lance tables.
|
||||
//!
|
||||
//! A Lance dataset needs three things done to it over time: its fragments
|
||||
//! compacted, its indices extended to cover rows written after they were built,
|
||||
//! and its old versions removed. None can run in the Go worker, because all
|
||||
//! three read and rewrite Lance files. This crate is the worker that can.
|
||||
|
||||
pub mod catalog;
|
||||
pub mod dataset;
|
||||
pub mod jobs;
|
||||
pub mod preview;
|
||||
|
||||
pub use jobs::handlers;
|
||||
@@ -0,0 +1,127 @@
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::Result;
|
||||
use clap::Parser;
|
||||
use std::sync::Arc;
|
||||
|
||||
use seaweed_worker_core::{Registry, TlsOptions, WorkerOptions};
|
||||
use weed_lance_worker::handlers;
|
||||
|
||||
/// Mirrors `weed worker`'s flags, because this is the same contract from another
|
||||
/// language and an operator should not have to learn a second set of names.
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(
|
||||
name = "weed-lance-worker",
|
||||
about = "SeaweedFS maintenance worker for Lance tables"
|
||||
)]
|
||||
struct Args {
|
||||
/// Admin server gRPC address.
|
||||
#[arg(long, default_value = "localhost:23646", env = "WEED_ADMIN")]
|
||||
admin: String,
|
||||
|
||||
/// Worker identity reported to admin.
|
||||
#[arg(long, default_value = "lance-worker", env = "WEED_WORKER_ID")]
|
||||
id: String,
|
||||
|
||||
/// Lance namespace the worker lists tables from.
|
||||
#[arg(
|
||||
long,
|
||||
default_value = "http://localhost:9101",
|
||||
env = "WEED_LANCE_NAMESPACE"
|
||||
)]
|
||||
namespace: String,
|
||||
|
||||
#[arg(long, default_value = "10", env = "WEED_HEARTBEAT_SECONDS")]
|
||||
heartbeat_seconds: u64,
|
||||
|
||||
#[arg(long, default_value = "1")]
|
||||
max_concurrency: i32,
|
||||
|
||||
/// Storage credentials to use where the namespace vends none. A gateway
|
||||
/// with STS configured vends its own and these are ignored.
|
||||
#[arg(long, env = "WEED_S3_ACCESS_KEY")]
|
||||
access_key: Option<String>,
|
||||
|
||||
#[arg(long, env = "WEED_S3_SECRET_KEY")]
|
||||
secret_key: Option<String>,
|
||||
|
||||
/// mTLS for the admin stream, the same certificates the Go worker reads
|
||||
/// from the [grpc.worker] section of security.toml. All three together, or
|
||||
/// none, in which case the stream is plaintext.
|
||||
#[arg(long, env = "WEED_GRPC_CA")]
|
||||
tls_ca: Option<String>,
|
||||
|
||||
#[arg(long, env = "WEED_GRPC_CLIENT_CERT")]
|
||||
tls_cert: Option<String>,
|
||||
|
||||
#[arg(long, env = "WEED_GRPC_CLIENT_KEY")]
|
||||
tls_key: Option<String>,
|
||||
|
||||
/// Name to verify admin's certificate against, when it is not the address
|
||||
/// this worker dials.
|
||||
#[arg(long)]
|
||||
tls_server_name: Option<String>,
|
||||
}
|
||||
|
||||
impl Args {
|
||||
/// The TLS configuration, or an error when the three certificate paths do
|
||||
/// not arrive together: a CA on its own would silently give one-way TLS,
|
||||
/// which the cluster's mutual setup refuses anyway.
|
||||
fn tls(&self) -> Result<Option<TlsOptions>> {
|
||||
match (
|
||||
self.tls_ca.clone(),
|
||||
self.tls_cert.clone(),
|
||||
self.tls_key.clone(),
|
||||
) {
|
||||
(None, None, None) => Ok(None),
|
||||
(Some(ca_path), Some(client_cert_path), Some(client_key_path)) => {
|
||||
Ok(Some(TlsOptions {
|
||||
ca_path,
|
||||
client_cert_path,
|
||||
client_key_path,
|
||||
server_name: self.tls_server_name.clone(),
|
||||
}))
|
||||
}
|
||||
_ => Err(anyhow::anyhow!(
|
||||
"--tls-ca, --tls-cert and --tls-key must be given together"
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
.with_env_filter(
|
||||
tracing_subscriber::EnvFilter::try_from_default_env()
|
||||
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
|
||||
)
|
||||
.init();
|
||||
|
||||
let args = Args::parse();
|
||||
let tls = args.tls()?;
|
||||
let options = WorkerOptions {
|
||||
admin_address: args.admin,
|
||||
worker_id: args.id,
|
||||
heartbeat_interval: Duration::from_secs(args.heartbeat_seconds),
|
||||
max_detection_concurrency: args.max_concurrency,
|
||||
max_execution_concurrency: args.max_concurrency,
|
||||
tls,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let mut fallback = weed_lance_worker::dataset::FallbackOptions::new();
|
||||
if let (Some(access), Some(secret)) = (args.access_key, args.secret_key) {
|
||||
fallback.insert("aws_access_key_id".to_string(), access);
|
||||
fallback.insert("aws_secret_access_key".to_string(), secret);
|
||||
}
|
||||
|
||||
let mut registry = Registry::new().with_preview(Arc::new(
|
||||
weed_lance_worker::preview::LancePreview::new(args.namespace.clone(), fallback.clone()),
|
||||
));
|
||||
for handler in handlers(args.namespace, fallback) {
|
||||
registry = registry.register(handler);
|
||||
}
|
||||
|
||||
seaweed_worker_core::run(options, registry).await
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
//! Sample rows of a Lance table, for a page that cannot read the format.
|
||||
//!
|
||||
//! Admin renders an Iceberg table by reading its Parquet files directly. There
|
||||
//! is no Go Lance reader, so for Lance the worker does the reading and hands
|
||||
//! back text.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use arrow_cast::display::{ArrayFormatter, FormatOptions};
|
||||
use async_trait::async_trait;
|
||||
use futures::TryStreamExt;
|
||||
use seaweed_worker_core::{Preview, PreviewProvider};
|
||||
|
||||
use crate::catalog::NamespaceClient;
|
||||
use crate::dataset::{self, FallbackOptions};
|
||||
use crate::jobs::FORMAT;
|
||||
|
||||
pub struct LancePreview {
|
||||
namespace_url: String,
|
||||
fallback: FallbackOptions,
|
||||
}
|
||||
|
||||
impl LancePreview {
|
||||
pub fn new(namespace_url: String, fallback: FallbackOptions) -> Self {
|
||||
Self {
|
||||
namespace_url,
|
||||
fallback,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PreviewProvider for LancePreview {
|
||||
fn format(&self) -> &str {
|
||||
FORMAT
|
||||
}
|
||||
|
||||
async fn preview(&self, object_id: &[String], row_limit: usize) -> Result<Preview> {
|
||||
let client = NamespaceClient::new(self.namespace_url.clone());
|
||||
let table = dataset::open(&client, object_id, &self.fallback).await?;
|
||||
|
||||
let total_rows = table.dataset.count_rows(None).await.unwrap_or(0) as i64;
|
||||
let mut scanner = table.dataset.scan();
|
||||
scanner.limit(Some(row_limit as i64), None)?;
|
||||
|
||||
let batches: Vec<_> = scanner
|
||||
.try_into_stream()
|
||||
.await?
|
||||
.try_collect()
|
||||
.await
|
||||
.with_context(|| format!("read rows from {}", table.location))?;
|
||||
|
||||
let columns = table
|
||||
.dataset
|
||||
.schema()
|
||||
.fields
|
||||
.iter()
|
||||
.map(|field| field.name.clone())
|
||||
.collect();
|
||||
|
||||
let mut rows = Vec::new();
|
||||
for batch in &batches {
|
||||
// ArrayFormatter renders each Arrow type the way its own tooling
|
||||
// does, so a vector column reads as a vector rather than as bytes.
|
||||
let formatters = batch
|
||||
.columns()
|
||||
.iter()
|
||||
.map(|array| ArrayFormatter::try_new(array.as_ref(), &FormatOptions::default()))
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
for index in 0..batch.num_rows() {
|
||||
rows.push(
|
||||
formatters
|
||||
.iter()
|
||||
.map(|formatter| formatter.value(index).to_string())
|
||||
.collect(),
|
||||
);
|
||||
}
|
||||
}
|
||||
rows.truncate(row_limit);
|
||||
|
||||
Ok(Preview {
|
||||
columns,
|
||||
rows,
|
||||
total_rows,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,606 @@
|
||||
//! Drives the compaction handler against a live namespace.
|
||||
//!
|
||||
//! Skipped unless WEED_LANCE_NAMESPACE names one, the way the Go integration
|
||||
//! tests skip without Docker: compaction rewrites real files, and there is
|
||||
//! nothing to learn from it against a fake.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Mutex;
|
||||
|
||||
use anyhow::Result;
|
||||
use seaweed_worker_core::pb::{
|
||||
config_value::Kind, ConfigValue, DetectionComplete, DetectionProposals, ExecuteJobRequest,
|
||||
JobCompleted, JobProgressUpdate, JobProposal, JobSpec, RunDetectionRequest,
|
||||
};
|
||||
use seaweed_worker_core::{DetectionSender, ExecutionSender, JobHandler, PreviewProvider};
|
||||
use weed_lance_worker::catalog::NamespaceClient;
|
||||
use weed_lance_worker::jobs::cleanup::CleanupVersionsHandler;
|
||||
use weed_lance_worker::jobs::compact::{CompactHandler, JOB_TYPE};
|
||||
use weed_lance_worker::jobs::indices::OptimizeIndicesHandler;
|
||||
use weed_lance_worker::preview::LancePreview;
|
||||
|
||||
#[derive(Default)]
|
||||
struct Recorder {
|
||||
proposals: Mutex<Vec<JobProposal>>,
|
||||
observations: Mutex<Vec<seaweed_worker_core::pb::ObjectObservation>>,
|
||||
completed: Mutex<Vec<JobCompleted>>,
|
||||
}
|
||||
|
||||
impl DetectionSender for Recorder {
|
||||
fn send_proposals(&self, proposals: DetectionProposals) -> Result<()> {
|
||||
self.proposals.lock().unwrap().extend(proposals.proposals);
|
||||
Ok(())
|
||||
}
|
||||
fn send_complete(&self, _complete: DetectionComplete) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
fn send_activity(&self, _activity: seaweed_worker_core::pb::ActivityEvent) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
fn send_observations(
|
||||
&self,
|
||||
observations: seaweed_worker_core::pb::WorkerObservations,
|
||||
) -> Result<()> {
|
||||
self.observations
|
||||
.lock()
|
||||
.unwrap()
|
||||
.extend(observations.observations);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl ExecutionSender for Recorder {
|
||||
fn send_progress(&self, _progress: JobProgressUpdate) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
fn send_completed(&self, completed: JobCompleted) -> Result<()> {
|
||||
self.completed.lock().unwrap().push(completed);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// These tests drive one live gateway and one shared catalog: `list_all_tables`
|
||||
/// sweeps everything, so a table another test is writing shows up in this test's
|
||||
/// detection. Rust runs a binary's tests concurrently, so take a lock.
|
||||
static GATEWAY: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
|
||||
|
||||
fn namespace_url() -> Option<String> {
|
||||
std::env::var("WEED_LANCE_NAMESPACE")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty())
|
||||
}
|
||||
|
||||
fn int_config(name: &str, value: i64) -> HashMap<String, ConfigValue> {
|
||||
let mut values = HashMap::new();
|
||||
values.insert(
|
||||
name.to_string(),
|
||||
ConfigValue {
|
||||
kind: Some(Kind::Int64Value(value)),
|
||||
},
|
||||
);
|
||||
values
|
||||
}
|
||||
|
||||
/// Declares a table through the namespace and writes `fragments` one-row
|
||||
/// appends into it, so a test brings its own state instead of depending on
|
||||
/// whatever a previous run left behind.
|
||||
async fn seed_fragmented_table(url: &str, name: &str, fragments: usize) -> Result<String> {
|
||||
seed_table(url, name, fragments, 1, false).await
|
||||
}
|
||||
|
||||
/// Writes `batches` appends of `rows_each` into a freshly declared table, and
|
||||
/// optionally builds a vector index after the first batch so the later ones are
|
||||
/// rows no index covers.
|
||||
async fn seed_table(
|
||||
url: &str,
|
||||
name: &str,
|
||||
batches: usize,
|
||||
rows_each: usize,
|
||||
with_index: bool,
|
||||
) -> Result<String> {
|
||||
use arrow_array::{
|
||||
FixedSizeListArray, Float32Array, Int64Array, RecordBatch, RecordBatchIterator,
|
||||
};
|
||||
use arrow_schema::{DataType, Field, Schema};
|
||||
use lance::dataset::{Dataset, WriteMode, WriteParams};
|
||||
use lance::io::{ObjectStoreParams, StorageOptionsAccessor};
|
||||
use std::sync::Arc;
|
||||
|
||||
// Declaring is the namespace's job, not the worker's, so the test asks for
|
||||
// it directly rather than widening the client the worker uses. The bucket and
|
||||
// namespace come first: a table cannot be declared under a parent that does
|
||||
// not exist, and a test that assumes one is a test that only passes twice.
|
||||
let http = reqwest::Client::new();
|
||||
for parent in ["vec", "vec$ml"] {
|
||||
http.post(format!("{url}/v1/namespace/{parent}/create"))
|
||||
.json(&serde_json::json!({"mode": "EXIST_OK"}))
|
||||
.send()
|
||||
.await?
|
||||
.error_for_status()?;
|
||||
}
|
||||
let encoded = format!("vec$ml${name}");
|
||||
http.post(format!("{url}/v1/table/{encoded}/declare"))
|
||||
.json(&serde_json::json!({}))
|
||||
.send()
|
||||
.await?
|
||||
.error_for_status()?;
|
||||
|
||||
let client = NamespaceClient::new(url.to_string());
|
||||
let id = vec!["vec".to_string(), "ml".to_string(), name.to_string()];
|
||||
let description = client.describe_table(&id).await?;
|
||||
|
||||
let mut options = description.storage_options.clone();
|
||||
options.extend(fallback());
|
||||
const DIM: i32 = 16;
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
Field::new("id", DataType::Int64, false),
|
||||
Field::new(
|
||||
"vec",
|
||||
DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), DIM),
|
||||
false,
|
||||
),
|
||||
]));
|
||||
|
||||
for i in 0..batches {
|
||||
let ids: Vec<i64> = (0..rows_each).map(|r| (i * rows_each + r) as i64).collect();
|
||||
let values: Vec<f32> = ids
|
||||
.iter()
|
||||
.flat_map(|id| (0..DIM).map(move |d| (*id as f32) + d as f32))
|
||||
.collect();
|
||||
let vectors = FixedSizeListArray::new(
|
||||
Arc::new(Field::new("item", DataType::Float32, true)),
|
||||
DIM,
|
||||
Arc::new(Float32Array::from(values)),
|
||||
None,
|
||||
);
|
||||
let batch = RecordBatch::try_new(
|
||||
schema.clone(),
|
||||
vec![Arc::new(Int64Array::from(ids)), Arc::new(vectors)],
|
||||
)?;
|
||||
let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
|
||||
let params = WriteParams {
|
||||
mode: if i == 0 {
|
||||
WriteMode::Overwrite
|
||||
} else {
|
||||
WriteMode::Append
|
||||
},
|
||||
store_params: Some(ObjectStoreParams {
|
||||
storage_options_accessor: Some(std::sync::Arc::new(
|
||||
StorageOptionsAccessor::with_static_options(options.clone()),
|
||||
)),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let dataset = Dataset::write(reader, description.location.as_str(), Some(params)).await?;
|
||||
|
||||
// The index is built after the first batch, so everything appended
|
||||
// afterwards is a row it does not cover.
|
||||
if with_index && i == 0 {
|
||||
use lance::index::vector::VectorIndexParams;
|
||||
use lance::index::DatasetIndexExt;
|
||||
use lance_index::vector::{ivf::IvfBuildParams, pq::PQBuildParams};
|
||||
use lance_index::IndexType;
|
||||
|
||||
let mut dataset = dataset;
|
||||
let params = VectorIndexParams::with_ivf_pq_params(
|
||||
lance_linalg::distance::MetricType::L2,
|
||||
IvfBuildParams::new(1),
|
||||
PQBuildParams::new(4, 8),
|
||||
);
|
||||
dataset
|
||||
.create_index(&["vec"], IndexType::Vector, None, ¶ms, true)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
Ok(encoded)
|
||||
}
|
||||
|
||||
/// A table with more fragments than the policy allows is proposed, and running
|
||||
/// the proposal leaves it with fewer than it started with.
|
||||
#[tokio::test]
|
||||
async fn compacts_a_fragmented_table() {
|
||||
let _gateway = GATEWAY.lock().await;
|
||||
let Some(url) = namespace_url() else {
|
||||
eprintln!("WEED_LANCE_NAMESPACE is unset, skipping");
|
||||
return;
|
||||
};
|
||||
let mut fallback = weed_lance_worker::dataset::FallbackOptions::new();
|
||||
fallback.insert("aws_access_key_id".to_string(), "any".to_string());
|
||||
fallback.insert("aws_secret_access_key".to_string(), "any".to_string());
|
||||
// Seeded here rather than by a script, so the test is repeatable: a previous
|
||||
// run compacts the table it depended on.
|
||||
let encoded = seed_fragmented_table(&url, "compactme", 12)
|
||||
.await
|
||||
.expect("seed a fragmented table");
|
||||
let handler = CompactHandler::new(url).with_fallback(fallback);
|
||||
let recorder = Recorder::default();
|
||||
|
||||
let request = RunDetectionRequest {
|
||||
request_id: "detect-1".to_string(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
worker_config_values: int_config("min_fragments", 4),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.detect(&request, &recorder)
|
||||
.await
|
||||
.expect("detection failed");
|
||||
|
||||
let proposals = recorder.proposals.lock().unwrap().clone();
|
||||
assert!(
|
||||
!proposals.is_empty(),
|
||||
"expected a proposal for the fragmented table"
|
||||
);
|
||||
let proposal = proposals
|
||||
.iter()
|
||||
.find(|p| p.summary.contains(encoded.as_str()))
|
||||
.expect("no proposal for the seeded table");
|
||||
|
||||
// Detection opened the dataset to decide, so it reports what it saw. This is
|
||||
// the only description of a Lance table anything outside the format can give.
|
||||
let observations = recorder.observations.lock().unwrap().clone();
|
||||
let observed = observations
|
||||
.iter()
|
||||
.find(|o| o.object_id.last().map(String::as_str) == Some("compactme"))
|
||||
.expect("detection reported no observation for the seeded table");
|
||||
assert_eq!(observed.format, "LANCE");
|
||||
for attribute in ["fragments", "rows", "versions", "schema"] {
|
||||
assert!(
|
||||
observed.attributes.contains_key(attribute),
|
||||
"observation is missing {attribute}: {:?}",
|
||||
observed.attributes.keys().collect::<Vec<_>>()
|
||||
);
|
||||
}
|
||||
|
||||
let execute = ExecuteJobRequest {
|
||||
request_id: "execute-1".to_string(),
|
||||
job: Some(JobSpec {
|
||||
job_id: "job-1".to_string(),
|
||||
job_type: JOB_TYPE.to_string(),
|
||||
parameters: proposal.parameters.clone(),
|
||||
..Default::default()
|
||||
}),
|
||||
worker_config_values: int_config("target_rows_per_fragment", 1_048_576),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.execute(&execute, &recorder)
|
||||
.await
|
||||
.expect("execution failed");
|
||||
|
||||
let completed = recorder.completed.lock().unwrap().clone();
|
||||
let result = completed.first().expect("no completion reported");
|
||||
assert!(
|
||||
result.success,
|
||||
"compaction reported failure: {}",
|
||||
result.error_message
|
||||
);
|
||||
let summary = result
|
||||
.result
|
||||
.as_ref()
|
||||
.map(|r| r.summary.clone())
|
||||
.unwrap_or_default();
|
||||
assert!(
|
||||
summary.contains("became"),
|
||||
"completion carried no fragment counts: {summary}"
|
||||
);
|
||||
eprintln!("compaction result: {summary}");
|
||||
}
|
||||
|
||||
fn fallback() -> weed_lance_worker::dataset::FallbackOptions {
|
||||
let mut options = weed_lance_worker::dataset::FallbackOptions::new();
|
||||
options.insert("aws_access_key_id".to_string(), "any".to_string());
|
||||
options.insert("aws_secret_access_key".to_string(), "any".to_string());
|
||||
options
|
||||
}
|
||||
|
||||
/// A table with more versions than the floor is proposed, and running the job
|
||||
/// reports what it removed. The compaction test above leaves one behind.
|
||||
#[tokio::test]
|
||||
async fn cleans_up_old_versions() {
|
||||
let _gateway = GATEWAY.lock().await;
|
||||
let Some(url) = namespace_url() else {
|
||||
eprintln!("WEED_LANCE_NAMESPACE is unset, skipping");
|
||||
return;
|
||||
};
|
||||
let encoded = seed_table(&url, "cleanme", 6, 4, false)
|
||||
.await
|
||||
.expect("seed a table with versions to clean");
|
||||
|
||||
let handler = CleanupVersionsHandler::new(url).with_fallback(fallback());
|
||||
let recorder = Recorder::default();
|
||||
|
||||
let request = RunDetectionRequest {
|
||||
request_id: "detect-cleanup".to_string(),
|
||||
job_type: "lance_cleanup_versions".to_string(),
|
||||
worker_config_values: int_config("min_versions_to_keep", 2),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.detect(&request, &recorder)
|
||||
.await
|
||||
.expect("detection failed");
|
||||
let proposals = recorder.proposals.lock().unwrap().clone();
|
||||
let proposal = proposals
|
||||
.iter()
|
||||
.find(|p| p.summary.contains(encoded.as_str()))
|
||||
.cloned()
|
||||
.expect("no cleanup proposal for the seeded table");
|
||||
|
||||
// Retain nothing, so every version outside the current one is fair game and
|
||||
// the job has something to report rather than a no-op.
|
||||
let execute = ExecuteJobRequest {
|
||||
request_id: "execute-cleanup".to_string(),
|
||||
job: Some(JobSpec {
|
||||
job_id: "job-cleanup".to_string(),
|
||||
job_type: "lance_cleanup_versions".to_string(),
|
||||
parameters: proposal.parameters.clone(),
|
||||
..Default::default()
|
||||
}),
|
||||
worker_config_values: int_config("retain_hours", 0),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.execute(&execute, &recorder)
|
||||
.await
|
||||
.expect("cleanup failed");
|
||||
|
||||
let completed = recorder.completed.lock().unwrap().clone();
|
||||
let result = completed.last().expect("no completion reported");
|
||||
assert!(
|
||||
result.success,
|
||||
"cleanup reported failure: {}",
|
||||
result.error_message
|
||||
);
|
||||
eprintln!(
|
||||
"cleanup result: {}",
|
||||
result.result.as_ref().unwrap().summary
|
||||
);
|
||||
}
|
||||
|
||||
/// A table with no indices has nothing to optimize, so detection proposes
|
||||
/// nothing rather than queueing work that would do nothing.
|
||||
#[tokio::test]
|
||||
async fn skips_tables_without_indices() {
|
||||
let _gateway = GATEWAY.lock().await;
|
||||
let Some(url) = namespace_url() else {
|
||||
eprintln!("WEED_LANCE_NAMESPACE is unset, skipping");
|
||||
return;
|
||||
};
|
||||
let encoded = seed_table(&url, "noindex", 2, 8, false)
|
||||
.await
|
||||
.expect("seed a table without an index");
|
||||
|
||||
let handler = OptimizeIndicesHandler::new(url).with_fallback(fallback());
|
||||
let recorder = Recorder::default();
|
||||
|
||||
let request = RunDetectionRequest {
|
||||
request_id: "detect-indices".to_string(),
|
||||
job_type: "lance_optimize_indices".to_string(),
|
||||
worker_config_values: int_config("max_unindexed_rows", 1),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.detect(&request, &recorder)
|
||||
.await
|
||||
.expect("detection failed");
|
||||
// Judge this table only: the catalog holds every other test's tables too,
|
||||
// and an indexed one with uncovered rows is supposed to be proposed.
|
||||
assert!(
|
||||
!recorder
|
||||
.proposals
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.any(|p| p.summary.contains(encoded.as_str())),
|
||||
"a table with no indices must not be proposed for reindexing"
|
||||
);
|
||||
}
|
||||
|
||||
/// The job with no Iceberg equivalent: rows appended after an index was built
|
||||
/// are invisible to a search of it until this runs. Needs a table with an index
|
||||
/// and rows outside it, which `indexed.py` in the scratchpad seeds.
|
||||
#[tokio::test]
|
||||
async fn reindexes_rows_an_index_does_not_cover() {
|
||||
let _gateway = GATEWAY.lock().await;
|
||||
let Some(url) = namespace_url() else {
|
||||
eprintln!("WEED_LANCE_NAMESPACE is unset, skipping");
|
||||
return;
|
||||
};
|
||||
let encoded = seed_table(&url, "reindexme", 2, 512, true)
|
||||
.await
|
||||
.expect("seed an indexed table with uncovered rows");
|
||||
let handler = OptimizeIndicesHandler::new(url).with_fallback(fallback());
|
||||
let recorder = Recorder::default();
|
||||
|
||||
let request = RunDetectionRequest {
|
||||
request_id: "detect-reindex".to_string(),
|
||||
job_type: "lance_optimize_indices".to_string(),
|
||||
worker_config_values: int_config("max_unindexed_rows", 100),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.detect(&request, &recorder)
|
||||
.await
|
||||
.expect("detection failed");
|
||||
let proposals = recorder.proposals.lock().unwrap().clone();
|
||||
let proposal = proposals
|
||||
.iter()
|
||||
.find(|p| p.summary.contains(encoded.as_str()))
|
||||
.cloned()
|
||||
.expect("the seeded indexed table was not proposed");
|
||||
|
||||
let execute = ExecuteJobRequest {
|
||||
request_id: "execute-reindex".to_string(),
|
||||
job: Some(JobSpec {
|
||||
job_id: "job-reindex".to_string(),
|
||||
job_type: "lance_optimize_indices".to_string(),
|
||||
parameters: proposal.parameters.clone(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
handler
|
||||
.execute(&execute, &recorder)
|
||||
.await
|
||||
.expect("reindex failed");
|
||||
|
||||
let completed = recorder.completed.lock().unwrap().clone();
|
||||
let result = completed.last().expect("no completion reported");
|
||||
assert!(
|
||||
result.success,
|
||||
"reindex reported failure: {}",
|
||||
result.error_message
|
||||
);
|
||||
let output = &result.result.as_ref().unwrap().output_values;
|
||||
let after = match output
|
||||
.get("unindexed_rows_after")
|
||||
.and_then(|v| v.kind.as_ref())
|
||||
{
|
||||
Some(Kind::Int64Value(value)) => *value,
|
||||
other => panic!("no unindexed_rows_after in {other:?}"),
|
||||
};
|
||||
assert_eq!(
|
||||
after, 0,
|
||||
"rows are still outside the index after optimizing"
|
||||
);
|
||||
eprintln!(
|
||||
"reindex result: {}",
|
||||
result.result.as_ref().unwrap().summary
|
||||
);
|
||||
}
|
||||
|
||||
/// The UI's whole reason for asking a worker: admin cannot read a Lance table,
|
||||
/// so the rows have to come back already rendered.
|
||||
#[tokio::test]
|
||||
async fn previews_rows_of_a_table() {
|
||||
let _gateway = GATEWAY.lock().await;
|
||||
let Some(url) = namespace_url() else {
|
||||
eprintln!("set WEED_LANCE_NAMESPACE to run this test");
|
||||
return;
|
||||
};
|
||||
seed_table(&url, "previewme", 2, 3, false)
|
||||
.await
|
||||
.expect("seed a table to preview");
|
||||
|
||||
let provider = LancePreview::new(url, fallback());
|
||||
let id = vec!["vec".to_string(), "ml".to_string(), "previewme".to_string()];
|
||||
let preview = provider.preview(&id, 4).await.expect("preview the table");
|
||||
|
||||
assert_eq!(preview.columns, vec!["id".to_string(), "vec".to_string()]);
|
||||
assert_eq!(preview.total_rows, 6, "total is the table, not the sample");
|
||||
assert_eq!(preview.rows.len(), 4, "row_limit bounds the sample");
|
||||
assert!(
|
||||
preview.rows[0][1].starts_with('['),
|
||||
"a vector column should render as a list, got {:?}",
|
||||
preview.rows[0][1]
|
||||
);
|
||||
}
|
||||
|
||||
/// The claim that removed managed versioning is not "one writer wins the
|
||||
/// conditional PUT" - that is only the mechanism. It is that concurrent writers
|
||||
/// lose nothing: the loser sees the conflict, rebases, and commits again. Eight
|
||||
/// writers appending at once must leave all eight batches in the table.
|
||||
#[tokio::test]
|
||||
async fn concurrent_writers_keep_every_commit() {
|
||||
let _gateway = GATEWAY.lock().await;
|
||||
let Some(url) = namespace_url() else {
|
||||
eprintln!("set WEED_LANCE_NAMESPACE to run this test");
|
||||
return;
|
||||
};
|
||||
const WRITERS: i64 = 8;
|
||||
const ROWS_EACH: i64 = 4;
|
||||
|
||||
seed_table(&url, "racers", 1, ROWS_EACH as usize, false)
|
||||
.await
|
||||
.expect("seed the table the writers will append to");
|
||||
|
||||
let client = NamespaceClient::new(url.clone());
|
||||
let id = vec!["vec".to_string(), "ml".to_string(), "racers".to_string()];
|
||||
let description = client.describe_table(&id).await.expect("describe");
|
||||
let mut options = description.storage_options.clone();
|
||||
options.extend(fallback());
|
||||
|
||||
let writes = (0..WRITERS).map(|writer| {
|
||||
let location = description.location.clone();
|
||||
let options = options.clone();
|
||||
tokio::spawn(
|
||||
async move { append_rows(&location, &options, writer * 1000, ROWS_EACH).await },
|
||||
)
|
||||
});
|
||||
for (writer, handle) in writes.enumerate() {
|
||||
handle
|
||||
.await
|
||||
.expect("writer panicked")
|
||||
.unwrap_or_else(|err| panic!("writer {writer} failed to commit: {err:#}"));
|
||||
}
|
||||
|
||||
let table = weed_lance_worker::dataset::open(&client, &id, &fallback())
|
||||
.await
|
||||
.expect("reopen the table");
|
||||
let rows = table.dataset.count_rows(None).await.expect("count rows");
|
||||
let expected = (ROWS_EACH + WRITERS * ROWS_EACH) as usize;
|
||||
assert_eq!(
|
||||
rows, expected,
|
||||
"concurrent commits lost data: {rows} rows, want {expected}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Appends one batch to an existing dataset, the way an independent writer would.
|
||||
async fn append_rows(
|
||||
location: &str,
|
||||
options: &HashMap<String, String>,
|
||||
first_id: i64,
|
||||
rows: i64,
|
||||
) -> Result<()> {
|
||||
use arrow_array::{
|
||||
FixedSizeListArray, Float32Array, Int64Array, RecordBatch, RecordBatchIterator,
|
||||
};
|
||||
use arrow_schema::{DataType, Field, Schema};
|
||||
use lance::dataset::{Dataset, WriteMode, WriteParams};
|
||||
use lance::io::{ObjectStoreParams, StorageOptionsAccessor};
|
||||
use std::sync::Arc;
|
||||
|
||||
const DIM: i32 = 16;
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
Field::new("id", DataType::Int64, false),
|
||||
Field::new(
|
||||
"vec",
|
||||
DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), DIM),
|
||||
false,
|
||||
),
|
||||
]));
|
||||
let ids: Vec<i64> = (0..rows).map(|r| first_id + r).collect();
|
||||
let values: Vec<f32> = ids
|
||||
.iter()
|
||||
.flat_map(|id| (0..DIM).map(move |d| (*id as f32) + d as f32))
|
||||
.collect();
|
||||
let vectors = FixedSizeListArray::new(
|
||||
Arc::new(Field::new("item", DataType::Float32, true)),
|
||||
DIM,
|
||||
Arc::new(Float32Array::from(values)),
|
||||
None,
|
||||
);
|
||||
let batch = RecordBatch::try_new(
|
||||
schema.clone(),
|
||||
vec![Arc::new(Int64Array::from(ids)), Arc::new(vectors)],
|
||||
)?;
|
||||
let params = WriteParams {
|
||||
mode: WriteMode::Append,
|
||||
store_params: Some(ObjectStoreParams {
|
||||
storage_options_accessor: Some(Arc::new(StorageOptionsAccessor::with_static_options(
|
||||
options.clone(),
|
||||
))),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
Dataset::write(
|
||||
RecordBatchIterator::new(vec![Ok(batch)], schema.clone()),
|
||||
location,
|
||||
Some(params),
|
||||
)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -310,6 +310,7 @@ func (c *distributedLockCluster) startS3(index int) error {
|
||||
"-port=" + strconv.Itoa(c.s3Ports[index]),
|
||||
"-port.grpc=" + strconv.Itoa(c.s3GrpcPorts[index]),
|
||||
"-port.iceberg=0",
|
||||
"-port.lance=0",
|
||||
"-filer=" + strings.Join(filers, ","),
|
||||
"-config=" + c.s3Config,
|
||||
"-iam.readOnly=false",
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
# Lance client container for Lance Namespace REST compatibility testing.
|
||||
# The point of this image is to exercise the namespace with the real client
|
||||
# rather than hand-built HTTP: every serious bug in this surface so far looked
|
||||
# fine to a request we wrote ourselves.
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN pip install --no-cache-dir lance-namespace pylance pyarrow
|
||||
|
||||
COPY test_lance_namespace.py /app/
|
||||
|
||||
CMD ["python3", "/app/test_lance_namespace.py", "--help"]
|
||||
@@ -60,7 +60,12 @@ func newOAuthTestEnv(t *testing.T) *oauthTestEnv {
|
||||
}
|
||||
|
||||
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
|
||||
if info, err := os.Stat(weedBinary); err != nil || info.IsDir() {
|
||||
if info, err := os.Stat(weedBinary); err == nil && !info.IsDir() {
|
||||
// Say which binary and how old it is. `make test` rebuilds first, but
|
||||
// `go test` on its own happily runs a weeks-old binary and reports a
|
||||
// pass for code that is not being exercised.
|
||||
t.Logf("using %s, built %s", weedBinary, info.ModTime().Format(time.RFC3339))
|
||||
} else {
|
||||
weedBinary = "weed"
|
||||
if _, err := exec.LookPath(weedBinary); err != nil {
|
||||
t.Skip("weed binary not found, skipping integration test")
|
||||
|
||||
@@ -62,6 +62,7 @@ type TestEnvironment struct {
|
||||
s3Port int
|
||||
s3GrpcPort int
|
||||
icebergPort int
|
||||
lancePort int
|
||||
masterPort int
|
||||
masterGrpcPort int
|
||||
filerPort int
|
||||
@@ -92,7 +93,12 @@ func newTestEnvironmentForMain() (*TestEnvironment, error) {
|
||||
|
||||
// Check for weed binary
|
||||
weedBinary := filepath.Join(seaweedDir, "weed", "weed")
|
||||
if _, err := os.Stat(weedBinary); os.IsNotExist(err) {
|
||||
if info, statErr := os.Stat(weedBinary); statErr == nil {
|
||||
// Name the binary and its age. `make test` rebuilds first, but a plain
|
||||
// `go test` will happily drive a weeks-old binary and report a pass for
|
||||
// code it never ran.
|
||||
fmt.Fprintf(os.Stderr, "using %s, built %s\n", weedBinary, info.ModTime().Format(time.RFC3339))
|
||||
} else if os.IsNotExist(statErr) {
|
||||
weedBinary = "weed"
|
||||
if _, err := exec.LookPath(weedBinary); err != nil {
|
||||
return nil, fmt.Errorf("weed binary not found")
|
||||
@@ -105,9 +111,9 @@ func newTestEnvironmentForMain() (*TestEnvironment, error) {
|
||||
return nil, fmt.Errorf("create temp dir: %w", err)
|
||||
}
|
||||
|
||||
// Allocate 9 unique ports atomically: s3, iceberg, s3Grpc, master, masterGrpc,
|
||||
// filer, filerGrpc, volume, volumeGrpc
|
||||
ports, err := testutil.AllocatePorts(9)
|
||||
// Allocate 10 unique ports atomically: s3, iceberg, s3Grpc, master, masterGrpc,
|
||||
// filer, filerGrpc, volume, volumeGrpc, lance
|
||||
ports, err := testutil.AllocatePorts(10)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("allocate ports: %w", err)
|
||||
}
|
||||
@@ -125,6 +131,7 @@ func newTestEnvironmentForMain() (*TestEnvironment, error) {
|
||||
filerGrpcPort: ports[6],
|
||||
volumePort: ports[7],
|
||||
volumeGrpcPort: ports[8],
|
||||
lancePort: ports[9],
|
||||
dockerAvailable: testutil.HasDocker(),
|
||||
}, nil
|
||||
}
|
||||
@@ -155,6 +162,7 @@ func (env *TestEnvironment) startSeaweedFSForMain() error {
|
||||
"-s3.port", fmt.Sprintf("%d", env.s3Port),
|
||||
"-s3.port.grpc", fmt.Sprintf("%d", env.s3GrpcPort),
|
||||
"-s3.port.iceberg", fmt.Sprintf("%d", env.icebergPort),
|
||||
"-s3.port.lance", fmt.Sprintf("%d", env.lancePort),
|
||||
"-ip.bind", "0.0.0.0",
|
||||
"-dir", env.dataDir,
|
||||
)
|
||||
@@ -211,6 +219,11 @@ func (env *TestEnvironment) IcebergURL() string {
|
||||
return fmt.Sprintf("http://127.0.0.1:%d", env.icebergPort)
|
||||
}
|
||||
|
||||
// LanceURL returns the Lance Namespace server URL
|
||||
func (env *TestEnvironment) LanceURL() string {
|
||||
return fmt.Sprintf("http://127.0.0.1:%d", env.lancePort)
|
||||
}
|
||||
|
||||
// TestIcebergConfig tests the /v1/config endpoint
|
||||
func TestIcebergConfig(t *testing.T) {
|
||||
if testing.Short() {
|
||||
@@ -252,7 +265,7 @@ func TestIcebergNamespaces(t *testing.T) {
|
||||
|
||||
// Create the default table bucket first via S3
|
||||
bucketName := "warehouse-ns-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
// Test GET /v1/namespaces (should return empty list initially)
|
||||
resp, err := http.Get(env.IcebergURL() + icebergPath(bucketName, "/v1/namespaces"))
|
||||
@@ -275,7 +288,7 @@ func TestStageCreateAndFinalizeFlow(t *testing.T) {
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-stage-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
namespace := "stage_ns_" + randomSuffix()
|
||||
tableName := "orders"
|
||||
@@ -369,7 +382,7 @@ func TestCommitMissingTableWithoutAssertCreate(t *testing.T) {
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-missing-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
namespace := "stage_missing_assert_ns_" + randomSuffix()
|
||||
tableName := "missing_table"
|
||||
@@ -458,11 +471,16 @@ func icebergPath(prefix, path string) string {
|
||||
// The request is AWS V4 signed for SERVICE=s3tables so the S3 Tables
|
||||
// route matcher accepts it; signing with regular SERVICE=s3 would let
|
||||
// the request fall through to the S3 CreateBucket handler.
|
||||
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
|
||||
// createTableBucket makes a table bucket of the given format. An empty format
|
||||
// leaves it to the server, which means ICEBERG.
|
||||
func createTableBucket(t *testing.T, env *TestEnvironment, bucketName, format string) {
|
||||
t.Helper()
|
||||
|
||||
endpoint := fmt.Sprintf("http://localhost:%d/buckets", env.s3Port)
|
||||
reqBody := fmt.Sprintf(`{"name":"%s"}`, bucketName)
|
||||
if format != "" {
|
||||
reqBody = fmt.Sprintf(`{"name":"%s","format":"%s"}`, bucketName, format)
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPut, endpoint, strings.NewReader(reqBody))
|
||||
if err != nil {
|
||||
|
||||
@@ -52,7 +52,7 @@ func TestIssue9103_ConfigDoesNotVendWarehousePrefix(t *testing.T) {
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-9103cfg-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
warehouse := fmt.Sprintf("s3://%s/", bucketName)
|
||||
u := fmt.Sprintf("%s/v1/config?warehouse=%s", env.IcebergURL(), url.QueryEscape(warehouse))
|
||||
@@ -88,7 +88,7 @@ func TestIssue9103_BareNamespacesListMissesNamespaceInAttachedBucket(t *testing.
|
||||
|
||||
env := sharedEnv
|
||||
bucketName := "warehouse-9103ns-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
namespace := "ovirt"
|
||||
status, _, err := doIcebergJSONRequest(env, http.MethodPost,
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestLanceNamespaceRealClient drives the namespace with the Lance client rather
|
||||
// than hand-built HTTP. The Go tests beside this one cover the catalog surface;
|
||||
// what this adds is that the location and storage_options the namespace vends
|
||||
// are actually enough to write and read a dataset, which needs the S3 layout
|
||||
// guard, the endpoint and the credentials all to be right at once.
|
||||
//
|
||||
// To run manually:
|
||||
//
|
||||
// cd test/s3tables/catalog
|
||||
// docker build -t lance-namespace-test -f Dockerfile.lance .
|
||||
func TestLanceNamespaceRealClient(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
|
||||
env := sharedEnv
|
||||
if !env.dockerAvailable {
|
||||
t.Skip("Docker not available, skipping Lance client integration test")
|
||||
}
|
||||
|
||||
bucketName := "lance-client-test-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName, s3tables.FormatLance)
|
||||
|
||||
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
|
||||
|
||||
buildCmd := exec.Command("docker", "build", "-t", "lance-namespace-test", "-f", "Dockerfile.lance", ".")
|
||||
buildCmd.Dir = testDir
|
||||
if out, err := buildCmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
|
||||
}
|
||||
|
||||
namespaceURL := fmt.Sprintf("http://host.docker.internal:%d", env.lancePort)
|
||||
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
|
||||
|
||||
cmd := exec.Command("docker", "run", "--rm",
|
||||
"--add-host", "host.docker.internal:host-gateway",
|
||||
"-v", fmt.Sprintf("%s:/app:ro", testDir),
|
||||
"lance-namespace-test",
|
||||
"python3", "/app/test_lance_namespace.py",
|
||||
"--namespace-url", namespaceURL,
|
||||
"--s3-endpoint", s3Endpoint,
|
||||
"--bucket", bucketName,
|
||||
)
|
||||
cmd.Dir = testDir
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
t.Logf("Running Lance client test against %s", namespaceURL)
|
||||
if err := cmd.Run(); err != nil {
|
||||
t.Errorf("Lance client test failed: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,378 @@
|
||||
// Integration tests for the Lance Namespace REST server, driven against a live
|
||||
// gateway rather than an in-memory filer. The bugs this surface has produced -
|
||||
// a deregister that deleted the dataset, an S3 door that refused every Lance
|
||||
// file - all looked fine against a fake.
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// lanceCall posts to the Lance namespace and returns the status and body.
|
||||
func lanceCall(t *testing.T, env *TestEnvironment, method, path, body string) (int, []byte) {
|
||||
t.Helper()
|
||||
req, err := http.NewRequest(method, env.LanceURL()+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("build request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", method, path, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
payload, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, payload
|
||||
}
|
||||
|
||||
func lanceMust(t *testing.T, env *TestEnvironment, method, path, body string, want int) []byte {
|
||||
t.Helper()
|
||||
status, payload := lanceCall(t, env, method, path, body)
|
||||
if status != want {
|
||||
t.Fatalf("%s %s = %d (%s), want %d", method, path, status, payload, want)
|
||||
}
|
||||
return payload
|
||||
}
|
||||
|
||||
// filerEntryExists reports whether a path exists on storage, which is how these
|
||||
// tests tell "the catalog forgot the table" from "the data is gone".
|
||||
func filerEntryExists(t *testing.T, env *TestEnvironment, path string) bool {
|
||||
t.Helper()
|
||||
url := fmt.Sprintf("http://127.0.0.1:%d%s", env.filerPort, path)
|
||||
resp, err := http.Get(url)
|
||||
if err != nil {
|
||||
t.Fatalf("filer GET %s: %v", path, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode == http.StatusOK
|
||||
}
|
||||
|
||||
// lanceTestBucket makes a uniquely named table bucket of the given format. Most
|
||||
// tests want LANCE; the one that checks what the Lance surface hides needs an
|
||||
// Iceberg bucket, because a declared bucket holds one format only.
|
||||
func lanceTestBucket(t *testing.T, env *TestEnvironment, prefix, format string) string {
|
||||
t.Helper()
|
||||
bucket := prefix + "-" + randomSuffix()
|
||||
createTableBucket(t, env, bucket, format)
|
||||
return bucket
|
||||
}
|
||||
|
||||
func TestLanceNamespaceLifecycle(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-ns", s3tables.FormatLance)
|
||||
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/create", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$sales/exists", `{}`, http.StatusOK)
|
||||
|
||||
// The root lists table buckets, which is the first namespace level here.
|
||||
var roots struct {
|
||||
Namespaces []string `json:"namespaces"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK), &roots); err != nil {
|
||||
t.Fatalf("decode root listing: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, name := range roots.Namespaces {
|
||||
if name == bucket {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("root listing %v does not include %s", roots.Namespaces, bucket)
|
||||
}
|
||||
|
||||
var children struct {
|
||||
Namespaces []string `json:"namespaces"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"/list", "", http.StatusOK), &children); err != nil {
|
||||
t.Fatalf("decode namespace listing: %v", err)
|
||||
}
|
||||
if len(children.Namespaces) != 1 || children.Namespaces[0] != "sales" {
|
||||
t.Fatalf("namespace listing = %v, want [sales]", children.Namespaces)
|
||||
}
|
||||
|
||||
// A namespace is never created as a side effect of naming one inside a
|
||||
// bucket that does not exist.
|
||||
status, _ := lanceCall(t, env, http.MethodPost, "/v1/namespace/nosuchbucket$ns/create", `{}`)
|
||||
if status != http.StatusNotFound {
|
||||
t.Fatalf("create under a missing bucket = %d, want 404", status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLanceTableLifecyclePreservesData(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-tbl", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
|
||||
table := "/v1/table/" + bucket + "$ml$vectors"
|
||||
var declared struct {
|
||||
Location string `json:"location"`
|
||||
StorageOptions map[string]string `json:"storage_options"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil {
|
||||
t.Fatalf("decode declare: %v", err)
|
||||
}
|
||||
want := fmt.Sprintf("s3://%s/ml/vectors", bucket)
|
||||
if declared.Location != want {
|
||||
t.Fatalf("declared location = %q, want %q", declared.Location, want)
|
||||
}
|
||||
|
||||
datasetPath := fmt.Sprintf("/buckets/%s/ml/vectors/", bucket)
|
||||
if !filerEntryExists(t, env, datasetPath) {
|
||||
t.Fatal("declare did not create the dataset directory")
|
||||
}
|
||||
|
||||
var listed struct {
|
||||
Tables []string `json:"tables"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$ml/table/list", "", http.StatusOK), &listed); err != nil {
|
||||
t.Fatalf("decode listing: %v", err)
|
||||
}
|
||||
if len(listed.Tables) != 1 || listed.Tables[0] != bucket+"$ml$vectors" {
|
||||
t.Fatalf("table listing = %v, want the full identifier", listed.Tables)
|
||||
}
|
||||
|
||||
// Deregistering forgets the table but keeps every byte. The catalog entry is
|
||||
// the dataset directory, so a drop of the entry would take the data too.
|
||||
lanceMust(t, env, http.MethodPost, table+"/deregister", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusNotFound)
|
||||
if !filerEntryExists(t, env, datasetPath) {
|
||||
t.Fatal("deregister deleted the dataset")
|
||||
}
|
||||
|
||||
registerBody := fmt.Sprintf(`{"location":%q}`, want)
|
||||
lanceMust(t, env, http.MethodPost, table+"/register", registerBody, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, table+"/exists", `{}`, http.StatusOK)
|
||||
|
||||
// Dropping is the operation that does remove the data.
|
||||
lanceMust(t, env, http.MethodPost, table+"/drop", `{}`, http.StatusOK)
|
||||
if filerEntryExists(t, env, datasetPath) {
|
||||
t.Fatal("drop left the dataset behind")
|
||||
}
|
||||
}
|
||||
|
||||
// A Lance client must never resolve an Iceberg table's location, or it writes a
|
||||
// dataset over a table another engine owns.
|
||||
func TestLanceHidesIcebergTables(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
// An Iceberg bucket, because a bucket that declares LANCE cannot hold an
|
||||
// Iceberg table at all now. The invariant still matters from this side: the
|
||||
// Lance surface can be pointed at any bucket, and must not describe or list
|
||||
// a table whose format it does not serve.
|
||||
bucket := lanceTestBucket(t, env, "lance-mixed", s3tables.FormatIceberg)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$mixed/create", `{}`, http.StatusOK)
|
||||
|
||||
createIcebergTable(t, env, bucket, "mixed", "ledger")
|
||||
|
||||
status, _ := lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/describe", `{}`)
|
||||
if status != http.StatusNotFound {
|
||||
t.Fatalf("describing an iceberg table through lance = %d, want 404", status)
|
||||
}
|
||||
status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$ledger/declare", `{}`)
|
||||
if status != http.StatusConflict {
|
||||
t.Fatalf("declaring over an iceberg table = %d, want 409", status)
|
||||
}
|
||||
// And a new Lance table cannot be smuggled in beside it either.
|
||||
status, _ = lanceCall(t, env, http.MethodPost, "/v1/table/"+bucket+"$mixed$vectors/declare", `{}`)
|
||||
if status != http.StatusConflict {
|
||||
t.Fatalf("declaring a lance table in an iceberg bucket = %d, want 409", status)
|
||||
}
|
||||
|
||||
var listed struct {
|
||||
Tables []string `json:"tables"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodGet, "/v1/namespace/"+bucket+"$mixed/table/list", "", http.StatusOK), &listed); err != nil {
|
||||
t.Fatalf("decode listing: %v", err)
|
||||
}
|
||||
if len(listed.Tables) != 0 {
|
||||
t.Fatalf("lance listing shows iceberg tables: %v", listed.Tables)
|
||||
}
|
||||
}
|
||||
|
||||
// The S3 door validates every object written into a table bucket. A Lance
|
||||
// dataset's files have to get through it, or the catalog is decorative.
|
||||
func TestLanceFilesAreAcceptedByTheS3Door(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-layout", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK)
|
||||
|
||||
accepted := []string{
|
||||
"ml/vectors/data/0111111010111000110101116.lance",
|
||||
"ml/vectors/_versions/18446744073709551614.manifest",
|
||||
"ml/vectors/_versions/18446744073709551614.manifest-a3a292ad",
|
||||
"ml/vectors/_transactions/0-ddb27ab7.txn",
|
||||
"ml/vectors/_indices/85814508-fts/index.idx",
|
||||
"ml/vectors/.lance-reserved",
|
||||
}
|
||||
for _, object := range accepted {
|
||||
if status := putS3Object(t, env, bucket, object); status != http.StatusOK {
|
||||
t.Errorf("PUT %s = %d, want 200", object, status)
|
||||
}
|
||||
}
|
||||
|
||||
// The guard still rejects files that belong to no table layout.
|
||||
for _, object := range []string{"ml/vectors/random.txt", "ml/vectors/notadir/x.lance"} {
|
||||
if status := putS3Object(t, env, bucket, object); status == http.StatusOK {
|
||||
t.Errorf("PUT %s = 200, want a rejection", object)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A Lance commit is a conditional PUT of the next manifest: object_store turns
|
||||
// PutMode::Create into If-None-Match: *, and lance treats the refusal as a
|
||||
// commit conflict and rebases. So the whole safety of concurrent writers rests
|
||||
// on this store answering that precondition atomically. It does, because the
|
||||
// gateway reduces the header to a filer WriteCondition evaluated at the
|
||||
// object's owner under a per-path lock.
|
||||
//
|
||||
// This is the reason the namespace does not manage versions itself.
|
||||
func TestLanceCommitPreconditionAdmitsOneWriter(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-commit", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/table/"+bucket+"$ml$vectors/declare", `{}`, http.StatusOK)
|
||||
|
||||
const writers = 8
|
||||
var wg sync.WaitGroup
|
||||
statuses := make([]int, writers)
|
||||
failures := make([]error, writers)
|
||||
for i := 0; i < writers; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
// Not t.Fatalf: from a goroutine it ends only that goroutine, and
|
||||
// the status stays zero, which reads as a bogus response later.
|
||||
statuses[i], failures[i] = putIfAbsent(env, bucket, "ml/vectors/_versions/1.manifest")
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
for i, err := range failures {
|
||||
if err != nil {
|
||||
t.Fatalf("writer %d could not reach the gateway: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
won, refused := 0, 0
|
||||
for _, status := range statuses {
|
||||
switch status {
|
||||
case http.StatusOK:
|
||||
won++
|
||||
case http.StatusPreconditionFailed:
|
||||
refused++
|
||||
default:
|
||||
t.Fatalf("unexpected status %d committing a manifest", status)
|
||||
}
|
||||
}
|
||||
if won != 1 {
|
||||
t.Fatalf("%d writers committed version 1; exactly one may win, %d were refused", won, refused)
|
||||
}
|
||||
}
|
||||
|
||||
// The namespace answers managed_versioning=false: the dataset owns its version
|
||||
// history, and a reader that never goes through this catalog still sees all of
|
||||
// it.
|
||||
func TestLanceDoesNotManageVersions(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("Skipping integration test in short mode")
|
||||
}
|
||||
env := sharedEnv
|
||||
bucket := lanceTestBucket(t, env, "lance-ver", s3tables.FormatLance)
|
||||
lanceMust(t, env, http.MethodPost, "/v1/namespace/"+bucket+"$ml/create", `{}`, http.StatusOK)
|
||||
table := "/v1/table/" + bucket + "$ml$vectors"
|
||||
|
||||
var declared struct {
|
||||
ManagedVersioning bool `json:"managed_versioning"`
|
||||
}
|
||||
if err := json.Unmarshal(lanceMust(t, env, http.MethodPost, table+"/declare", `{}`, http.StatusOK), &declared); err != nil {
|
||||
t.Fatalf("decode declare: %v", err)
|
||||
}
|
||||
if declared.ManagedVersioning {
|
||||
t.Fatal("managed_versioning must be false; the dataset owns its versions")
|
||||
}
|
||||
|
||||
// The version ops answer with the spec's Unsupported code rather than a
|
||||
// bare 404, so a client that asks learns why.
|
||||
status, _ := lanceCall(t, env, http.MethodPost, table+"/version/list", `{}`)
|
||||
if status != http.StatusNotImplemented {
|
||||
t.Fatalf("version/list = %d, want 501", status)
|
||||
}
|
||||
}
|
||||
|
||||
// putIfAbsent writes an object only if the key is free, the way a Lance commit
|
||||
// does. It returns an error rather than failing the test, so it is safe to call
|
||||
// from the racing goroutines.
|
||||
func putIfAbsent(env *TestEnvironment, bucket, object string) (int, error) {
|
||||
url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object)
|
||||
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("manifest")))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
req.Header.Set("If-None-Match", "*")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
// putS3Object writes an object through the S3 gateway and returns the status.
|
||||
func putS3Object(t *testing.T, env *TestEnvironment, bucket, object string) int {
|
||||
t.Helper()
|
||||
url := fmt.Sprintf("http://127.0.0.1:%d/%s/%s", env.s3Port, bucket, object)
|
||||
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader([]byte("x")))
|
||||
if err != nil {
|
||||
t.Fatalf("build request: %v", err)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("PUT %s: %v", object, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode
|
||||
}
|
||||
|
||||
// createIcebergTable registers an ordinary Iceberg table so the mixed-catalog
|
||||
// tests have one to be confused by.
|
||||
func createIcebergTable(t *testing.T, env *TestEnvironment, bucket, namespace, name string) {
|
||||
t.Helper()
|
||||
body := fmt.Sprintf(`{"name":%q,"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}}`, name)
|
||||
url := fmt.Sprintf("%s/v1/%s/namespaces/%s/tables", env.IcebergURL(), bucket, namespace)
|
||||
resp, err := http.Post(url, "application/json", strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatalf("create iceberg table: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
payload, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("create iceberg table = %d: %s", resp.StatusCode, payload)
|
||||
}
|
||||
}
|
||||
@@ -34,7 +34,7 @@ func TestPyIcebergRestCatalog(t *testing.T) {
|
||||
|
||||
// Create the test bucket first
|
||||
bucketName := "pyiceberg-compat-test-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
// Build the test working directory path
|
||||
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
|
||||
@@ -93,7 +93,7 @@ func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
|
||||
|
||||
// Create the test bucket first (using unauthenticated request, which works with DefaultAllow)
|
||||
bucketName := "pyiceberg-auth-test-" + randomSuffix()
|
||||
createTableBucket(t, env, bucketName)
|
||||
createTableBucket(t, env, bucketName, "")
|
||||
|
||||
// Build the test working directory path
|
||||
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
|
||||
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Drive the SeaweedFS Lance Namespace with the real Lance client.
|
||||
|
||||
The catalog half can be checked with plain HTTP, and the Go integration tests
|
||||
do. What only a real client proves is that the location and storage_options it
|
||||
hands back are enough to write and read a dataset: the layout guard on the S3
|
||||
door, the endpoint and allow_http options, and the credentials all have to be
|
||||
right at once, and a hand-built request checks none of that.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import warnings
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
import lance
|
||||
import lance_namespace as ln
|
||||
import pyarrow as pa
|
||||
|
||||
|
||||
def sample_table():
|
||||
return pa.table(
|
||||
{
|
||||
"id": pa.array([1, 2, 3, 4]),
|
||||
"vec": pa.array([[1.0, 2.0]] * 4, type=pa.list_(pa.float32(), 2)),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--namespace-url", required=True, help="Lance namespace REST URL")
|
||||
parser.add_argument("--s3-endpoint", required=True, help="S3 endpoint the dataset lives behind")
|
||||
parser.add_argument("--bucket", required=True, help="table bucket to use, already created")
|
||||
parser.add_argument("--access-key", default="any")
|
||||
parser.add_argument("--secret-key", default="any")
|
||||
args = parser.parse_args()
|
||||
|
||||
ns = ln.connect("rest", {"uri": args.namespace_url})
|
||||
ns.create_namespace(ln.CreateNamespaceRequest(id=[args.bucket, "ml"]))
|
||||
table_id = [args.bucket, "ml", "vectors"]
|
||||
|
||||
declared = ns.declare_table(ln.DeclareTableRequest(id=table_id))
|
||||
print(f"declared {table_id} at {declared.location}")
|
||||
if not declared.location:
|
||||
print("FAIL: declare returned no location", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
described = ns.describe_table(ln.DescribeTableRequest(id=table_id))
|
||||
if described.location != declared.location:
|
||||
print(
|
||||
f"FAIL: describe location {described.location} != declare {declared.location}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
options = dict(described.storage_options or {})
|
||||
print(f"storage options from the namespace: {sorted(options)}")
|
||||
# The endpoint is overridden with the container's view of the same gateway:
|
||||
# what the namespace vends is correct for its own host, and a test harness
|
||||
# bound to a wildcard address vends nothing at all. That the namespace vends
|
||||
# a usable endpoint when it can is covered by the unit tests; what matters
|
||||
# here is that everything else it hands back is enough to reach the data.
|
||||
options["aws_endpoint"] = args.s3_endpoint
|
||||
options["allow_http"] = "true"
|
||||
# A deployment without STS still needs credentials to sign with.
|
||||
options.setdefault("aws_access_key_id", args.access_key)
|
||||
options.setdefault("aws_secret_access_key", args.secret_key)
|
||||
|
||||
lance.write_dataset(sample_table(), described.location, storage_options=options,
|
||||
mode="overwrite")
|
||||
dataset = lance.dataset(described.location, storage_options=options)
|
||||
rows = dataset.count_rows()
|
||||
print(f"wrote and read back {rows} rows at version {dataset.version}")
|
||||
if rows != 4:
|
||||
print(f"FAIL: read back {rows} rows, want 4", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# The table is listed, and a client that skips the namespace entirely can
|
||||
# still open the dataset by URI.
|
||||
listed = ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables
|
||||
if f"{args.bucket}$ml$vectors" not in listed:
|
||||
print(f"FAIL: {listed} does not contain the table", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if lance.dataset(declared.location, storage_options=options).count_rows() != 4:
|
||||
print("FAIL: the dataset is not readable straight off its URI", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# Deregistering hides the table and keeps the data, which is the difference
|
||||
# between it and a drop.
|
||||
ns.deregister_table(ln.DeregisterTableRequest(id=table_id))
|
||||
if f"{args.bucket}$ml$vectors" in ns.list_tables(ln.ListTablesRequest(id=[args.bucket, "ml"])).tables:
|
||||
print("FAIL: a deregistered table is still listed", file=sys.stderr)
|
||||
return 1
|
||||
if lance.dataset(declared.location, storage_options=options).count_rows() != 4:
|
||||
print("FAIL: deregister destroyed the dataset", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print("PASS")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -38,6 +38,7 @@ var miniDefaultPorts = []int{
|
||||
9340, // volume.port
|
||||
8333, // s3.port
|
||||
8181, // s3.port.iceberg
|
||||
9101, // s3.port.lance
|
||||
7333, // webdav.port
|
||||
23646, // admin.port
|
||||
}
|
||||
|
||||
@@ -156,11 +156,12 @@ type AdminServer struct {
|
||||
|
||||
s3TablesManager *s3tables.Manager
|
||||
icebergPort int
|
||||
lancePort int
|
||||
}
|
||||
|
||||
// Type definitions moved to types.go
|
||||
|
||||
func NewAdminServer(masters string, filerGroup string, templateFS http.FileSystem, dataDir string, icebergPort int) *AdminServer {
|
||||
func NewAdminServer(masters string, filerGroup string, templateFS http.FileSystem, dataDir string, icebergPort, lancePort int) *AdminServer {
|
||||
grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.admin")
|
||||
|
||||
// Create master client with multiple master support
|
||||
@@ -196,6 +197,7 @@ func NewAdminServer(masters string, filerGroup string, templateFS http.FileSyste
|
||||
collectionStatsCacheThreshold: defaultStatsCacheTimeout,
|
||||
s3TablesManager: newS3TablesManager(),
|
||||
icebergPort: icebergPort,
|
||||
lancePort: lancePort,
|
||||
pluginLock: lockManager,
|
||||
adminPresenceLock: presenceLock,
|
||||
bgCancel: bgCancel,
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"github.com/apache/iceberg-go"
|
||||
"github.com/parquet-go/parquet-go"
|
||||
"github.com/seaweedfs/seaweedfs/weed/admin/plugin"
|
||||
"github.com/seaweedfs/seaweedfs/weed/filer"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
@@ -27,6 +28,10 @@ const (
|
||||
icebergPreviewMaxListed = 500
|
||||
icebergPreviewMaxCellChars = 200
|
||||
icebergPreviewMaxMetaBytes = 64 << 20
|
||||
|
||||
// workerPreviewTimeout bounds one round trip to a worker. A page waiting on
|
||||
// a worker that has gone quiet should say so rather than hang.
|
||||
workerPreviewTimeout = 15 * time.Second
|
||||
)
|
||||
|
||||
type IcebergDataFileInfo struct {
|
||||
@@ -60,6 +65,10 @@ type IcebergDataPreviewData struct {
|
||||
PreviewNotes []string `json:"preview_notes,omitempty"`
|
||||
PreviewError string `json:"preview_error,omitempty"`
|
||||
LastUpdated time.Time `json:"last_updated"`
|
||||
// Format is what the catalog recorded. Anything but ICEBERG means the rows
|
||||
// below, if any, came from a worker rather than from metadata this server read.
|
||||
Format string `json:"format,omitempty"`
|
||||
PreviewedBy string `json:"previewed_by,omitempty"`
|
||||
}
|
||||
|
||||
// GetIcebergTableDataPreview walks the selected snapshot's manifests and reads
|
||||
@@ -92,6 +101,11 @@ func (s *AdminServer) GetIcebergTableDataPreview(ctx context.Context, catalogNam
|
||||
if err := s.executeS3TablesOperation(ctx, "GetTable", req, &resp); err != nil {
|
||||
return data, err
|
||||
}
|
||||
data.Format = resp.Format
|
||||
if !strings.EqualFold(resp.Format, s3tables.FormatIceberg) && resp.Format != "" {
|
||||
s.applyWorkerPreview(ctx, &data, bucketArn, namespaceParts, tableName, rowLimit)
|
||||
return data, nil
|
||||
}
|
||||
if resp.Metadata == nil || len(resp.Metadata.FullMetadata) == 0 {
|
||||
data.PreviewError = "Table has no Iceberg metadata."
|
||||
return data, nil
|
||||
@@ -467,3 +481,56 @@ func (w *sliceWriter) Write(p []byte) (int, error) {
|
||||
w.n += c
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// applyWorkerPreview asks the worker that last described this table for sample
|
||||
// rows. Admin has no reader for a format it does not implement, so this is the
|
||||
// only way the page shows anything but a location.
|
||||
//
|
||||
// The rows are fetched, never cached: they are the table's data rather than a
|
||||
// description of it, and a stale copy sitting in admin would be worse than
|
||||
// asking.
|
||||
func (s *AdminServer) applyWorkerPreview(ctx context.Context, data *IcebergDataPreviewData, bucketArn string, namespaceParts []string, tableName string, rowLimit int) {
|
||||
plugin := s.GetPlugin()
|
||||
if plugin == nil {
|
||||
data.PreviewError = fmt.Sprintf("Reading a %s table needs a plugin worker, and none is configured.", data.Format)
|
||||
return
|
||||
}
|
||||
bucketName, err := s3tables.ParseBucketNameFromARN(bucketArn)
|
||||
if err != nil {
|
||||
data.PreviewError = err.Error()
|
||||
return
|
||||
}
|
||||
|
||||
objectID := append(append([]string{bucketName}, namespaceParts...), tableName)
|
||||
requestCtx, cancel := context.WithTimeout(ctx, workerPreviewTimeout)
|
||||
defer cancel()
|
||||
|
||||
response, err := plugin.RequestObjectPreview(requestCtx, objectID, data.Format, rowLimit)
|
||||
if err != nil {
|
||||
data.PreviewError = fmt.Sprintf("Could not read this %s table: %v", data.Format, err)
|
||||
return
|
||||
}
|
||||
|
||||
data.Columns = response.Columns
|
||||
data.Rows = make([][]string, 0, len(response.Rows))
|
||||
for _, row := range response.Rows {
|
||||
cells := make([]string, len(row.Values))
|
||||
for i, value := range row.Values {
|
||||
cells[i] = truncateCell(value)
|
||||
}
|
||||
data.Rows = append(data.Rows, cells)
|
||||
}
|
||||
data.TotalRecords = response.TotalRows
|
||||
data.PreviewedBy = pluginWorkerForObject(plugin, objectID, data.Format)
|
||||
if int64(len(data.Rows)) < response.TotalRows {
|
||||
data.PreviewNotes = append(data.PreviewNotes, fmt.Sprintf("Showing %d of %d rows.", len(data.Rows), response.TotalRows))
|
||||
}
|
||||
}
|
||||
|
||||
// pluginWorkerForObject names the worker that answered, for the page to show.
|
||||
func pluginWorkerForObject(p *plugin.Plugin, objectID []string, format string) string {
|
||||
if observed, ok := p.Observations().GetFormat(objectID, format); ok {
|
||||
return observed.WorkerID
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -982,3 +982,27 @@ func parsePositiveInt(raw string, defaultValue int) int {
|
||||
}
|
||||
|
||||
// cloneConfigValueMap is now exported by the plugin package as CloneConfigValueMap
|
||||
|
||||
// GetPluginObservationsAPI returns what workers last reported about the objects
|
||||
// they inspected. Accepts an optional ?format= filter.
|
||||
//
|
||||
// These are cached, not live: a worker reports what it saw when it last looked,
|
||||
// and the timestamp is served with each one so a reader can judge the age.
|
||||
func (s *AdminServer) GetPluginObservationsAPI(w http.ResponseWriter, r *http.Request) {
|
||||
plugin := s.GetPlugin()
|
||||
if plugin == nil {
|
||||
writeJSON(w, http.StatusOK, []interface{}{})
|
||||
return
|
||||
}
|
||||
|
||||
formatFilter := strings.TrimSpace(r.URL.Query().Get("format"))
|
||||
observed := plugin.Observations().List()
|
||||
payload := make([]interface{}, 0, len(observed))
|
||||
for _, o := range observed {
|
||||
if formatFilter != "" && !strings.EqualFold(o.Format, formatFilter) {
|
||||
continue
|
||||
}
|
||||
payload = append(payload, o)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, payload)
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@ type S3TablesBucketsData struct {
|
||||
Buckets []S3TablesBucketSummary `json:"buckets"`
|
||||
TotalBuckets int `json:"total_buckets"`
|
||||
IcebergPort int `json:"iceberg_port"`
|
||||
LancePort int `json:"lance_port"`
|
||||
LastUpdated time.Time `json:"last_updated"`
|
||||
}
|
||||
|
||||
@@ -33,29 +34,39 @@ type S3TablesBucketSummary struct {
|
||||
Name string `json:"name"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
// Format is empty for a bucket created before formats were declared. Such a
|
||||
// bucket takes tables of either format, which is what it always did.
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type S3TablesNamespacesData struct {
|
||||
Username string `json:"username"`
|
||||
BucketARN string `json:"bucket_arn"`
|
||||
BucketFormat string `json:"bucket_format,omitempty"`
|
||||
Namespaces []s3tables.NamespaceSummary `json:"namespaces"`
|
||||
TotalNamespaces int `json:"total_namespaces"`
|
||||
LastUpdated time.Time `json:"last_updated"`
|
||||
}
|
||||
|
||||
type S3TablesTablesData struct {
|
||||
Username string `json:"username"`
|
||||
BucketARN string `json:"bucket_arn"`
|
||||
Namespace string `json:"namespace"`
|
||||
Tables []s3tables.TableSummary `json:"tables"`
|
||||
TotalTables int `json:"total_tables"`
|
||||
LastUpdated time.Time `json:"last_updated"`
|
||||
Username string `json:"username"`
|
||||
BucketARN string `json:"bucket_arn"`
|
||||
BucketFormat string `json:"bucket_format,omitempty"`
|
||||
Namespace string `json:"namespace"`
|
||||
Tables []s3tables.TableSummary `json:"tables"`
|
||||
TotalTables int `json:"total_tables"`
|
||||
// ObservedRows holds what a worker last counted, by table name, for formats
|
||||
// this server cannot read itself. A table nothing has looked at is absent
|
||||
// rather than zero: those are different facts.
|
||||
ObservedRows map[string]string `json:"observed_rows,omitempty"`
|
||||
LastUpdated time.Time `json:"last_updated"`
|
||||
}
|
||||
|
||||
type tableBucketMetadata struct {
|
||||
Name string `json:"name"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
// S3Tables manager helpers
|
||||
@@ -137,6 +148,7 @@ func (s *AdminServer) GetS3TablesBucketsData(ctx context.Context) (S3TablesBucke
|
||||
Name: entry.Entry.Name,
|
||||
OwnerAccountID: metadata.OwnerAccountID,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
Format: metadata.Format,
|
||||
})
|
||||
}
|
||||
return nil
|
||||
@@ -148,10 +160,60 @@ func (s *AdminServer) GetS3TablesBucketsData(ctx context.Context) (S3TablesBucke
|
||||
Buckets: buckets,
|
||||
TotalBuckets: len(buckets),
|
||||
IcebergPort: s.icebergPort,
|
||||
LancePort: s.lancePort,
|
||||
LastUpdated: time.Now(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// observedRowCounts collects what workers last reported for these tables. For a
|
||||
// format admin cannot read, this is the only row count that exists.
|
||||
func (s *AdminServer) observedRowCounts(bucketArn string, namespaceParts []string, tables []s3tables.TableSummary) map[string]string {
|
||||
plugin := s.GetPlugin()
|
||||
if plugin == nil || len(tables) == 0 {
|
||||
return nil
|
||||
}
|
||||
bucketName, err := s3tables.ParseBucketNameFromARN(bucketArn)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
counts := make(map[string]string)
|
||||
for _, table := range tables {
|
||||
objectID := append(append([]string{bucketName}, namespaceParts...), table.Name)
|
||||
if observed, ok := plugin.Observations().GetFormat(objectID, table.Format); ok {
|
||||
if rows := observed.AttributeString("rows"); rows != "" {
|
||||
counts[table.Name] = rows
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(counts) == 0 {
|
||||
return nil
|
||||
}
|
||||
return counts
|
||||
}
|
||||
|
||||
// catalogPortForFormat is the port serving a format, or 0 when this cluster
|
||||
// does not run that catalog.
|
||||
func (s *AdminServer) catalogPortForFormat(format string) int {
|
||||
if strings.EqualFold(format, s3tables.FormatLance) {
|
||||
return s.lancePort
|
||||
}
|
||||
return s.icebergPort
|
||||
}
|
||||
|
||||
// tableBucketFormat reports what the bucket says it holds, or "" for one made
|
||||
// before the declaration existed. A page inside a bucket asks so it can label
|
||||
// itself and constrain what can be created; failing to read it is not worth
|
||||
// failing the page over, so it degrades to undeclared.
|
||||
func (s *AdminServer) tableBucketFormat(ctx context.Context, bucketArn string) string {
|
||||
var resp s3tables.GetTableBucketResponse
|
||||
req := &s3tables.GetTableBucketRequest{TableBucketARN: bucketArn}
|
||||
if err := s.executeS3TablesOperation(ctx, "GetTableBucket", req, &resp); err != nil {
|
||||
glog.V(1).Infof("S3Tables: failed to read format of %s: %v", bucketArn, err)
|
||||
return ""
|
||||
}
|
||||
return resp.Format
|
||||
}
|
||||
|
||||
func (s *AdminServer) GetS3TablesNamespacesData(ctx context.Context, bucketArn string) (S3TablesNamespacesData, error) {
|
||||
var resp s3tables.ListNamespacesResponse
|
||||
req := &s3tables.ListNamespacesRequest{TableBucketARN: bucketArn, MaxNamespaces: s3TablesAdminListLimit}
|
||||
@@ -160,6 +222,7 @@ func (s *AdminServer) GetS3TablesNamespacesData(ctx context.Context, bucketArn s
|
||||
}
|
||||
return S3TablesNamespacesData{
|
||||
BucketARN: bucketArn,
|
||||
BucketFormat: s.tableBucketFormat(ctx, bucketArn),
|
||||
Namespaces: resp.Namespaces,
|
||||
TotalNamespaces: len(resp.Namespaces),
|
||||
LastUpdated: time.Now(),
|
||||
@@ -180,13 +243,16 @@ func (s *AdminServer) GetS3TablesTablesData(ctx context.Context, bucketArn, name
|
||||
if err := s.executeS3TablesOperation(ctx, "ListTables", req, &resp); err != nil {
|
||||
return S3TablesTablesData{}, err
|
||||
}
|
||||
return S3TablesTablesData{
|
||||
BucketARN: bucketArn,
|
||||
Namespace: namespace,
|
||||
Tables: resp.Tables,
|
||||
TotalTables: len(resp.Tables),
|
||||
LastUpdated: time.Now(),
|
||||
}, nil
|
||||
data := S3TablesTablesData{
|
||||
BucketARN: bucketArn,
|
||||
BucketFormat: s.tableBucketFormat(ctx, bucketArn),
|
||||
Namespace: namespace,
|
||||
Tables: resp.Tables,
|
||||
TotalTables: len(resp.Tables),
|
||||
LastUpdated: time.Now(),
|
||||
}
|
||||
data.ObservedRows = s.observedRowCounts(bucketArn, ns, resp.Tables)
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// Iceberg Catalog data providers
|
||||
@@ -299,9 +365,75 @@ func (s *AdminServer) GetIcebergTableDetailsData(ctx context.Context, catalogNam
|
||||
}
|
||||
|
||||
applyIcebergMetadata(resp.Metadata, &details)
|
||||
s.applyWorkerObservation(&details, bucketArn, namespaceParts, resp.Name)
|
||||
return details, nil
|
||||
}
|
||||
|
||||
// applyWorkerObservation fills in what a plugin worker last reported about a
|
||||
// table this server cannot read itself. Only the catalog knows a Lance table
|
||||
// exists; only a worker with the format's runtime can say what is in it, so
|
||||
// without this the page has nothing to show but a location.
|
||||
//
|
||||
// The observation is cached, not live, which is why the page carries the time
|
||||
// and the worker that took it.
|
||||
func (s *AdminServer) applyWorkerObservation(details *IcebergTableDetailsData, bucketArn string, namespaceParts []string, tableName string) {
|
||||
if len(details.SchemaFields) > 0 {
|
||||
return
|
||||
}
|
||||
plugin := s.GetPlugin()
|
||||
if plugin == nil {
|
||||
return
|
||||
}
|
||||
bucketName, err := s3tables.ParseBucketNameFromARN(bucketArn)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
objectID := append(append([]string{bucketName}, namespaceParts...), tableName)
|
||||
observed, ok := plugin.Observations().GetFormat(objectID, details.Format)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
details.ObservedBy = observed.WorkerID
|
||||
details.ObservedAt = observed.ObservedAt
|
||||
details.SchemaFields = observationSchemaFields(observed.AttributeString("schema"))
|
||||
for _, name := range []string{"rows", "fragments", "versions"} {
|
||||
if value := observed.AttributeString(name); value != "" {
|
||||
details.Properties = append(details.Properties, IcebergPropertyInfo{Key: name, Value: value})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// observationSchemaFields parses the schema a worker reported. A schema it
|
||||
// could not render is not worth failing the page over.
|
||||
func observationSchemaFields(schema string) []IcebergSchemaFieldInfo {
|
||||
if schema == "" {
|
||||
return nil
|
||||
}
|
||||
var reported []struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Nullable bool `json:"nullable"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(schema), &reported); err != nil {
|
||||
return nil
|
||||
}
|
||||
fields := make([]IcebergSchemaFieldInfo, 0, len(reported))
|
||||
for i, field := range reported {
|
||||
encoded, err := json.Marshal(field.Type)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
fields = append(fields, IcebergSchemaFieldInfo{
|
||||
ID: i + 1,
|
||||
Name: field.Name,
|
||||
Type: encoded,
|
||||
Required: !field.Nullable,
|
||||
})
|
||||
}
|
||||
return fields
|
||||
}
|
||||
|
||||
type icebergFullMetadata struct {
|
||||
FormatVersion int `json:"format-version"`
|
||||
TableUUID string `json:"table-uuid"`
|
||||
@@ -582,9 +714,10 @@ func (s *AdminServer) CreateS3TablesBucket(w http.ResponseWriter, r *http.Reques
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Name string `json:"name"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
Owner string `json:"owner"`
|
||||
Name string `json:"name"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
Owner string `json:"owner"`
|
||||
Format string `json:"format"`
|
||||
}
|
||||
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
|
||||
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
|
||||
@@ -605,7 +738,24 @@ func (s *AdminServer) CreateS3TablesBucket(w http.ResponseWriter, r *http.Reques
|
||||
return
|
||||
}
|
||||
}
|
||||
createReq := &s3tables.CreateTableBucketRequest{Name: req.Name, Tags: req.Tags}
|
||||
format := s3tables.FormatIceberg
|
||||
if req.Format != "" {
|
||||
normalized, ok := s3tables.NormalizeFormat(req.Format)
|
||||
if !ok {
|
||||
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Unsupported format %q", req.Format))
|
||||
return
|
||||
}
|
||||
format = normalized
|
||||
}
|
||||
// A bucket of a format this cluster does not serve is a bucket no client can
|
||||
// reach. The picker disables the option; refuse it here too, since the API
|
||||
// is reachable without the page.
|
||||
if port := s.catalogPortForFormat(format); port == 0 {
|
||||
writeJSONError(w, http.StatusBadRequest,
|
||||
fmt.Sprintf("No %s endpoint is configured, so a %s bucket would be unreachable", format, format))
|
||||
return
|
||||
}
|
||||
createReq := &s3tables.CreateTableBucketRequest{Name: req.Name, Tags: req.Tags, Format: format}
|
||||
var resp s3tables.CreateTableBucketResponse
|
||||
if err := s.executeS3TablesOperation(r.Context(), "CreateTableBucket", createReq, &resp); err != nil {
|
||||
writeS3TablesError(w, err)
|
||||
|
||||
@@ -778,4 +778,8 @@ type IcebergTableDetailsData struct {
|
||||
TotalSizeBytes int64 `json:"total_size_bytes"`
|
||||
HasTotalSize bool `json:"has_total_size"`
|
||||
MetadataError string `json:"metadata_error,omitempty"`
|
||||
// Set when the details came from a plugin worker rather than from metadata
|
||||
// this server can read, so the page can say whose account it is and when.
|
||||
ObservedBy string `json:"observed_by,omitempty"`
|
||||
ObservedAt time.Time `json:"observed_at,omitempty"`
|
||||
}
|
||||
|
||||
@@ -266,6 +266,7 @@ func (h *AdminHandlers) registerAPIRoutes(api *mux.Router, enforceWrite bool) {
|
||||
pluginApi.HandleFunc("/jobs/{jobId}/detail", h.adminServer.GetPluginJobDetailAPI).Methods(http.MethodGet)
|
||||
pluginApi.HandleFunc("/activities", h.adminServer.GetPluginActivitiesAPI).Methods(http.MethodGet)
|
||||
pluginApi.HandleFunc("/scheduler-states", h.adminServer.GetPluginSchedulerStatesAPI).Methods(http.MethodGet)
|
||||
pluginApi.HandleFunc("/observations", h.adminServer.GetPluginObservationsAPI).Methods(http.MethodGet)
|
||||
pluginApi.HandleFunc("/scheduler-status", h.adminServer.GetPluginSchedulerStatusAPI).Methods(http.MethodGet)
|
||||
pluginApi.HandleFunc("/job-types/{jobType}/descriptor", h.adminServer.GetPluginJobTypeDescriptorAPI).Methods(http.MethodGet)
|
||||
pluginApi.HandleFunc("/job-types/{jobType}/schema", h.adminServer.RequestPluginJobTypeSchemaAPI).Methods(http.MethodPost)
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
package plugin
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
||||
)
|
||||
|
||||
// maxObservations caps the store. Observations are a convenience for display,
|
||||
// so a cluster with more objects than this loses the oldest rather than growing
|
||||
// admin's memory without limit.
|
||||
const maxObservations = 10000
|
||||
|
||||
// observationKey identifies one observed object across workers.
|
||||
func observationKey(objectID []string) string {
|
||||
return strings.Join(objectID, "\x1f")
|
||||
}
|
||||
|
||||
// ObservationStore keeps the last thing a worker said about each object.
|
||||
//
|
||||
// It is deliberately not authoritative: a worker reports what it saw when it
|
||||
// last looked, and admin serves that back with its timestamp so a reader can
|
||||
// judge how stale it is. Nothing schedules work from it.
|
||||
type ObservationStore struct {
|
||||
mu sync.RWMutex
|
||||
entries map[string]*Observation
|
||||
}
|
||||
|
||||
// Observation is one object as a worker last reported it.
|
||||
type Observation struct {
|
||||
ObjectID []string `json:"object_id"`
|
||||
ObjectKind string `json:"object_kind"`
|
||||
Format string `json:"format"`
|
||||
// Attributes are flattened on the way in: they exist to be displayed and
|
||||
// served as JSON, and the typed form is the worker's business.
|
||||
Attributes map[string]interface{} `json:"attributes"`
|
||||
JobType string `json:"job_type"`
|
||||
WorkerID string `json:"worker_id"`
|
||||
ObservedAt time.Time `json:"observed_at"`
|
||||
}
|
||||
|
||||
func NewObservationStore() *ObservationStore {
|
||||
return &ObservationStore{entries: make(map[string]*Observation)}
|
||||
}
|
||||
|
||||
// Record stores what one worker reported, replacing whatever was there for the
|
||||
// same object. A later observation from a different worker still wins: the
|
||||
// newest look at an object is the useful one.
|
||||
func (s *ObservationStore) Record(workerID string, report *plugin_pb.WorkerObservations) {
|
||||
if report == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
for _, observed := range report.Observations {
|
||||
if observed == nil || len(observed.ObjectId) == 0 {
|
||||
continue
|
||||
}
|
||||
at := time.Now()
|
||||
if observed.ObservedAt != nil {
|
||||
at = observed.ObservedAt.AsTime()
|
||||
}
|
||||
s.entries[observationKey(observed.ObjectId)] = &Observation{
|
||||
ObjectID: observed.ObjectId,
|
||||
ObjectKind: observed.ObjectKind,
|
||||
Format: observed.Format,
|
||||
Attributes: flattenAttributes(observed.Attributes),
|
||||
JobType: report.JobType,
|
||||
WorkerID: workerID,
|
||||
ObservedAt: at,
|
||||
}
|
||||
}
|
||||
s.evictOldest()
|
||||
}
|
||||
|
||||
// evictOldest trims the store back under the cap. Called with the lock held.
|
||||
func (s *ObservationStore) evictOldest() {
|
||||
if len(s.entries) <= maxObservations {
|
||||
return
|
||||
}
|
||||
keys := make([]string, 0, len(s.entries))
|
||||
for key := range s.entries {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Slice(keys, func(i, j int) bool {
|
||||
return s.entries[keys[i]].ObservedAt.Before(s.entries[keys[j]].ObservedAt)
|
||||
})
|
||||
for _, key := range keys[:len(s.entries)-maxObservations] {
|
||||
delete(s.entries, key)
|
||||
}
|
||||
}
|
||||
|
||||
// flattenAttributes unwraps the ConfigValue envelope that configValueMapToPlain
|
||||
// preserves. An observation exists to be read, and {"int64_value":"16"} is not
|
||||
// a number anyone wants to render; a protojson int64 also arrives as a string,
|
||||
// so it is converted back.
|
||||
func flattenAttributes(values map[string]*plugin_pb.ConfigValue) map[string]interface{} {
|
||||
plain := configValueMapToPlain(values)
|
||||
if plain == nil {
|
||||
return nil
|
||||
}
|
||||
flat := make(map[string]interface{}, len(plain))
|
||||
for name, value := range plain {
|
||||
wrapper, ok := value.(map[string]interface{})
|
||||
if !ok || len(wrapper) != 1 {
|
||||
flat[name] = value
|
||||
continue
|
||||
}
|
||||
for kind, inner := range wrapper {
|
||||
if kind == "int64_value" {
|
||||
if text, ok := inner.(string); ok {
|
||||
if parsed, err := strconv.ParseInt(text, 10, 64); err == nil {
|
||||
flat[name] = parsed
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
flat[name] = inner
|
||||
}
|
||||
}
|
||||
return flat
|
||||
}
|
||||
|
||||
// Get returns the last observation of one object, if any.
|
||||
func (s *ObservationStore) Get(objectID []string) (*Observation, bool) {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
observed, ok := s.entries[observationKey(objectID)]
|
||||
return observed, ok
|
||||
}
|
||||
|
||||
// GetFormat returns the last observation of one object only when it describes
|
||||
// the format asked for. A table can be dropped and remade in another format at
|
||||
// the same path, and the stale observation would then describe something that
|
||||
// no longer exists.
|
||||
func (s *ObservationStore) GetFormat(objectID []string, format string) (*Observation, bool) {
|
||||
observed, ok := s.Get(objectID)
|
||||
if !ok || !strings.EqualFold(observed.Format, format) {
|
||||
return nil, false
|
||||
}
|
||||
return observed, true
|
||||
}
|
||||
|
||||
// List returns every observation, newest first.
|
||||
func (s *ObservationStore) List() []*Observation {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
all := make([]*Observation, 0, len(s.entries))
|
||||
for _, observed := range s.entries {
|
||||
all = append(all, observed)
|
||||
}
|
||||
sort.Slice(all, func(i, j int) bool {
|
||||
return all[i].ObservedAt.After(all[j].ObservedAt)
|
||||
})
|
||||
return all
|
||||
}
|
||||
|
||||
// AttributeString renders one attribute for display, or "" when it is absent.
|
||||
func (o *Observation) AttributeString(name string) string {
|
||||
if o == nil {
|
||||
return ""
|
||||
}
|
||||
value, ok := o.Attributes[name]
|
||||
if !ok || value == nil {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("%v", value)
|
||||
}
|
||||
|
||||
// Observations exposes the store so admin handlers can serve it.
|
||||
func (r *Plugin) Observations() *ObservationStore {
|
||||
return r.observations
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package plugin
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
||||
"google.golang.org/protobuf/types/known/timestamppb"
|
||||
)
|
||||
|
||||
func intValue(v int64) *plugin_pb.ConfigValue {
|
||||
return &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_Int64Value{Int64Value: v}}
|
||||
}
|
||||
|
||||
func stringValue(v string) *plugin_pb.ConfigValue {
|
||||
return &plugin_pb.ConfigValue{Kind: &plugin_pb.ConfigValue_StringValue{StringValue: v}}
|
||||
}
|
||||
|
||||
// An observation exists to be rendered, so the ConfigValue envelope comes off
|
||||
// on the way in. A page asking for "fragments" wants 16, not a wrapper holding
|
||||
// the string "16".
|
||||
func TestObservationAttributesAreFlattened(t *testing.T) {
|
||||
store := NewObservationStore()
|
||||
store.Record("worker-1", &plugin_pb.WorkerObservations{
|
||||
JobType: "lance_compact",
|
||||
Observations: []*plugin_pb.ObjectObservation{{
|
||||
ObjectId: []string{"bucket", "ns", "table"},
|
||||
ObjectKind: "table",
|
||||
Format: "LANCE",
|
||||
Attributes: map[string]*plugin_pb.ConfigValue{
|
||||
"fragments": intValue(16),
|
||||
"schema": stringValue(`[{"name":"id"}]`),
|
||||
},
|
||||
ObservedAt: timestamppb.New(time.Now()),
|
||||
}},
|
||||
})
|
||||
|
||||
observed, ok := store.Get([]string{"bucket", "ns", "table"})
|
||||
if !ok {
|
||||
t.Fatal("observation was not stored")
|
||||
}
|
||||
if got, ok := observed.Attributes["fragments"].(int64); !ok || got != 16 {
|
||||
t.Fatalf("fragments = %#v, want int64 16", observed.Attributes["fragments"])
|
||||
}
|
||||
if got := observed.AttributeString("fragments"); got != "16" {
|
||||
t.Fatalf("AttributeString(fragments) = %q, want \"16\"", got)
|
||||
}
|
||||
if got := observed.AttributeString("schema"); got != `[{"name":"id"}]` {
|
||||
t.Fatalf("AttributeString(schema) = %q", got)
|
||||
}
|
||||
if observed.WorkerID != "worker-1" || observed.JobType != "lance_compact" {
|
||||
t.Fatalf("provenance lost: %+v", observed)
|
||||
}
|
||||
}
|
||||
|
||||
// The newest look at an object is the useful one, whichever worker took it.
|
||||
func TestObservationIsReplacedByALaterOne(t *testing.T) {
|
||||
store := NewObservationStore()
|
||||
record := func(worker string, fragments int64) {
|
||||
store.Record(worker, &plugin_pb.WorkerObservations{
|
||||
JobType: "lance_compact",
|
||||
Observations: []*plugin_pb.ObjectObservation{{
|
||||
ObjectId: []string{"bucket", "ns", "table"},
|
||||
Format: "LANCE",
|
||||
Attributes: map[string]*plugin_pb.ConfigValue{"fragments": intValue(fragments)},
|
||||
ObservedAt: timestamppb.New(time.Now()),
|
||||
}},
|
||||
})
|
||||
}
|
||||
record("worker-1", 16)
|
||||
record("worker-2", 1)
|
||||
|
||||
observed, _ := store.Get([]string{"bucket", "ns", "table"})
|
||||
if got := observed.AttributeString("fragments"); got != "1" {
|
||||
t.Fatalf("fragments = %q, want the later observation's 1", got)
|
||||
}
|
||||
if observed.WorkerID != "worker-2" {
|
||||
t.Fatalf("worker = %q, want worker-2", observed.WorkerID)
|
||||
}
|
||||
if len(store.List()) != 1 {
|
||||
t.Fatalf("store holds %d entries, want 1", len(store.List()))
|
||||
}
|
||||
}
|
||||
|
||||
// Missing attributes read as empty rather than panicking a template.
|
||||
func TestObservationAttributeStringHandlesAbsence(t *testing.T) {
|
||||
var absent *Observation
|
||||
if got := absent.AttributeString("fragments"); got != "" {
|
||||
t.Fatalf("nil observation returned %q", got)
|
||||
}
|
||||
store := NewObservationStore()
|
||||
store.Record("worker-1", &plugin_pb.WorkerObservations{
|
||||
Observations: []*plugin_pb.ObjectObservation{{
|
||||
ObjectId: []string{"b", "n", "t"},
|
||||
}},
|
||||
})
|
||||
observed, _ := store.Get([]string{"b", "n", "t"})
|
||||
if got := observed.AttributeString("nothing"); got != "" {
|
||||
t.Fatalf("absent attribute returned %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A table path can be dropped and remade in another format. The observation left
|
||||
// behind describes something that is no longer there, so a caller asking about
|
||||
// the new format must not be handed it.
|
||||
func TestObservationLookupIsScopedToFormat(t *testing.T) {
|
||||
store := NewObservationStore()
|
||||
objectID := []string{"bucket", "ns", "table"}
|
||||
store.Record("worker-1", &plugin_pb.WorkerObservations{
|
||||
Observations: []*plugin_pb.ObjectObservation{{
|
||||
ObjectId: objectID,
|
||||
Format: "LANCE",
|
||||
ObservedAt: timestamppb.New(time.Now()),
|
||||
}},
|
||||
})
|
||||
|
||||
if _, ok := store.GetFormat(objectID, "ICEBERG"); ok {
|
||||
t.Fatal("a LANCE observation was returned for an ICEBERG table")
|
||||
}
|
||||
if _, ok := store.GetFormat(objectID, "lance"); !ok {
|
||||
t.Fatal("format matching must not depend on case")
|
||||
}
|
||||
if _, ok := store.GetFormat(objectID, ""); ok {
|
||||
t.Fatal("an unknown format must not match a recorded one")
|
||||
}
|
||||
}
|
||||
@@ -97,6 +97,11 @@ type Plugin struct {
|
||||
pendingExecutionMu sync.Mutex
|
||||
pendingExecution map[string]chan *plugin_pb.JobCompleted
|
||||
|
||||
pendingPreviewMu sync.Mutex
|
||||
pendingPreview map[string]chan *plugin_pb.ObjectPreviewResponse
|
||||
|
||||
observations *ObservationStore
|
||||
|
||||
jobsMu sync.RWMutex
|
||||
jobs map[string]*TrackedJob
|
||||
// serialize stale job cleanup to avoid duplicate expirations
|
||||
@@ -180,6 +185,8 @@ func New(options Options) (*Plugin, error) {
|
||||
pendingSchema: make(map[string]chan *plugin_pb.ConfigSchemaResponse),
|
||||
pendingDetection: make(map[string]*pendingDetectionState),
|
||||
pendingExecution: make(map[string]chan *plugin_pb.JobCompleted),
|
||||
pendingPreview: make(map[string]chan *plugin_pb.ObjectPreviewResponse),
|
||||
observations: NewObservationStore(),
|
||||
nextDetectionAt: make(map[string]time.Time),
|
||||
detectionInFlight: make(map[string]bool),
|
||||
detectorLeases: make(map[string]string),
|
||||
@@ -235,6 +242,13 @@ func (r *Plugin) Shutdown() {
|
||||
}
|
||||
r.pendingSchemaMu.Unlock()
|
||||
|
||||
r.pendingPreviewMu.Lock()
|
||||
for requestID, ch := range r.pendingPreview {
|
||||
close(ch)
|
||||
delete(r.pendingPreview, requestID)
|
||||
}
|
||||
r.pendingPreviewMu.Unlock()
|
||||
|
||||
r.pendingDetectionMu.Lock()
|
||||
for requestID, state := range r.pendingDetection {
|
||||
close(state.complete)
|
||||
@@ -978,6 +992,10 @@ func (r *Plugin) handleWorkerMessage(workerID string, message *plugin_pb.WorkerT
|
||||
r.handleJobProgressUpdate(workerID, body.JobProgressUpdate)
|
||||
case *plugin_pb.WorkerToAdminMessage_JobCompleted:
|
||||
r.handleJobCompleted(body.JobCompleted)
|
||||
case *plugin_pb.WorkerToAdminMessage_Observations:
|
||||
r.observations.Record(workerID, body.Observations)
|
||||
case *plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse:
|
||||
r.handleObjectPreviewResponse(body.ObjectPreviewResponse)
|
||||
case *plugin_pb.WorkerToAdminMessage_Acknowledge:
|
||||
if !body.Acknowledge.Accepted {
|
||||
glog.Warningf("Plugin worker %s rejected request %s: %s", workerID, body.Acknowledge.RequestId, body.Acknowledge.Message)
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package plugin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
||||
"google.golang.org/protobuf/types/known/timestamppb"
|
||||
)
|
||||
|
||||
// maxPreviewRows caps what admin will ask for. A preview is a look at the
|
||||
// object, not an export, and the rows cross the control stream.
|
||||
const maxPreviewRows = 200
|
||||
|
||||
// RequestObjectPreview asks the worker that last described an object for sample
|
||||
// rows of it, and waits for the answer.
|
||||
//
|
||||
// The worker is chosen from the observation store rather than by job type: the
|
||||
// one that last looked at this object is the one that can read it, and it says
|
||||
// so by having reported. That also means a preview is only available once
|
||||
// detection has run, which the caller should surface rather than hide.
|
||||
func (r *Plugin) RequestObjectPreview(ctx context.Context, objectID []string, format string, rowLimit int) (*plugin_pb.ObjectPreviewResponse, error) {
|
||||
if len(objectID) == 0 {
|
||||
return nil, fmt.Errorf("preview needs an object id")
|
||||
}
|
||||
// The conversion happens only inside the bound check, so what reaches the
|
||||
// wire cannot depend on int being wider than int32 here.
|
||||
requestedRows := int32(maxPreviewRows)
|
||||
if rowLimit >= 1 && rowLimit <= maxPreviewRows {
|
||||
requestedRows = int32(rowLimit)
|
||||
}
|
||||
|
||||
observed, ok := r.observations.GetFormat(objectID, format)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("no worker has described this object as %s yet", format)
|
||||
}
|
||||
if _, connected := r.registry.Get(observed.WorkerID); !connected {
|
||||
return nil, fmt.Errorf("worker %s is not connected", observed.WorkerID)
|
||||
}
|
||||
|
||||
requestID, err := newRequestID("preview")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
responseCh := make(chan *plugin_pb.ObjectPreviewResponse, 1)
|
||||
r.pendingPreviewMu.Lock()
|
||||
r.pendingPreview[requestID] = responseCh
|
||||
r.pendingPreviewMu.Unlock()
|
||||
defer func() {
|
||||
r.pendingPreviewMu.Lock()
|
||||
delete(r.pendingPreview, requestID)
|
||||
r.pendingPreviewMu.Unlock()
|
||||
}()
|
||||
|
||||
request := &plugin_pb.AdminToWorkerMessage{
|
||||
RequestId: requestID,
|
||||
SentAt: timestamppb.Now(),
|
||||
Body: &plugin_pb.AdminToWorkerMessage_RequestObjectPreview{
|
||||
RequestObjectPreview: &plugin_pb.RequestObjectPreview{
|
||||
ObjectId: objectID,
|
||||
Format: format,
|
||||
RowLimit: requestedRows,
|
||||
},
|
||||
},
|
||||
}
|
||||
if err := r.sendToWorker(observed.WorkerID, request); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case response, ok := <-responseCh:
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("preview request %s interrupted", requestID)
|
||||
}
|
||||
if response == nil {
|
||||
return nil, fmt.Errorf("preview request %s returned nothing", requestID)
|
||||
}
|
||||
if !response.Success {
|
||||
return nil, fmt.Errorf("worker %s could not preview the object: %s", observed.WorkerID, response.ErrorMessage)
|
||||
}
|
||||
return response, nil
|
||||
}
|
||||
}
|
||||
|
||||
// handleObjectPreviewResponse routes one reply back to whoever is waiting.
|
||||
func (r *Plugin) handleObjectPreviewResponse(response *plugin_pb.ObjectPreviewResponse) {
|
||||
if response == nil {
|
||||
return
|
||||
}
|
||||
// Held across the send: Shutdown closes these channels under the same lock,
|
||||
// and a send that raced it would panic on a closed channel.
|
||||
r.pendingPreviewMu.Lock()
|
||||
defer r.pendingPreviewMu.Unlock()
|
||||
ch := r.pendingPreview[response.RequestId]
|
||||
if ch == nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case ch <- response:
|
||||
default:
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
package plugin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
|
||||
"google.golang.org/protobuf/types/known/timestamppb"
|
||||
)
|
||||
|
||||
func previewPlugin(t *testing.T, workerID string) (*Plugin, *streamSession) {
|
||||
t.Helper()
|
||||
pluginSvc, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New plugin error: %v", err)
|
||||
}
|
||||
t.Cleanup(pluginSvc.Shutdown)
|
||||
|
||||
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{WorkerId: workerID})
|
||||
session := &streamSession{workerID: workerID, outgoing: make(chan *plugin_pb.AdminToWorkerMessage, 4), done: make(chan struct{})}
|
||||
pluginSvc.putSession(session)
|
||||
return pluginSvc, session
|
||||
}
|
||||
|
||||
func observeObject(p *Plugin, workerID string, objectID []string) {
|
||||
p.observations.Record(workerID, &plugin_pb.WorkerObservations{
|
||||
JobType: "lance_compact",
|
||||
Observations: []*plugin_pb.ObjectObservation{{
|
||||
ObjectId: objectID,
|
||||
Format: "LANCE",
|
||||
ObservedAt: timestamppb.Now(),
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
// The round trip the details page makes: ask the worker that described this
|
||||
// object, and render what it sends back.
|
||||
func TestRequestObjectPreviewReturnsTheWorkersRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
const workerID = "lance-worker-1"
|
||||
objectID := []string{"vectors", "ml", "embeddings"}
|
||||
pluginSvc, session := previewPlugin(t, workerID)
|
||||
observeObject(pluginSvc, workerID, objectID)
|
||||
|
||||
type result struct {
|
||||
response *plugin_pb.ObjectPreviewResponse
|
||||
err error
|
||||
}
|
||||
results := make(chan result, 1)
|
||||
go func() {
|
||||
response, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 2)
|
||||
results <- result{response, err}
|
||||
}()
|
||||
|
||||
var request *plugin_pb.AdminToWorkerMessage
|
||||
select {
|
||||
case request = <-session.outgoing:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("no preview request reached the worker")
|
||||
}
|
||||
asked := request.GetRequestObjectPreview()
|
||||
if asked == nil {
|
||||
t.Fatalf("expected a preview request, got %T", request.Body)
|
||||
}
|
||||
if asked.RowLimit != 2 || asked.Format != "LANCE" {
|
||||
t.Fatalf("request lost its parameters: %+v", asked)
|
||||
}
|
||||
|
||||
pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{
|
||||
WorkerId: workerID,
|
||||
Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{
|
||||
ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{
|
||||
RequestId: request.RequestId,
|
||||
Success: true,
|
||||
Columns: []string{"id", "vec"},
|
||||
Rows: []*plugin_pb.PreviewRow{{Values: []string{"1", "[0.1, 0.2]"}}},
|
||||
TotalRows: 1024,
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
select {
|
||||
case got := <-results:
|
||||
if got.err != nil {
|
||||
t.Fatalf("preview failed: %v", got.err)
|
||||
}
|
||||
if len(got.response.Rows) != 1 || got.response.Rows[0].Values[1] != "[0.1, 0.2]" {
|
||||
t.Fatalf("rows did not survive the round trip: %+v", got.response.Rows)
|
||||
}
|
||||
if got.response.TotalRows != 1024 {
|
||||
t.Fatalf("total_rows = %d, want 1024", got.response.TotalRows)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("preview never returned")
|
||||
}
|
||||
}
|
||||
|
||||
// A worker that cannot read the object says so, and the page shows its reason
|
||||
// rather than an empty table.
|
||||
func TestRequestObjectPreviewSurfacesTheWorkersError(t *testing.T) {
|
||||
t.Parallel()
|
||||
const workerID = "lance-worker-1"
|
||||
objectID := []string{"vectors", "ml", "embeddings"}
|
||||
pluginSvc, session := previewPlugin(t, workerID)
|
||||
observeObject(pluginSvc, workerID, objectID)
|
||||
|
||||
errs := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10)
|
||||
errs <- err
|
||||
}()
|
||||
|
||||
request := <-session.outgoing
|
||||
pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{
|
||||
WorkerId: workerID,
|
||||
Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{
|
||||
ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{
|
||||
RequestId: request.RequestId,
|
||||
Success: false,
|
||||
ErrorMessage: "open lance dataset: access denied",
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
select {
|
||||
case err := <-errs:
|
||||
if err == nil || !strings.Contains(err.Error(), "access denied") {
|
||||
t.Fatalf("error = %v, want the worker's reason", err)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("preview never returned")
|
||||
}
|
||||
}
|
||||
|
||||
// Nothing has looked at the object, so there is nobody to ask. Worth its own
|
||||
// message: it means detection has not run, not that the table is unreadable.
|
||||
func TestRequestObjectPreviewNeedsAnObservation(t *testing.T) {
|
||||
t.Parallel()
|
||||
pluginSvc, _ := previewPlugin(t, "lance-worker-1")
|
||||
|
||||
_, err := pluginSvc.RequestObjectPreview(context.Background(), []string{"vectors", "ml", "nothing"}, "LANCE", 10)
|
||||
if err == nil || !strings.Contains(err.Error(), "described") {
|
||||
t.Fatalf("error = %v, want one naming the missing description", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The worker that described the object has since gone away.
|
||||
func TestRequestObjectPreviewNeedsAConnectedWorker(t *testing.T) {
|
||||
t.Parallel()
|
||||
objectID := []string{"vectors", "ml", "embeddings"}
|
||||
pluginSvc, _ := previewPlugin(t, "lance-worker-1")
|
||||
observeObject(pluginSvc, "lance-worker-gone", objectID)
|
||||
|
||||
_, err := pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10)
|
||||
if err == nil || !strings.Contains(err.Error(), "not connected") {
|
||||
t.Fatalf("error = %v, want one naming the absent worker", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown closes every pending channel, and a reply can arrive at the same
|
||||
// moment; delivering it outside the lock panics with "send on closed channel".
|
||||
// The window is narrow enough that this test does not reliably reproduce it -
|
||||
// widening it with a Gosched between the lookup and the send does, every time -
|
||||
// so treat this as exercising the path rather than as a regression alarm.
|
||||
func TestObjectPreviewResponseRacesShutdown(t *testing.T) {
|
||||
t.Parallel()
|
||||
const workerID = "lance-worker-1"
|
||||
objectID := []string{"vectors", "ml", "embeddings"}
|
||||
|
||||
for i := 0; i < 50; i++ {
|
||||
pluginSvc, session := previewPlugin(t, workerID)
|
||||
observeObject(pluginSvc, workerID, objectID)
|
||||
|
||||
go func() {
|
||||
_, _ = pluginSvc.RequestObjectPreview(context.Background(), objectID, "LANCE", 10)
|
||||
}()
|
||||
request := <-session.outgoing
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(2)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
pluginSvc.Shutdown()
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
pluginSvc.handleWorkerMessage(workerID, &plugin_pb.WorkerToAdminMessage{
|
||||
WorkerId: workerID,
|
||||
Body: &plugin_pb.WorkerToAdminMessage_ObjectPreviewResponse{
|
||||
ObjectPreviewResponse: &plugin_pb.ObjectPreviewResponse{
|
||||
RequestId: request.RequestId,
|
||||
Success: true,
|
||||
},
|
||||
},
|
||||
})
|
||||
}()
|
||||
wg.Wait()
|
||||
}
|
||||
}
|
||||
@@ -22,11 +22,15 @@ const (
|
||||
// LaneLifecycle handles S3 object store lifecycle management
|
||||
// (expiration, transition, abort incomplete multipart uploads).
|
||||
LaneLifecycle SchedulerLane = "lifecycle"
|
||||
|
||||
// LaneLance handles table-bucket Lance maintenance: fragment compaction,
|
||||
// index optimization and version cleanup.
|
||||
LaneLance SchedulerLane = "lance"
|
||||
)
|
||||
|
||||
// AllLanes returns every defined scheduler lane in a stable order.
|
||||
func AllLanes() []SchedulerLane {
|
||||
return []SchedulerLane{LaneDefault, LaneIceberg, LaneLifecycle}
|
||||
return []SchedulerLane{LaneDefault, LaneIceberg, LaneLifecycle, LaneLance}
|
||||
}
|
||||
|
||||
// laneIdleSleep maps each lane to its default idle sleep duration.
|
||||
@@ -36,6 +40,7 @@ var laneIdleSleep = map[SchedulerLane]time.Duration{
|
||||
LaneDefault: 61 * time.Second,
|
||||
LaneIceberg: 61 * time.Second,
|
||||
LaneLifecycle: 5 * time.Minute,
|
||||
LaneLance: 61 * time.Second,
|
||||
}
|
||||
|
||||
// laneRequiresLock maps each lane to whether its job types must be
|
||||
@@ -48,6 +53,9 @@ var laneRequiresLock = map[SchedulerLane]bool{
|
||||
LaneDefault: true,
|
||||
LaneIceberg: false,
|
||||
LaneLifecycle: false,
|
||||
// Lance maintenance rewrites files inside one table and shares no global
|
||||
// state, so it has no more need of the cluster admin lock than Iceberg does.
|
||||
LaneLance: false,
|
||||
}
|
||||
|
||||
// LaneRequiresLock returns true if the given lane serialises its job types
|
||||
@@ -83,6 +91,13 @@ var jobTypeLaneMap = map[string]SchedulerLane{
|
||||
|
||||
// S3 lifecycle management
|
||||
"s3_lifecycle": LaneLifecycle,
|
||||
|
||||
// Lance table maintenance. Without these the job types fall back to the
|
||||
// default lane, which serialises everything under the cluster admin lock
|
||||
// and would queue a compaction behind volume balancing.
|
||||
"lance_compact": LaneLance,
|
||||
"lance_optimize_indices": LaneLance,
|
||||
"lance_cleanup_versions": LaneLance,
|
||||
}
|
||||
|
||||
// JobTypeLane returns the scheduler lane for the given job type.
|
||||
|
||||
@@ -5,10 +5,16 @@ import (
|
||||
)
|
||||
|
||||
func TestJobTypeLaneMapCoversKnownTypes(t *testing.T) {
|
||||
// Every job type in the map must resolve to a valid lane.
|
||||
// Every job type in the map must resolve to a lane the scheduler runs.
|
||||
// Checked against AllLanes rather than a list spelled out here, so adding a
|
||||
// lane does not silently leave its job types unscheduled.
|
||||
known := make(map[SchedulerLane]bool, len(AllLanes()))
|
||||
for _, lane := range AllLanes() {
|
||||
known[lane] = true
|
||||
}
|
||||
for jobType, lane := range jobTypeLaneMap {
|
||||
if lane != LaneDefault && lane != LaneIceberg && lane != LaneLifecycle {
|
||||
t.Errorf("jobTypeLaneMap[%q] = %q, want a known lane", jobType, lane)
|
||||
if !known[lane] {
|
||||
t.Errorf("jobTypeLaneMap[%q] = %q, want a lane from AllLanes()", jobType, lane)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -108,7 +108,8 @@ function initS3TablesBuckets() {
|
||||
const tagsInput = document.getElementById('s3tablesBucketTags').value.trim();
|
||||
const tags = parseTagsInput(tagsInput);
|
||||
if (tags === null) return;
|
||||
const payload = { name: name, tags: tags, owner: owner };
|
||||
const formatInput = document.querySelector('#s3tablesBucketFormatPicker input[name="format"]:checked');
|
||||
const payload = { name: name, tags: tags, owner: owner, format: formatInput ? formatInput.value : 'ICEBERG' };
|
||||
|
||||
try {
|
||||
const response = await fetch(s3tBasePath('/api/s3tables/buckets'), {
|
||||
@@ -129,6 +130,51 @@ function initS3TablesBuckets() {
|
||||
});
|
||||
}
|
||||
|
||||
// The endpoint is the whole reason the format matters, so show it changing
|
||||
// rather than making the operator work it out after the fact.
|
||||
const formatPicker = document.getElementById('s3tablesBucketFormatPicker');
|
||||
const formatHint = document.getElementById('s3tablesBucketFormatHint');
|
||||
if (formatPicker && formatHint) {
|
||||
const icebergPort = formatHint.dataset.icebergPort;
|
||||
const lancePort = formatHint.dataset.lancePort;
|
||||
const origin = window.location.protocol + '//' + window.location.hostname;
|
||||
const describeEndpoint = function () {
|
||||
const chosen = formatPicker.querySelector('input[name="format"]:checked');
|
||||
const isLance = chosen && chosen.value === 'LANCE';
|
||||
const port = isLance ? lancePort : icebergPort;
|
||||
const name = (document.getElementById('s3tablesBucketName').value || '').trim();
|
||||
// The bucket name is whatever the operator is typing, so it goes in
|
||||
// as text. Building this with innerHTML would run their input.
|
||||
formatHint.textContent = '';
|
||||
if (!port || port === '0') {
|
||||
formatHint.appendChild(document.createTextNode('A bucket holds one format. Tables of the other are refused. '));
|
||||
const warning = document.createElement('span');
|
||||
warning.className = 'text-warning';
|
||||
warning.textContent = 'No server is running for this format.';
|
||||
formatHint.appendChild(warning);
|
||||
return;
|
||||
}
|
||||
const path = isLance
|
||||
? '/v1/namespace/' + (name || '<bucket>') + '/list'
|
||||
: '/v1/' + (name || '<bucket>') + '/namespaces';
|
||||
formatHint.appendChild(document.createTextNode('Clients reach this bucket at '));
|
||||
const endpoint = document.createElement('code');
|
||||
endpoint.textContent = origin + ':' + port + path;
|
||||
formatHint.appendChild(endpoint);
|
||||
};
|
||||
formatPicker.addEventListener('change', describeEndpoint);
|
||||
const bucketNameField = document.getElementById('s3tablesBucketName');
|
||||
if (bucketNameField) {
|
||||
bucketNameField.addEventListener('input', describeEndpoint);
|
||||
}
|
||||
describeEndpoint();
|
||||
}
|
||||
|
||||
// The banner prints localhost server-side; the browser knows the real host.
|
||||
document.querySelectorAll('.s3tables-origin').forEach(function (el) {
|
||||
el.textContent = window.location.protocol + '//' + window.location.hostname + ':' + el.dataset.port;
|
||||
});
|
||||
|
||||
const policyForm = document.getElementById('s3tablesBucketPolicyForm');
|
||||
if (policyForm) {
|
||||
policyForm.addEventListener('submit', async function (e) {
|
||||
|
||||
Binary file not shown.
@@ -55,105 +55,107 @@ templ IcebergTableData(data dash.IcebergDataPreviewData) {
|
||||
<i class="fas fa-info-circle me-2"></i>{ note }
|
||||
</div>
|
||||
}
|
||||
<div class="row mb-4">
|
||||
<div class="col-xl-4 col-md-6 mb-4">
|
||||
<div class="card border-left-primary shadow h-100 py-2">
|
||||
<div class="card-body">
|
||||
<div class="row no-gutters align-items-center">
|
||||
<div class="col mr-2">
|
||||
<div class="text-xs font-weight-bold text-primary text-uppercase mb-1">
|
||||
Snapshot
|
||||
</div>
|
||||
<div class="h6 mb-0 font-weight-bold text-gray-800">
|
||||
if data.SnapshotID != 0 {
|
||||
{ fmt.Sprintf("%d", data.SnapshotID) }
|
||||
if data.SnapshotID == data.CurrentSnapshotID {
|
||||
<span class="badge bg-success ms-1">current</span>
|
||||
if !previewIsWorkerSourced(data) {
|
||||
<div class="row mb-4">
|
||||
<div class="col-xl-4 col-md-6 mb-4">
|
||||
<div class="card border-left-primary shadow h-100 py-2">
|
||||
<div class="card-body">
|
||||
<div class="row no-gutters align-items-center">
|
||||
<div class="col mr-2">
|
||||
<div class="text-xs font-weight-bold text-primary text-uppercase mb-1">
|
||||
Snapshot
|
||||
</div>
|
||||
<div class="h6 mb-0 font-weight-bold text-gray-800">
|
||||
if data.SnapshotID != 0 {
|
||||
{ fmt.Sprintf("%d", data.SnapshotID) }
|
||||
if data.SnapshotID == data.CurrentSnapshotID {
|
||||
<span class="badge bg-success ms-1">current</span>
|
||||
}
|
||||
} else {
|
||||
-
|
||||
}
|
||||
} else {
|
||||
-
|
||||
</div>
|
||||
if !data.SnapshotTime.IsZero() {
|
||||
<div class="small text-muted">{ data.SnapshotTime.Format("2006-01-02 15:04:05") }</div>
|
||||
}
|
||||
</div>
|
||||
if !data.SnapshotTime.IsZero() {
|
||||
<div class="small text-muted">{ data.SnapshotTime.Format("2006-01-02 15:04:05") }</div>
|
||||
}
|
||||
</div>
|
||||
<div class="col-auto">
|
||||
if len(data.Snapshots) > 1 {
|
||||
<div class="dropdown">
|
||||
<button class="btn btn-sm btn-outline-secondary dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false">
|
||||
Switch
|
||||
</button>
|
||||
<ul class="dropdown-menu dropdown-menu-end">
|
||||
for i, snapshot := range data.Snapshots {
|
||||
if i < 25 {
|
||||
<div class="col-auto">
|
||||
if len(data.Snapshots) > 1 {
|
||||
<div class="dropdown">
|
||||
<button class="btn btn-sm btn-outline-secondary dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false">
|
||||
Switch
|
||||
</button>
|
||||
<ul class="dropdown-menu dropdown-menu-end">
|
||||
for i, snapshot := range data.Snapshots {
|
||||
if i < 25 {
|
||||
<li>
|
||||
<a class="dropdown-item small" href={ dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, snapshot.SnapshotID, data.RowLimit, "")) }>
|
||||
{ fmt.Sprintf("%d", snapshot.SnapshotID) }
|
||||
if !snapshot.Timestamp.IsZero() {
|
||||
<span class="text-muted ms-1">{ snapshot.Timestamp.Format("2006-01-02 15:04") }</span>
|
||||
}
|
||||
</a>
|
||||
</li>
|
||||
}
|
||||
}
|
||||
if len(data.Snapshots) > 25 {
|
||||
<li><hr class="dropdown-divider"/></li>
|
||||
<li>
|
||||
<a class="dropdown-item small" href={ dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, snapshot.SnapshotID, data.RowLimit, "")) }>
|
||||
{ fmt.Sprintf("%d", snapshot.SnapshotID) }
|
||||
if !snapshot.Timestamp.IsZero() {
|
||||
<span class="text-muted ms-1">{ snapshot.Timestamp.Format("2006-01-02 15:04") }</span>
|
||||
}
|
||||
<a class="dropdown-item small text-muted" href={ dash.PUrl(ctx, fmt.Sprintf("/object-store/s3tables/buckets/%s/namespaces/%s/tables/%s", url.PathEscape(data.CatalogName), url.PathEscape(data.NamespaceName), url.PathEscape(data.TableName))) }>
|
||||
{ fmt.Sprintf("%d more in snapshot history…", len(data.Snapshots)-25) }
|
||||
</a>
|
||||
</li>
|
||||
}
|
||||
}
|
||||
if len(data.Snapshots) > 25 {
|
||||
<li><hr class="dropdown-divider"/></li>
|
||||
<li>
|
||||
<a class="dropdown-item small text-muted" href={ dash.PUrl(ctx, fmt.Sprintf("/object-store/s3tables/buckets/%s/namespaces/%s/tables/%s", url.PathEscape(data.CatalogName), url.PathEscape(data.NamespaceName), url.PathEscape(data.TableName))) }>
|
||||
{ fmt.Sprintf("%d more in snapshot history…", len(data.Snapshots)-25) }
|
||||
</a>
|
||||
</li>
|
||||
}
|
||||
</ul>
|
||||
</ul>
|
||||
</div>
|
||||
} else {
|
||||
<i class="fas fa-history fa-2x text-gray-300"></i>
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-xl-4 col-md-6 mb-4">
|
||||
<div class="card border-left-success shadow h-100 py-2">
|
||||
<div class="card-body">
|
||||
<div class="row no-gutters align-items-center">
|
||||
<div class="col mr-2">
|
||||
<div class="text-xs font-weight-bold text-success text-uppercase mb-1">
|
||||
Data Files
|
||||
</div>
|
||||
} else {
|
||||
<i class="fas fa-history fa-2x text-gray-300"></i>
|
||||
}
|
||||
<div class="h5 mb-0 font-weight-bold text-gray-800">
|
||||
{ formatNumber(int64(data.TotalDataFiles)) }
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-auto">
|
||||
<i class="fas fa-copy fa-2x text-gray-300"></i>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-xl-4 col-md-6 mb-4">
|
||||
<div class="card border-left-info shadow h-100 py-2">
|
||||
<div class="card-body">
|
||||
<div class="row no-gutters align-items-center">
|
||||
<div class="col mr-2">
|
||||
<div class="text-xs font-weight-bold text-info text-uppercase mb-1">
|
||||
Total Records
|
||||
</div>
|
||||
<div class="h5 mb-0 font-weight-bold text-gray-800">
|
||||
{ formatNumber(data.TotalRecords) }
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-auto">
|
||||
<i class="fas fa-list-ol fa-2x text-gray-300"></i>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-xl-4 col-md-6 mb-4">
|
||||
<div class="card border-left-success shadow h-100 py-2">
|
||||
<div class="card-body">
|
||||
<div class="row no-gutters align-items-center">
|
||||
<div class="col mr-2">
|
||||
<div class="text-xs font-weight-bold text-success text-uppercase mb-1">
|
||||
Data Files
|
||||
</div>
|
||||
<div class="h5 mb-0 font-weight-bold text-gray-800">
|
||||
{ formatNumber(int64(data.TotalDataFiles)) }
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-auto">
|
||||
<i class="fas fa-copy fa-2x text-gray-300"></i>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-xl-4 col-md-6 mb-4">
|
||||
<div class="card border-left-info shadow h-100 py-2">
|
||||
<div class="card-body">
|
||||
<div class="row no-gutters align-items-center">
|
||||
<div class="col mr-2">
|
||||
<div class="text-xs font-weight-bold text-info text-uppercase mb-1">
|
||||
Total Records
|
||||
</div>
|
||||
<div class="h5 mb-0 font-weight-bold text-gray-800">
|
||||
{ formatNumber(data.TotalRecords) }
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-auto">
|
||||
<i class="fas fa-list-ol fa-2x text-gray-300"></i>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
<div class="row">
|
||||
<div class="col-12 mb-4">
|
||||
<div class="card shadow">
|
||||
@@ -199,9 +201,15 @@ templ IcebergTableData(data dash.IcebergDataPreviewData) {
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="small text-muted">
|
||||
{ fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles) }
|
||||
</div>
|
||||
if previewIsWorkerSourced(data) {
|
||||
<div class="small text-muted">
|
||||
{ fmt.Sprintf("Showing %d row(s), read from the %s dataset by worker %s.", len(data.Rows), data.Format, data.PreviewedBy) }
|
||||
</div>
|
||||
} else {
|
||||
<div class="small text-muted">
|
||||
{ fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles) }
|
||||
</div>
|
||||
}
|
||||
} else {
|
||||
<div class="text-center text-muted py-4">No rows to preview.</div>
|
||||
}
|
||||
@@ -209,50 +217,52 @@ templ IcebergTableData(data dash.IcebergDataPreviewData) {
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="col-12 mb-4">
|
||||
<div class="card shadow">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-copy me-2"></i>Data Files
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="table-responsive">
|
||||
<table class="table table-sm table-hover">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Path</th>
|
||||
<th>Format</th>
|
||||
<th>Records</th>
|
||||
<th>Size</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
for _, file := range data.DataFiles {
|
||||
if !previewIsWorkerSourced(data) {
|
||||
<div class="row">
|
||||
<div class="col-12 mb-4">
|
||||
<div class="card shadow">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-copy me-2"></i>Data Files
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="table-responsive">
|
||||
<table class="table table-sm table-hover">
|
||||
<thead>
|
||||
<tr>
|
||||
<td><code class="small">{ file.Path }</code></td>
|
||||
<td><span class="badge bg-secondary">{ file.Format }</span></td>
|
||||
<td>{ formatNumber(file.RecordCount) }</td>
|
||||
<td>{ formatBytes(file.SizeBytes) }</td>
|
||||
<td>
|
||||
<a class="btn btn-sm btn-outline-primary" href={ dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, data.RowLimit, file.Path)) }>
|
||||
<i class="fas fa-eye me-1"></i>Preview
|
||||
</a>
|
||||
</td>
|
||||
<th>Path</th>
|
||||
<th>Format</th>
|
||||
<th>Records</th>
|
||||
<th>Size</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
}
|
||||
if len(data.DataFiles) == 0 {
|
||||
<tr>
|
||||
<td colspan="5" class="text-center text-muted">No data files in this snapshot.</td>
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
</thead>
|
||||
<tbody>
|
||||
for _, file := range data.DataFiles {
|
||||
<tr>
|
||||
<td><code class="small">{ file.Path }</code></td>
|
||||
<td><span class="badge bg-secondary">{ file.Format }</span></td>
|
||||
<td>{ formatNumber(file.RecordCount) }</td>
|
||||
<td>{ formatBytes(file.SizeBytes) }</td>
|
||||
<td>
|
||||
<a class="btn btn-sm btn-outline-primary" href={ dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, data.RowLimit, file.Path)) }>
|
||||
<i class="fas fa-eye me-1"></i>Preview
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
}
|
||||
if len(data.DataFiles) == 0 {
|
||||
<tr>
|
||||
<td colspan="5" class="text-center text-muted">No data files in this snapshot.</td>
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Code generated by templ - DO NOT EDIT.
|
||||
|
||||
// templ: version: v0.3.1001
|
||||
// templ: version: v0.3.1020
|
||||
package app
|
||||
|
||||
//lint:file-ignore SA4006 This context is only used if a nested component is present.
|
||||
@@ -183,479 +183,515 @@ func IcebergTableData(data dash.IcebergDataPreviewData) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 14, "<div class=\"row mb-4\"><div class=\"col-xl-4 col-md-6 mb-4\"><div class=\"card border-left-primary shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-primary text-uppercase mb-1\">Snapshot</div><div class=\"h6 mb-0 font-weight-bold text-gray-800\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if data.SnapshotID != 0 {
|
||||
var templ_7745c5c3_Var12 string
|
||||
templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.SnapshotID))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 69, Col: 45}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12))
|
||||
if !previewIsWorkerSourced(data) {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 14, "<div class=\"row mb-4\"><div class=\"col-xl-4 col-md-6 mb-4\"><div class=\"card border-left-primary shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-primary text-uppercase mb-1\">Snapshot</div><div class=\"h6 mb-0 font-weight-bold text-gray-800\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 15, " ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if data.SnapshotID == data.CurrentSnapshotID {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "<span class=\"badge bg-success ms-1\">current</span>")
|
||||
if data.SnapshotID != 0 {
|
||||
var templ_7745c5c3_Var12 string
|
||||
templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.SnapshotID))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 70, Col: 46}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 15, " ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if data.SnapshotID == data.CurrentSnapshotID {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "<span class=\"badge bg-success ms-1\">current</span>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "-")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "-")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if !data.SnapshotTime.IsZero() {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "<div class=\"small text-muted\">")
|
||||
if !data.SnapshotTime.IsZero() {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "<div class=\"small text-muted\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var13 string
|
||||
templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(data.SnapshotTime.Format("2006-01-02 15:04:05"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 79, Col: 88}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "</div><div class=\"col-auto\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var13 string
|
||||
templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(data.SnapshotTime.Format("2006-01-02 15:04:05"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 78, Col: 87}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "</div><div class=\"col-auto\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if len(data.Snapshots) > 1 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "<div class=\"dropdown\"><button class=\"btn btn-sm btn-outline-secondary dropdown-toggle\" type=\"button\" data-bs-toggle=\"dropdown\" aria-expanded=\"false\">Switch</button><ul class=\"dropdown-menu dropdown-menu-end\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for i, snapshot := range data.Snapshots {
|
||||
if i < 25 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "<li><a class=\"dropdown-item small\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var14 templ.SafeURL
|
||||
templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, snapshot.SnapshotID, data.RowLimit, "")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 91, Col: 180}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var15 string
|
||||
templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", snapshot.SnapshotID))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 92, Col: 54}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, " ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if !snapshot.Timestamp.IsZero() {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "<span class=\"text-muted ms-1\">")
|
||||
if len(data.Snapshots) > 1 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "<div class=\"dropdown\"><button class=\"btn btn-sm btn-outline-secondary dropdown-toggle\" type=\"button\" data-bs-toggle=\"dropdown\" aria-expanded=\"false\">Switch</button><ul class=\"dropdown-menu dropdown-menu-end\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for i, snapshot := range data.Snapshots {
|
||||
if i < 25 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "<li><a class=\"dropdown-item small\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var16 string
|
||||
templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(snapshot.Timestamp.Format("2006-01-02 15:04"))
|
||||
var templ_7745c5c3_Var14 templ.SafeURL
|
||||
templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, snapshot.SnapshotID, data.RowLimit, "")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 94, Col: 92}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 92, Col: 181}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</span>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var15 string
|
||||
templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", snapshot.SnapshotID))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 93, Col: 55}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, " ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if !snapshot.Timestamp.IsZero() {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "<span class=\"text-muted ms-1\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var16 string
|
||||
templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(snapshot.Timestamp.Format("2006-01-02 15:04"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 95, Col: 93}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</span>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</a></li>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</a></li>")
|
||||
}
|
||||
if len(data.Snapshots) > 25 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "<li><hr class=\"dropdown-divider\"></li><li><a class=\"dropdown-item small text-muted\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var17 templ.SafeURL
|
||||
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, fmt.Sprintf("/object-store/s3tables/buckets/%s/namespaces/%s/tables/%s", url.PathEscape(data.CatalogName), url.PathEscape(data.NamespaceName), url.PathEscape(data.TableName))))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 104, Col: 252}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var18 string
|
||||
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d more in snapshot history…", len(data.Snapshots)-25))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 105, Col: 85}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "</a></li>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(data.Snapshots) > 25 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "<li><hr class=\"dropdown-divider\"></li><li><a class=\"dropdown-item small text-muted\" href=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "</ul></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var17 templ.SafeURL
|
||||
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, fmt.Sprintf("/object-store/s3tables/buckets/%s/namespaces/%s/tables/%s", url.PathEscape(data.CatalogName), url.PathEscape(data.NamespaceName), url.PathEscape(data.TableName))))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 103, Col: 251}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var18 string
|
||||
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d more in snapshot history…", len(data.Snapshots)-25))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 104, Col: 84}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "</a></li>")
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "<i class=\"fas fa-history fa-2x text-gray-300\"></i>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "</ul></div>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "</div></div></div></div></div><div class=\"col-xl-4 col-md-6 mb-4\"><div class=\"card border-left-success shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-success text-uppercase mb-1\">Data Files</div><div class=\"h5 mb-0 font-weight-bold text-gray-800\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "<i class=\"fas fa-history fa-2x text-gray-300\"></i>")
|
||||
var templ_7745c5c3_Var19 string
|
||||
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(int64(data.TotalDataFiles)))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 128, Col: 51}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "</div></div><div class=\"col-auto\"><i class=\"fas fa-copy fa-2x text-gray-300\"></i></div></div></div></div></div><div class=\"col-xl-4 col-md-6 mb-4\"><div class=\"card border-left-info shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-info text-uppercase mb-1\">Total Records</div><div class=\"h5 mb-0 font-weight-bold text-gray-800\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var20 string
|
||||
templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(data.TotalRecords))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 147, Col: 42}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "</div></div><div class=\"col-auto\"><i class=\"fas fa-list-ol fa-2x text-gray-300\"></i></div></div></div></div></div></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "</div></div></div></div></div><div class=\"col-xl-4 col-md-6 mb-4\"><div class=\"card border-left-success shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-success text-uppercase mb-1\">Data Files</div><div class=\"h5 mb-0 font-weight-bold text-gray-800\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var19 string
|
||||
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(int64(data.TotalDataFiles)))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 127, Col: 50}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "</div></div><div class=\"col-auto\"><i class=\"fas fa-copy fa-2x text-gray-300\"></i></div></div></div></div></div><div class=\"col-xl-4 col-md-6 mb-4\"><div class=\"card border-left-info shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-info text-uppercase mb-1\">Total Records</div><div class=\"h5 mb-0 font-weight-bold text-gray-800\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var20 string
|
||||
templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(data.TotalRecords))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 146, Col: 41}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "</div></div><div class=\"col-auto\"><i class=\"fas fa-list-ol fa-2x text-gray-300\"></i></div></div></div></div></div></div><div class=\"row\"><div class=\"col-12 mb-4\"><div class=\"card shadow\"><div class=\"card-header py-3 d-flex justify-content-between align-items-center\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-table me-2\"></i>Sample Rows ")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "<div class=\"row\"><div class=\"col-12 mb-4\"><div class=\"card shadow\"><div class=\"card-header py-3 d-flex justify-content-between align-items-center\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-table me-2\"></i>Sample Rows ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if data.SelectedFile != "" {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "<span class=\"badge bg-secondary ms-2\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "<span class=\"badge bg-secondary ms-2\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var21 string
|
||||
templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinStringErrs(path.Base(data.SelectedFile))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 164, Col: 75}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 166, Col: 75}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var21))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "</span> <a class=\"small ms-2\" href=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "</span> <a class=\"small ms-2\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var22 templ.SafeURL
|
||||
templ_7745c5c3_Var22, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, data.RowLimit, "")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 165, Col: 161}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 167, Col: 161}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var22))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "\">all files</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "\">all files</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "</h6><div class=\"btn-group btn-group-sm\" role=\"group\" aria-label=\"Row limit\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "</h6><div class=\"btn-group btn-group-sm\" role=\"group\" aria-label=\"Row limit\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for _, limit := range []int{25, 50, 100, 200} {
|
||||
if limit == data.RowLimit {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "<a class=\"btn btn-primary\" href=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, "<a class=\"btn btn-primary\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var23 templ.SafeURL
|
||||
templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, limit, data.SelectedFile)))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 171, Col: 174}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 173, Col: 174}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var23))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, "\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var24 string
|
||||
templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", limit))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 171, Col: 203}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 173, Col: 203}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var24))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "<a class=\"btn btn-outline-primary\" href=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "<a class=\"btn btn-outline-primary\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var25 templ.SafeURL
|
||||
templ_7745c5c3_Var25, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, limit, data.SelectedFile)))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 173, Col: 182}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 175, Col: 182}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var25))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var26 string
|
||||
templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", limit))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 173, Col: 211}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 175, Col: 211}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var26))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "</div></div><div class=\"card-body\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "</div></div><div class=\"card-body\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if len(data.Columns) > 0 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "<div class=\"table-responsive\"><table class=\"table table-sm table-hover table-striped\"><thead><tr><th class=\"text-muted\">#</th>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "<div class=\"table-responsive\"><table class=\"table table-sm table-hover table-striped\"><thead><tr><th class=\"text-muted\">#</th>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for _, col := range data.Columns {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "<th>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "<th>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var27 string
|
||||
templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinStringErrs(col)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 186, Col: 20}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 188, Col: 20}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "</th>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "</th>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "</tr></thead> <tbody>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "</tr></thead> <tbody>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for i, row := range data.Rows {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "<tr><td class=\"text-muted\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "<tr><td class=\"text-muted\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var28 string
|
||||
templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", i+1))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 193, Col: 58}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 195, Col: 58}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "</td>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "</td>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for _, cell := range row {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "<td><span class=\"small\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "<td><span class=\"small\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var29 string
|
||||
templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinStringErrs(cell)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 195, Col: 42}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 197, Col: 42}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "</span></td>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "</span></td>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "</tr>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "</tr>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "</tbody></table></div><div class=\"small text-muted\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "</tbody></table></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var30 string
|
||||
templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 203, Col: 98}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
if previewIsWorkerSourced(data) {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "<div class=\"small text-muted\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var30 string
|
||||
templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("Showing %d row(s), read from the %s dataset by worker %s.", len(data.Rows), data.Format, data.PreviewedBy))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 206, Col: 129}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "<div class=\"small text-muted\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var31 string
|
||||
templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("Showing %d row(s) from %d data file(s).", len(data.Rows), data.ScannedFiles))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 210, Col: 99}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "<div class=\"text-center text-muted py-4\">No rows to preview.</div>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "<div class=\"text-center text-muted py-4\">No rows to preview.</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "</div></div></div></div><div class=\"row\"><div class=\"col-12 mb-4\"><div class=\"card shadow\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-copy me-2\"></i>Data Files</h6></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-sm table-hover\"><thead><tr><th>Path</th><th>Format</th><th>Records</th><th>Size</th><th></th></tr></thead> <tbody>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "</div></div></div></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
for _, file := range data.DataFiles {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "<tr><td><code class=\"small\">")
|
||||
if !previewIsWorkerSourced(data) {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "<div class=\"row\"><div class=\"col-12 mb-4\"><div class=\"card shadow\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-copy me-2\"></i>Data Files</h6></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-sm table-hover\"><thead><tr><th>Path</th><th>Format</th><th>Records</th><th>Size</th><th></th></tr></thead> <tbody>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var31 string
|
||||
templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(file.Path)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 235, Col: 45}
|
||||
for _, file := range data.DataFiles {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "<tr><td><code class=\"small\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var32 string
|
||||
templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinStringErrs(file.Path)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 244, Col: 46}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, "</code></td><td><span class=\"badge bg-secondary\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var33 string
|
||||
templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinStringErrs(file.Format)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 245, Col: 61}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "</span></td><td>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var34 string
|
||||
templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(file.RecordCount))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 246, Col: 47}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "</td><td>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var35 string
|
||||
templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(file.SizeBytes))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 247, Col: 44}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "</td><td><a class=\"btn btn-sm btn-outline-primary\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var36 templ.SafeURL
|
||||
templ_7745c5c3_Var36, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, data.RowLimit, file.Path)))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 249, Col: 193}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var36))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 71, "\"><i class=\"fas fa-eye me-1\"></i>Preview</a></td></tr>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
|
||||
if len(data.DataFiles) == 0 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "<tr><td colspan=\"5\" class=\"text-center text-muted\">No data files in this snapshot.</td></tr>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "</tbody></table></div></div></div></div></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "</code></td><td><span class=\"badge bg-secondary\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var32 string
|
||||
templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinStringErrs(file.Format)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 236, Col: 60}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "</span></td><td>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var33 string
|
||||
templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinStringErrs(formatNumber(file.RecordCount))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 237, Col: 46}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "</td><td>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var34 string
|
||||
templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(file.SizeBytes))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 238, Col: 43}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "</td><td><a class=\"btn btn-sm btn-outline-primary\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var35 templ.SafeURL
|
||||
templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, icebergTableDataURL(data.CatalogName, data.NamespaceName, data.TableName, data.SnapshotID, data.RowLimit, file.Path)))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/iceberg_table_data.templ`, Line: 240, Col: 192}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "\"><i class=\"fas fa-eye me-1\"></i>Preview</a></td></tr>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
if len(data.DataFiles) == 0 {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, "<tr><td colspan=\"5\" class=\"text-center text-muted\">No data files in this snapshot.</td></tr>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "</tbody></table></div></div></div></div></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
@@ -3,6 +3,10 @@ package app
|
||||
import (
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// icebergTableDataURL builds the table data-preview page URL. Zero snapshotID
|
||||
@@ -26,3 +30,10 @@ func icebergTableDataURL(catalog, namespace, table string, snapshotID int64, lim
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
// previewIsWorkerSourced reports whether these rows came from a plugin worker
|
||||
// rather than from metadata admin read itself. Snapshots and data files are
|
||||
// Iceberg's shape, and showing them empty for another format reads as a fault.
|
||||
func previewIsWorkerSourced(data dash.IcebergDataPreviewData) bool {
|
||||
return data.Format != "" && !strings.EqualFold(data.Format, s3tables.FormatIceberg)
|
||||
}
|
||||
|
||||
@@ -27,7 +27,12 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) {
|
||||
{ data.NamespaceName }
|
||||
</a>
|
||||
</li>
|
||||
<li class="breadcrumb-item active">{ data.TableName }</li>
|
||||
<li class="breadcrumb-item active">
|
||||
{ data.TableName }
|
||||
if data.Format != "" {
|
||||
<span class={ formatBadgeClass(data.Format), "ms-2 align-middle" } title={ formatBadgeTitle(data.Format) }>{ formatLabel(data.Format) }</span>
|
||||
}
|
||||
</li>
|
||||
</ol>
|
||||
</nav>
|
||||
</h1>
|
||||
@@ -223,6 +228,11 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) {
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-list me-2"></i>Schema
|
||||
if data.ObservedBy != "" {
|
||||
<span class="badge bg-secondary ms-2" title={ "reported by " + data.ObservedBy }>
|
||||
as seen { data.ObservedAt.Format("2006-01-02 15:04") }
|
||||
</span>
|
||||
}
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
@@ -253,7 +263,13 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) {
|
||||
}
|
||||
if len(data.SchemaFields) == 0 {
|
||||
<tr>
|
||||
<td colspan="4" class="text-center text-muted">No schema available.</td>
|
||||
if data.Format != "" && data.Format != "ICEBERG" {
|
||||
<td colspan="4" class="text-center text-muted">
|
||||
The catalog records where this { data.Format } table lives, not what is in it. Run a { data.Format } plugin worker and its schema appears here.
|
||||
</td>
|
||||
} else {
|
||||
<td colspan="4" class="text-center text-muted">No schema available.</td>
|
||||
}
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
@@ -263,104 +279,143 @@ templ IcebergTableDetails(data dash.IcebergTableDetailsData) {
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="col-12 mb-4">
|
||||
<div class="card shadow">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-layer-group me-2"></i>Partitions
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="table-responsive">
|
||||
<table class="table table-hover">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Transform</th>
|
||||
<th>Source ID</th>
|
||||
<th>Field ID</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
for _, field := range data.PartitionFields {
|
||||
if !isLanceFormat(data.Format) {
|
||||
<div class="row">
|
||||
<div class="col-12 mb-4">
|
||||
<div class="card shadow">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-layer-group me-2"></i>Partitions
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="table-responsive">
|
||||
<table class="table table-hover">
|
||||
<thead>
|
||||
<tr>
|
||||
<td>{ field.Name }</td>
|
||||
<td>{ field.Transform }</td>
|
||||
<td>{ fmt.Sprintf("%d", field.SourceID) }</td>
|
||||
<td>{ fmt.Sprintf("%d", field.FieldID) }</td>
|
||||
<th>Name</th>
|
||||
<th>Transform</th>
|
||||
<th>Source ID</th>
|
||||
<th>Field ID</th>
|
||||
</tr>
|
||||
}
|
||||
if len(data.PartitionFields) == 0 {
|
||||
<tr>
|
||||
<td colspan="4" class="text-center text-muted">No partitions defined.</td>
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
</thead>
|
||||
<tbody>
|
||||
for _, field := range data.PartitionFields {
|
||||
<tr>
|
||||
<td>{ field.Name }</td>
|
||||
<td>{ field.Transform }</td>
|
||||
<td>{ fmt.Sprintf("%d", field.SourceID) }</td>
|
||||
<td>{ fmt.Sprintf("%d", field.FieldID) }</td>
|
||||
</tr>
|
||||
}
|
||||
if len(data.PartitionFields) == 0 {
|
||||
<tr>
|
||||
<td colspan="4" class="text-center text-muted">No partitions defined.</td>
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
<div class="card shadow mb-4">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-history me-2"></i>Snapshot History
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="table-responsive">
|
||||
<table class="table table-hover">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Snapshot ID</th>
|
||||
<th>Timestamp</th>
|
||||
<th>Operation</th>
|
||||
<th>Manifest List</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
for _, snapshot := range data.Snapshots {
|
||||
}
|
||||
if !isLanceFormat(data.Format) {
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
<div class="card shadow mb-4">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-history me-2"></i>Snapshot History
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="table-responsive">
|
||||
<table class="table table-hover">
|
||||
<thead>
|
||||
<tr>
|
||||
<td>{ fmt.Sprintf("%d", snapshot.SnapshotID) }</td>
|
||||
<td>
|
||||
if snapshot.Timestamp.IsZero() {
|
||||
<span class="text-muted">-</span>
|
||||
} else {
|
||||
{ snapshot.Timestamp.Format("2006-01-02 15:04") }
|
||||
}
|
||||
</td>
|
||||
<td>
|
||||
if snapshot.Operation != "" {
|
||||
{ snapshot.Operation }
|
||||
} else {
|
||||
<span class="text-muted">-</span>
|
||||
}
|
||||
</td>
|
||||
<td>
|
||||
if snapshot.ManifestList != "" {
|
||||
<code class="small">{ snapshot.ManifestList }</code>
|
||||
} else {
|
||||
<span class="text-muted">-</span>
|
||||
}
|
||||
</td>
|
||||
<th>Snapshot ID</th>
|
||||
<th>Timestamp</th>
|
||||
<th>Operation</th>
|
||||
<th>Manifest List</th>
|
||||
</tr>
|
||||
}
|
||||
if len(data.Snapshots) == 0 {
|
||||
<tr>
|
||||
<td colspan="4" class="text-center text-muted">No snapshots available.</td>
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
</thead>
|
||||
<tbody>
|
||||
for _, snapshot := range data.Snapshots {
|
||||
<tr>
|
||||
<td>{ fmt.Sprintf("%d", snapshot.SnapshotID) }</td>
|
||||
<td>
|
||||
if snapshot.Timestamp.IsZero() {
|
||||
<span class="text-muted">-</span>
|
||||
} else {
|
||||
{ snapshot.Timestamp.Format("2006-01-02 15:04") }
|
||||
}
|
||||
</td>
|
||||
<td>
|
||||
if snapshot.Operation != "" {
|
||||
{ snapshot.Operation }
|
||||
} else {
|
||||
<span class="text-muted">-</span>
|
||||
}
|
||||
</td>
|
||||
<td>
|
||||
if snapshot.ManifestList != "" {
|
||||
<code class="small">{ snapshot.ManifestList }</code>
|
||||
} else {
|
||||
<span class="text-muted">-</span>
|
||||
}
|
||||
</td>
|
||||
</tr>
|
||||
}
|
||||
if len(data.Snapshots) == 0 {
|
||||
<tr>
|
||||
if data.Format != "" && data.Format != "ICEBERG" {
|
||||
<td colspan="4" class="text-center text-muted">
|
||||
A { data.Format } table keeps its own version history; the catalog does not mirror it here.
|
||||
</td>
|
||||
} else {
|
||||
<td colspan="4" class="text-center text-muted">No snapshots available.</td>
|
||||
}
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
if isLanceFormat(data.Format) {
|
||||
<div class="row">
|
||||
<div class="col-12 mb-4">
|
||||
<div class="card shadow">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-history me-2"></i>Versions
|
||||
if data.ObservedBy != "" {
|
||||
<span class="badge bg-secondary ms-2" title={ "reported by " + data.ObservedBy }>
|
||||
as seen { data.ObservedAt.Format("2006-01-02 15:04") }
|
||||
</span>
|
||||
}
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
if len(data.Properties) > 0 {
|
||||
<p class="text-muted mb-0">
|
||||
A { data.Format } dataset keeps its own version history. What a worker last saw is listed under Properties above.
|
||||
</p>
|
||||
} else {
|
||||
<p class="text-muted mb-0">
|
||||
No worker has reported on this table yet, so its version history is unknown here. It lives in the dataset either way.
|
||||
</p>
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
<div class="modal fade" id="deleteIcebergTableModal" tabindex="-1" aria-labelledby="deleteIcebergTableModalLabel" aria-hidden="true">
|
||||
<div class="modal-dialog">
|
||||
<div class="modal-content">
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -26,40 +26,46 @@ templ S3TablesBuckets(data dash.S3TablesBucketsData) {
|
||||
</a>
|
||||
</div>
|
||||
}
|
||||
if data.LancePort > 0 {
|
||||
<div class="btn-group me-2">
|
||||
<a id="s3tables-lance-rest-link" href={ templ.SafeURL(fmt.Sprintf("//localhost:%d/v1/table", data.LancePort)) } target="_blank" class="btn btn-sm btn-outline-secondary">
|
||||
<i class="fas fa-vector-square me-1"></i>Lance Namespace API
|
||||
</a>
|
||||
</div>
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
<div id="s3tables-buckets-content" data-iceberg-port={ fmt.Sprintf("%d", data.IcebergPort) }>
|
||||
if data.IcebergPort > 0 {
|
||||
<div id="s3tables-buckets-content" data-iceberg-port={ fmt.Sprintf("%d", data.IcebergPort) } data-lance-port={ fmt.Sprintf("%d", data.LancePort) }>
|
||||
if data.IcebergPort > 0 || data.LancePort > 0 {
|
||||
<div class="alert alert-info mb-4">
|
||||
<div class="d-flex align-items-center">
|
||||
<div class="d-flex align-items-start">
|
||||
<i class="fas fa-info-circle fa-2x me-3"></i>
|
||||
<div>
|
||||
<strong>Iceberg REST Catalog</strong>
|
||||
<p id="s3tables-iceberg-info" class="mb-0 mt-1" data-iceberg-port={ fmt.Sprintf("%d", data.IcebergPort) }>
|
||||
Each table bucket is an Iceberg catalog. Connect clients to:
|
||||
<code><span id="s3tables-iceberg-protocol">http</span>://<span id="s3tables-iceberg-host">localhost</span>:{ fmt.Sprintf("%d", data.IcebergPort) }/v1</code>
|
||||
<strong>Catalog endpoints</strong>
|
||||
<p class="mb-0 mt-1">
|
||||
Each table bucket is a catalog. Which endpoint serves it depends on the format it holds.
|
||||
</p>
|
||||
<script>
|
||||
const s3tablesInfo = document.getElementById('s3tables-iceberg-info');
|
||||
const s3tablesIcebergHost = window.location.hostname;
|
||||
const s3tablesIcebergProtocol = window.location.protocol.slice(0, -1);
|
||||
const s3tablesIcebergPort = s3tablesInfo ? s3tablesInfo.dataset.icebergPort : '';
|
||||
document.getElementById('s3tables-iceberg-host').innerText = s3tablesIcebergHost;
|
||||
const s3tablesIcebergProtocolEl = document.getElementById('s3tables-iceberg-protocol');
|
||||
if (s3tablesIcebergProtocolEl) {
|
||||
s3tablesIcebergProtocolEl.innerText = s3tablesIcebergProtocol;
|
||||
<div class="mt-2">
|
||||
if data.IcebergPort > 0 {
|
||||
<div>
|
||||
<span class="badge bg-primary">ICEBERG</span>
|
||||
<code class="ms-1"><span class="s3tables-origin" data-port={ fmt.Sprintf("%d", data.IcebergPort) }>{ fmt.Sprintf("http://localhost:%d", data.IcebergPort) }</span>/v1</code>
|
||||
</div>
|
||||
}
|
||||
const s3tablesRestLink = document.getElementById('s3tables-iceberg-rest-link');
|
||||
if (s3tablesRestLink && s3tablesIcebergPort) {
|
||||
s3tablesRestLink.href = `//${s3tablesIcebergHost}:${s3tablesIcebergPort}/v1/config`;
|
||||
if data.LancePort > 0 {
|
||||
<div class="mt-1">
|
||||
<span class="badge bg-success">LANCE</span>
|
||||
<code class="ms-1"><span class="s3tables-origin" data-port={ fmt.Sprintf("%d", data.LancePort) }>{ fmt.Sprintf("http://localhost:%d", data.LancePort) }</span></code>
|
||||
</div>
|
||||
}
|
||||
</script>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
} else {
|
||||
}
|
||||
if data.IcebergPort == 0 && data.LancePort == 0 {
|
||||
<div class="alert alert-warning mb-4">
|
||||
<i class="fas fa-exclamation-triangle me-2"></i>Iceberg REST endpoint is disabled. Start `weed s3` with `-s3.port.iceberg` to enable it.
|
||||
<i class="fas fa-exclamation-triangle me-2"></i>No catalog endpoint is running. Start `weed s3` with `-s3.port.iceberg` or `-s3.port.lance`, and point admin at it with `-iceberg.port` or `-lance.port`.
|
||||
</div>
|
||||
}
|
||||
<div class="row mb-4">
|
||||
@@ -139,9 +145,10 @@ templ S3TablesBuckets(data dash.S3TablesBucketsData) {
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Format</th>
|
||||
<th>Owner</th>
|
||||
<th>ARN</th>
|
||||
<th>Iceberg Endpoint</th>
|
||||
<th>Catalog Endpoint</th>
|
||||
<th>Created</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
@@ -150,9 +157,12 @@ templ S3TablesBuckets(data dash.S3TablesBucketsData) {
|
||||
for _, bucket := range data.Buckets {
|
||||
<tr>
|
||||
<td>{ bucket.Name }</td>
|
||||
<td>
|
||||
<span class={ formatBadgeClass(bucket.Format) } title={ formatBadgeTitle(bucket.Format) }>{ formatLabel(bucket.Format) }</span>
|
||||
</td>
|
||||
<td>{ bucket.OwnerAccountID }</td>
|
||||
<td class="text-muted small">{ bucket.ARN }</td>
|
||||
<td><code class="small">/v1/{ bucket.Name }/namespaces</code></td>
|
||||
<td><code class="small">{ bucketCatalogPath(bucket.Format, bucket.Name) }</code></td>
|
||||
<td>{ bucket.CreatedAt.Format("2006-01-02 15:04") }</td>
|
||||
<td>
|
||||
<div class="btn-group btn-group-sm" role="group">
|
||||
@@ -181,7 +191,7 @@ templ S3TablesBuckets(data dash.S3TablesBucketsData) {
|
||||
}
|
||||
if len(data.Buckets) == 0 {
|
||||
<tr>
|
||||
<td colspan="6" class="text-center text-muted py-4">
|
||||
<td colspan="7" class="text-center text-muted py-4">
|
||||
<i class="fas fa-table fa-3x mb-3 text-muted"></i>
|
||||
<div>
|
||||
<h5>No table buckets found</h5>
|
||||
@@ -200,17 +210,18 @@ templ S3TablesBuckets(data dash.S3TablesBucketsData) {
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
if data.IcebergPort > 0 {
|
||||
if data.IcebergPort > 0 || data.LancePort > 0 {
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
<div class="card shadow mb-4">
|
||||
<div class="card-header py-3">
|
||||
<h6 class="m-0 font-weight-bold text-primary">
|
||||
<i class="fas fa-code me-2"></i>Iceberg Client Examples
|
||||
<i class="fas fa-code me-2"></i>Client Examples
|
||||
</h6>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<h6>DuckDB</h6>
|
||||
if data.IcebergPort > 0 {
|
||||
<h6><span class="badge bg-primary me-2">ICEBERG</span>DuckDB</h6>
|
||||
<pre class="bg-light p-3 border rounded">
|
||||
<code id="s3tables-duckdb-example">
|
||||
{ `INSTALL iceberg;
|
||||
@@ -225,7 +236,7 @@ CREATE SECRET (
|
||||
SELECT * FROM iceberg_scan('s3://my-table-bucket/my-namespace/my-table');` }
|
||||
</code>
|
||||
</pre>
|
||||
<h6 class="mt-4">Python (PyIceberg)</h6>
|
||||
<h6 class="mt-4"><span class="badge bg-primary me-2">ICEBERG</span>Python (PyIceberg)</h6>
|
||||
<pre class="bg-light p-3 border rounded">
|
||||
<code id="s3tables-pyiceberg-example">
|
||||
{ `from pyiceberg.catalog import load_catalog
|
||||
@@ -242,11 +253,40 @@ catalog = load_catalog(
|
||||
namespaces = catalog.list_namespaces()` }
|
||||
</code>
|
||||
</pre>
|
||||
}
|
||||
if data.LancePort > 0 {
|
||||
<h6 class={ templ.KV("mt-4", data.IcebergPort > 0) }><span class="badge bg-success me-2">LANCE</span>Python (lance-namespace)</h6>
|
||||
<pre class="bg-light p-3 border rounded">
|
||||
<code id="s3tables-lance-namespace-example">
|
||||
{ `from lance_namespace import connect
|
||||
|
||||
ns = connect("rest", {"uri": "http://localhost:` + fmt.Sprintf("%d", data.LancePort) + `"})
|
||||
|
||||
ns.list_namespaces(id=["my-table-bucket"])
|
||||
ns.describe_table(id=["my-table-bucket", "my-namespace", "my-table"])` }
|
||||
</code>
|
||||
</pre>
|
||||
<h6 class="mt-4"><span class="badge bg-success me-2">LANCE</span>Python (pylance)</h6>
|
||||
<pre class="bg-light p-3 border rounded">
|
||||
<code id="s3tables-pylance-example">
|
||||
{ `import lance
|
||||
from lance_namespace import connect
|
||||
|
||||
ns = connect("rest", {"uri": "http://localhost:` + fmt.Sprintf("%d", data.LancePort) + `"})
|
||||
table = ns.describe_table(id=["my-table-bucket", "my-namespace", "my-table"])
|
||||
|
||||
# The namespace vends both the location and the credentials to read it.
|
||||
dataset = lance.dataset(table.location, storage_options=table.storage_options)` }
|
||||
</code>
|
||||
</pre>
|
||||
}
|
||||
<script>
|
||||
const s3tablesExamplesHost = window.location.hostname;
|
||||
const s3tablesExamples = [
|
||||
document.getElementById('s3tables-duckdb-example'),
|
||||
document.getElementById('s3tables-pyiceberg-example')
|
||||
document.getElementById('s3tables-pyiceberg-example'),
|
||||
document.getElementById('s3tables-lance-namespace-example'),
|
||||
document.getElementById('s3tables-pylance-example')
|
||||
];
|
||||
s3tablesExamples.forEach(example => {
|
||||
if (!example) {
|
||||
@@ -276,6 +316,39 @@ namespaces = catalog.list_namespaces()` }
|
||||
<label for="s3tablesBucketName" class="form-label">Bucket Name</label>
|
||||
<input type="text" class="form-control" id="s3tablesBucketName" name="name" placeholder="table-bucket-name" required/>
|
||||
</div>
|
||||
<div class="mb-3">
|
||||
<label class="form-label">Table Format</label>
|
||||
<div class="row g-2" id="s3tablesBucketFormatPicker">
|
||||
<div class="col-6">
|
||||
if data.IcebergPort > 0 {
|
||||
<input type="radio" class="btn-check" name="format" id="s3tablesBucketFormatIceberg" value="ICEBERG" checked/>
|
||||
} else {
|
||||
<input type="radio" class="btn-check" name="format" id="s3tablesBucketFormatIceberg" value="ICEBERG" disabled/>
|
||||
}
|
||||
<label class="btn btn-outline-primary w-100 text-start p-3" for="s3tablesBucketFormatIceberg">
|
||||
<span class="d-block fw-semibold"><i class="fas fa-snowflake me-1"></i>Iceberg</span>
|
||||
<span class="d-block small text-muted mt-1">Served over the Iceberg REST catalog. Spark, Trino, PyIceberg, DuckDB.</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="col-6">
|
||||
if data.LancePort > 0 && data.IcebergPort <= 0 {
|
||||
// The only format this cluster serves, so it is the one preselected.
|
||||
<input type="radio" class="btn-check" name="format" id="s3tablesBucketFormatLance" value="LANCE" checked/>
|
||||
} else if data.LancePort > 0 {
|
||||
<input type="radio" class="btn-check" name="format" id="s3tablesBucketFormatLance" value="LANCE"/>
|
||||
} else {
|
||||
<input type="radio" class="btn-check" name="format" id="s3tablesBucketFormatLance" value="LANCE" disabled/>
|
||||
}
|
||||
<label class="btn btn-outline-success w-100 text-start p-3" for="s3tablesBucketFormatLance">
|
||||
<span class="d-block fw-semibold"><i class="fas fa-vector-square me-1"></i>Lance</span>
|
||||
<span class="d-block small text-muted mt-1">Served over the Lance Namespace API. pylance, LanceDB, Ray.</span>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-text" id="s3tablesBucketFormatHint" data-iceberg-port={ fmt.Sprintf("%d", data.IcebergPort) } data-lance-port={ fmt.Sprintf("%d", data.LancePort) }>
|
||||
A bucket holds one format. Tables of the other are refused.
|
||||
</div>
|
||||
</div>
|
||||
<div class="mb-3">
|
||||
<label for="s3tablesBucketOwner" class="form-label">Owner (Optional)</label>
|
||||
<select class="form-select" id="s3tablesBucketOwner" name="owner">
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -12,6 +12,7 @@ templ S3TablesNamespaces(data dash.S3TablesNamespacesData) {
|
||||
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
|
||||
<h1 class="h2">
|
||||
<i class="fas fa-layer-group me-2"></i>S3 Tables Namespaces
|
||||
<span class={ formatBadgeClass(data.BucketFormat), "ms-2 align-middle" } title={ formatBadgeTitle(data.BucketFormat) }>{ formatLabel(data.BucketFormat) }</span>
|
||||
</h1>
|
||||
<div class="btn-toolbar mb-2 mb-md-0">
|
||||
<div class="btn-group me-2">
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -12,6 +12,7 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
|
||||
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
|
||||
<h1 class="h2">
|
||||
<i class="fas fa-table me-2"></i>S3 Tables
|
||||
<span class={ formatBadgeClass(data.BucketFormat), "ms-2 align-middle" } title={ formatBadgeTitle(data.BucketFormat) }>{ formatLabel(data.BucketFormat) }</span>
|
||||
</h1>
|
||||
<div class="btn-toolbar mb-2 mb-md-0">
|
||||
<div class="btn-group me-2">
|
||||
@@ -93,6 +94,8 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Format</th>
|
||||
<th>Rows</th>
|
||||
<th>Table ARN</th>
|
||||
<th>Created</th>
|
||||
<th>Modified</th>
|
||||
@@ -106,6 +109,16 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
|
||||
<tr>
|
||||
{{ tableName := table.Name }}
|
||||
<td>{ tableName }</td>
|
||||
<td>
|
||||
<span class={ formatBadgeClass(table.Format) } title={ formatBadgeTitle(table.Format) }>{ formatLabel(table.Format) }</span>
|
||||
</td>
|
||||
<td>
|
||||
if rows, ok := data.ObservedRows[tableName]; ok {
|
||||
<span title="Reported by a plugin worker">{ rows }</span>
|
||||
} else {
|
||||
<span class="text-muted">—</span>
|
||||
}
|
||||
</td>
|
||||
<td class="text-muted small">{ table.TableARN }</td>
|
||||
<td>{ table.CreatedAt.Format("2006-01-02 15:04") }</td>
|
||||
<td>{ table.ModifiedAt.Format("2006-01-02 15:04") }</td>
|
||||
@@ -126,7 +139,7 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
|
||||
<td>
|
||||
<div class="btn-group btn-group-sm" role="group">
|
||||
if parseErr == nil {
|
||||
<a class="btn btn-outline-primary btn-sm" href={ dash.PUrl(ctx, fmt.Sprintf("/object-store/s3tables/buckets/%s/namespaces/%s/tables/%s", url.PathEscape(bucketName), url.PathEscape(data.Namespace), url.PathEscape(tableName))) } title="View Iceberg Details">
|
||||
<a class="btn btn-outline-primary btn-sm" href={ dash.PUrl(ctx, fmt.Sprintf("/object-store/s3tables/buckets/%s/namespaces/%s/tables/%s", url.PathEscape(bucketName), url.PathEscape(data.Namespace), url.PathEscape(tableName))) } title="View Table Details">
|
||||
<i class="fas fa-eye"></i>
|
||||
</a>
|
||||
} else {
|
||||
@@ -149,7 +162,7 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
|
||||
}
|
||||
if len(data.Tables) == 0 {
|
||||
<tr>
|
||||
<td colspan="7" class="text-center text-muted py-4">
|
||||
<td colspan="9" class="text-center text-muted py-4">
|
||||
<i class="fas fa-table fa-3x mb-3 text-muted"></i>
|
||||
<div>
|
||||
<h5>No tables found</h5>
|
||||
@@ -186,9 +199,16 @@ templ S3TablesTables(data dash.S3TablesTablesData) {
|
||||
</div>
|
||||
<div class="mb-3">
|
||||
<label for="s3tablesTableFormat" class="form-label">Format</label>
|
||||
<select class="form-select" id="s3tablesTableFormat" name="format">
|
||||
<option value="ICEBERG" selected>ICEBERG</option>
|
||||
</select>
|
||||
if data.BucketFormat != "" {
|
||||
<input type="text" class="form-control-plaintext ps-2 border rounded bg-light" id="s3tablesTableFormat" name="format" value={ formatLabel(data.BucketFormat) } readonly/>
|
||||
<div class="form-text">Set by the bucket, which holds one format.</div>
|
||||
} else {
|
||||
<select class="form-select" id="s3tablesTableFormat" name="format">
|
||||
<option value="ICEBERG" selected>ICEBERG</option>
|
||||
<option value="LANCE">LANCE</option>
|
||||
</select>
|
||||
<div class="form-text">This bucket was created before formats were declared, so either is allowed.</div>
|
||||
}
|
||||
</div>
|
||||
<div class="mb-3">
|
||||
<label for="s3tablesTableMetadata" class="form-label">Metadata JSON (optional)</label>
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,63 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// Shared rendering for the table format a bucket or table carries. A bucket is a
|
||||
// catalog and a catalog serves one protocol, so the format decides which
|
||||
// endpoint a client uses and which panels on a page mean anything.
|
||||
|
||||
// formatLabel is what the badge says. A bucket created before formats were
|
||||
// declared has none; that is a fact about its age, not a fault, so it says so
|
||||
// plainly rather than claiming a format it never chose.
|
||||
func formatLabel(format string) string {
|
||||
if strings.TrimSpace(format) == "" {
|
||||
return "unset"
|
||||
}
|
||||
return strings.ToUpper(format)
|
||||
}
|
||||
|
||||
// formatBadgeClass keeps the two formats distinguishable at a glance without
|
||||
// reaching for a colour that means something else in this UI.
|
||||
func formatBadgeClass(format string) string {
|
||||
switch strings.ToUpper(strings.TrimSpace(format)) {
|
||||
case s3tables.FormatIceberg:
|
||||
return "badge bg-primary"
|
||||
case s3tables.FormatLance:
|
||||
return "badge bg-success"
|
||||
default:
|
||||
return "badge bg-light text-muted border"
|
||||
}
|
||||
}
|
||||
|
||||
// formatBadgeTitle explains the badge on hover, which is the only place there is
|
||||
// room to say what an undeclared bucket does.
|
||||
func formatBadgeTitle(format string) string {
|
||||
switch strings.ToUpper(strings.TrimSpace(format)) {
|
||||
case s3tables.FormatIceberg:
|
||||
return "Served over the Iceberg REST catalog"
|
||||
case s3tables.FormatLance:
|
||||
return "Served over the Lance Namespace API"
|
||||
default:
|
||||
return "Created before formats were declared; accepts tables of either format"
|
||||
}
|
||||
}
|
||||
|
||||
// isLanceFormat reports whether this is a format the cluster records but cannot
|
||||
// read, which is what decides between the Iceberg panels and the worker's.
|
||||
func isLanceFormat(format string) bool {
|
||||
return strings.EqualFold(strings.TrimSpace(format), s3tables.FormatLance)
|
||||
}
|
||||
|
||||
// bucketCatalogPath is the path a client uses to reach one bucket, which differs
|
||||
// per format because the two are different protocols on different ports.
|
||||
func bucketCatalogPath(format, bucket string) string {
|
||||
if isLanceFormat(format) {
|
||||
return fmt.Sprintf("/v1/namespace/%s/list", bucket)
|
||||
}
|
||||
return fmt.Sprintf("/v1/%s/namespaces", bucket)
|
||||
}
|
||||
@@ -44,6 +44,7 @@ templ Layout(view ViewContext, content templ.Component) {
|
||||
strings.HasPrefix(currentPath, "/plugin/lanes/default/")
|
||||
isIcebergWorkerPage := currentPath == "/plugin/lanes/iceberg" || strings.HasPrefix(currentPath, "/plugin/lanes/iceberg/")
|
||||
isLifecycleWorkerPage := currentPath == "/plugin/lanes/lifecycle" || strings.HasPrefix(currentPath, "/plugin/lanes/lifecycle/")
|
||||
isLanceWorkerPage := currentPath == "/plugin/lanes/lance" || strings.HasPrefix(currentPath, "/plugin/lanes/lance/")
|
||||
}}
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
@@ -294,6 +295,17 @@ templ Layout(view ViewContext, content templ.Component) {
|
||||
</a>
|
||||
}
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
if isLifecycleWorkerPage {
|
||||
<a class="nav-link active" href={ view.P("/plugin/lanes/lifecycle") }>
|
||||
<i class="fas fa-hourglass-half me-2"></i>Lifecycle
|
||||
</a>
|
||||
} else {
|
||||
<a class="nav-link" href={ view.P("/plugin/lanes/lifecycle") }>
|
||||
<i class="fas fa-hourglass-half me-2"></i>Lifecycle
|
||||
</a>
|
||||
}
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
if isIcebergWorkerPage {
|
||||
<a class="nav-link active" href={ view.P("/plugin/lanes/iceberg") }>
|
||||
@@ -306,13 +318,13 @@ templ Layout(view ViewContext, content templ.Component) {
|
||||
}
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
if isLifecycleWorkerPage {
|
||||
<a class="nav-link active" href={ view.P("/plugin/lanes/lifecycle") }>
|
||||
<i class="fas fa-hourglass-half me-2"></i>Lifecycle
|
||||
if isLanceWorkerPage {
|
||||
<a class="nav-link active" href={ view.P("/plugin/lanes/lance") }>
|
||||
<i class="fas fa-vector-square me-2"></i>Lance
|
||||
</a>
|
||||
} else {
|
||||
<a class="nav-link" href={ view.P("/plugin/lanes/lifecycle") }>
|
||||
<i class="fas fa-hourglass-half me-2"></i>Lifecycle
|
||||
<a class="nav-link" href={ view.P("/plugin/lanes/lance") }>
|
||||
<i class="fas fa-vector-square me-2"></i>Lance
|
||||
</a>
|
||||
}
|
||||
</li>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Code generated by templ - DO NOT EDIT.
|
||||
|
||||
// templ: version: v0.3.1001
|
||||
// templ: version: v0.3.1020
|
||||
package layout
|
||||
|
||||
//lint:file-ignore SA4006 This context is only used if a nested component is present.
|
||||
@@ -77,16 +77,17 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
strings.HasPrefix(currentPath, "/plugin/lanes/default/")
|
||||
isIcebergWorkerPage := currentPath == "/plugin/lanes/iceberg" || strings.HasPrefix(currentPath, "/plugin/lanes/iceberg/")
|
||||
isLifecycleWorkerPage := currentPath == "/plugin/lanes/lifecycle" || strings.HasPrefix(currentPath, "/plugin/lanes/lifecycle/")
|
||||
isLanceWorkerPage := currentPath == "/plugin/lanes/lance" || strings.HasPrefix(currentPath, "/plugin/lanes/lance/")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "<!doctype html><html lang=\"en\"><head><meta charset=\"UTF-8\"><title>SeaweedFS Admin</title><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><meta name=\"csrf-token\" content=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var2 string
|
||||
templ_7745c5c3_Var2, templ_7745c5c3_Err = templ.JoinStringErrs(csrfToken)
|
||||
templ_7745c5c3_Var2, templ_7745c5c3_Err = templ.ResolveAttributeValue(csrfToken)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 54, Col: 47}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 55, Col: 47}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var2))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var2)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -97,7 +98,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var3 templ.SafeURL
|
||||
templ_7745c5c3_Var3, templ_7745c5c3_Err = templ.JoinURLErrs(string(view.P("/static/favicon.ico")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 55, Col: 65}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 56, Col: 65}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var3))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -110,7 +111,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var4 templ.SafeURL
|
||||
templ_7745c5c3_Var4, templ_7745c5c3_Err = templ.JoinURLErrs(string(view.P("/static/css/bootstrap.min.css")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 58, Col: 64}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 59, Col: 64}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var4))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -123,7 +124,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var5 templ.SafeURL
|
||||
templ_7745c5c3_Var5, templ_7745c5c3_Err = templ.JoinURLErrs(string(view.P("/static/css/fontawesome.min.css")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 60, Col: 66}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 61, Col: 66}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var5))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -134,11 +135,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var6 string
|
||||
templ_7745c5c3_Var6, templ_7745c5c3_Err = templ.JoinStringErrs(string(view.P("/static/js/htmx.min.js")))
|
||||
templ_7745c5c3_Var6, templ_7745c5c3_Err = templ.ResolveAttributeValue(string(view.P("/static/js/htmx.min.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 62, Col: 58}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 63, Col: 58}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var6))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var6)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -149,7 +150,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var7 templ.SafeURL
|
||||
templ_7745c5c3_Var7, templ_7745c5c3_Err = templ.JoinURLErrs(string(view.P("/static/css/admin.css")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 64, Col: 73}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 65, Col: 73}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var7))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -170,7 +171,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var8 templ.SafeURL
|
||||
templ_7745c5c3_Var8, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/admin"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 73, Col: 71}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 74, Col: 71}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var8))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -183,7 +184,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var9 string
|
||||
templ_7745c5c3_Var9, templ_7745c5c3_Err = templ.JoinStringErrs(username)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 89, Col: 73}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 90, Col: 73}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var9))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -196,7 +197,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var10 templ.SafeURL
|
||||
templ_7745c5c3_Var10, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/logout"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 92, Col: 85}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 93, Col: 85}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var10))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -209,7 +210,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var11 templ.SafeURL
|
||||
templ_7745c5c3_Var11, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/admin"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 111, Col: 71}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 112, Col: 71}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var11))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -229,11 +230,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var13 string
|
||||
templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(templ.CSSClasses(templ_7745c5c3_Var12).String())
|
||||
templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.ResolveAttributeValue(templ.CSSClasses(templ_7745c5c3_Var12).String())
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 1, Col: 0}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var13)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -242,11 +243,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var14 string
|
||||
templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%t", isClusterPage))
|
||||
templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.ResolveAttributeValue(fmt.Sprintf("%t", isClusterPage))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 116, Col: 207}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 117, Col: 207}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var14)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -264,11 +265,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var16 string
|
||||
templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(templ.CSSClasses(templ_7745c5c3_Var15).String())
|
||||
templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.ResolveAttributeValue(templ.CSSClasses(templ_7745c5c3_Var15).String())
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 1, Col: 0}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var16)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -279,7 +280,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var17 templ.SafeURL
|
||||
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/cluster/masters"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 123, Col: 98}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 124, Col: 98}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -292,7 +293,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var18 templ.SafeURL
|
||||
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/cluster/volume-servers"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 128, Col: 105}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 129, Col: 105}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -305,7 +306,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var19 templ.SafeURL
|
||||
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/cluster/filers"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 133, Col: 97}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 134, Col: 97}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -318,7 +319,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var20 templ.SafeURL
|
||||
templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/cluster/s3"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 138, Col: 93}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 139, Col: 93}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -331,7 +332,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var21 templ.SafeURL
|
||||
templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/cluster/mount-clients"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 143, Col: 104}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 144, Col: 104}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var21))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -351,11 +352,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var23 string
|
||||
templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinStringErrs(templ.CSSClasses(templ_7745c5c3_Var22).String())
|
||||
templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.ResolveAttributeValue(templ.CSSClasses(templ_7745c5c3_Var22).String())
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 1, Col: 0}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var23))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var23)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -364,11 +365,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var24 string
|
||||
templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%t", isStoragePage))
|
||||
templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.ResolveAttributeValue(fmt.Sprintf("%t", isStoragePage))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 151, Col: 207}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 152, Col: 207}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var24))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var24)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -386,11 +387,11 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var26 string
|
||||
templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.JoinStringErrs(templ.CSSClasses(templ_7745c5c3_Var25).String())
|
||||
templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.ResolveAttributeValue(templ.CSSClasses(templ_7745c5c3_Var25).String())
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 1, Col: 0}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var26))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var26)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -401,7 +402,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var27 templ.SafeURL
|
||||
templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/storage/volumes"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 158, Col: 98}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 159, Col: 98}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -414,7 +415,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var28 templ.SafeURL
|
||||
templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/storage/ec-shards"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 163, Col: 100}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 164, Col: 100}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -427,7 +428,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var29 templ.SafeURL
|
||||
templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/storage/collections"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 168, Col: 102}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 169, Col: 102}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -440,7 +441,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var30 templ.SafeURL
|
||||
templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/object-store/buckets"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 182, Col: 86}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 183, Col: 86}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -453,7 +454,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var31 templ.SafeURL
|
||||
templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/object-store/s3tables/buckets"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 187, Col: 95}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 188, Col: 95}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -466,7 +467,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var32 templ.SafeURL
|
||||
templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/object-store/users"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 192, Col: 84}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 193, Col: 84}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -479,7 +480,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var33 templ.SafeURL
|
||||
templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/object-store/groups"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 197, Col: 85}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 198, Col: 85}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -492,7 +493,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var34 templ.SafeURL
|
||||
templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/object-store/service-accounts"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 202, Col: 95}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 203, Col: 95}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -505,7 +506,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var35 templ.SafeURL
|
||||
templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/object-store/policies"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 207, Col: 87}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 208, Col: 87}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -518,7 +519,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var36 templ.SafeURL
|
||||
templ_7745c5c3_Var36, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/files"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 218, Col: 71}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 219, Col: 71}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var36))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -552,7 +553,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var37 templ.SafeURL
|
||||
templ_7745c5c3_Var37, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/mq/brokers"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 239, Col: 108}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 240, Col: 108}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var37))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -570,7 +571,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var38 templ.SafeURL
|
||||
templ_7745c5c3_Var38, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/mq/brokers"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 243, Col: 101}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 244, Col: 101}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var38))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -593,7 +594,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var39 templ.SafeURL
|
||||
templ_7745c5c3_Var39, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/mq/topics"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 250, Col: 107}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 251, Col: 107}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var39))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -611,7 +612,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var40 templ.SafeURL
|
||||
templ_7745c5c3_Var40, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/mq/topics"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 254, Col: 100}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 255, Col: 100}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var40))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -634,7 +635,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var41 templ.SafeURL
|
||||
templ_7745c5c3_Var41, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/mq/brokers"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 266, Col: 97}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 267, Col: 97}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var41))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -647,7 +648,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var42 templ.SafeURL
|
||||
templ_7745c5c3_Var42, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/mq/topics"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 271, Col: 96}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 272, Col: 96}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var42))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -670,7 +671,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var43 templ.SafeURL
|
||||
templ_7745c5c3_Var43, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/default"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 288, Col: 97}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 289, Col: 97}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var43))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -688,7 +689,7 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
var templ_7745c5c3_Var44 templ.SafeURL
|
||||
templ_7745c5c3_Var44, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/default"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 292, Col: 90}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 293, Col: 90}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var44))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
@@ -703,21 +704,21 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if isIcebergWorkerPage {
|
||||
if isLifecycleWorkerPage {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "<a class=\"nav-link active\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var45 templ.SafeURL
|
||||
templ_7745c5c3_Var45, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/iceberg"))
|
||||
templ_7745c5c3_Var45, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lifecycle"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 299, Col: 97}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 300, Col: 99}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var45))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "\"><i class=\"fas fa-snowflake me-2\"></i>Iceberg</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "\"><i class=\"fas fa-hourglass-half me-2\"></i>Lifecycle</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -727,15 +728,15 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var46 templ.SafeURL
|
||||
templ_7745c5c3_Var46, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/iceberg"))
|
||||
templ_7745c5c3_Var46, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lifecycle"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 303, Col: 90}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 304, Col: 92}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var46))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "\"><i class=\"fas fa-snowflake me-2\"></i>Iceberg</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "\"><i class=\"fas fa-hourglass-half me-2\"></i>Lifecycle</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -744,21 +745,21 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if isLifecycleWorkerPage {
|
||||
if isIcebergWorkerPage {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "<a class=\"nav-link active\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var47 templ.SafeURL
|
||||
templ_7745c5c3_Var47, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lifecycle"))
|
||||
templ_7745c5c3_Var47, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/iceberg"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 310, Col: 99}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 311, Col: 97}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var47))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "\"><i class=\"fas fa-hourglass-half me-2\"></i>Lifecycle</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "\"><i class=\"fas fa-snowflake me-2\"></i>Iceberg</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -768,20 +769,61 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var48 templ.SafeURL
|
||||
templ_7745c5c3_Var48, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lifecycle"))
|
||||
templ_7745c5c3_Var48, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/iceberg"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 314, Col: 92}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 315, Col: 90}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var48))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "\"><i class=\"fas fa-hourglass-half me-2\"></i>Lifecycle</a>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "\"><i class=\"fas fa-snowflake me-2\"></i>Iceberg</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "</li></ul></div></div><!-- Sidebar backdrop for mobile --><div class=\"sidebar-backdrop\" id=\"sidebarBackdrop\"></div><!-- Main content --><main class=\"col-md-9 ms-sm-auto col-lg-10 px-3 px-md-4\"><div class=\"pt-3\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "</li><li class=\"nav-item\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if isLanceWorkerPage {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "<a class=\"nav-link active\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var49 templ.SafeURL
|
||||
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lance"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 322, Col: 95}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var49))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 71, "\"><i class=\"fas fa-vector-square me-2\"></i>Lance</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
} else {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "<a class=\"nav-link\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var50 templ.SafeURL
|
||||
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.JoinURLErrs(view.P("/plugin/lanes/lance"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 326, Col: 88}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var50))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "\"><i class=\"fas fa-vector-square me-2\"></i>Lance</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 74, "</li></ul></div></div><!-- Sidebar backdrop for mobile --><div class=\"sidebar-backdrop\" id=\"sidebarBackdrop\"></div><!-- Main content --><main class=\"col-md-9 ms-sm-auto col-lg-10 px-3 px-md-4\"><div class=\"pt-3\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -789,127 +831,127 @@ func Layout(view ViewContext, content templ.Component) templ.Component {
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "</div></main></div></div><!-- Footer --><footer class=\"footer mt-auto py-3 bg-light\"><div class=\"container-fluid text-center\"><small class=\"text-muted\">© ")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 75, "</div></main></div></div><!-- Footer --><footer class=\"footer mt-auto py-3 bg-light\"><div class=\"container-fluid text-center\"><small class=\"text-muted\">© ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var49 string
|
||||
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", time.Now().Year()))
|
||||
var templ_7745c5c3_Var51 string
|
||||
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", time.Now().Year()))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 339, Col: 60}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 351, Col: 60}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var49))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var51))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 71, " SeaweedFS Admin v")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var50 string
|
||||
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.JoinStringErrs(version.VERSION_NUMBER)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 339, Col: 102}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var50))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, " ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if version.COMMIT != "" {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "<span class=\"mx-1\">(")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var51 string
|
||||
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.JoinStringErrs(version.COMMIT)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 341, Col: 55}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var51))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 74, ")</span> ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
if !strings.Contains(version.VERSION, "enterprise") {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 75, "<span class=\"mx-2\">•</span> <a href=\"https://seaweedfs.com\" target=\"_blank\" class=\"text-decoration-none\"><i class=\"fas fa-star me-1\"></i>Enterprise Version Available</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 76, "</small></div></footer><!-- Bootstrap JS --><script src=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 76, " SeaweedFS Admin v")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var52 string
|
||||
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(string(view.P("/static/js/bootstrap.bundle.min.js")))
|
||||
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(version.VERSION_NUMBER)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 354, Col: 70}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 351, Col: 102}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var52))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 77, "\"></script><!-- Modal Alerts JS (replaces native alert/confirm) --><script src=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 77, " ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var53 string
|
||||
templ_7745c5c3_Var53, templ_7745c5c3_Err = templ.JoinStringErrs(string(view.P("/static/js/modal-alerts.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 356, Col: 62}
|
||||
if version.COMMIT != "" {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 78, "<span class=\"mx-1\">(")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var53 string
|
||||
templ_7745c5c3_Var53, templ_7745c5c3_Err = templ.JoinStringErrs(version.COMMIT)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 353, Col: 55}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var53))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 79, ")</span> ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var53))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
if !strings.Contains(version.VERSION, "enterprise") {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 80, "<span class=\"mx-2\">•</span> <a href=\"https://seaweedfs.com\" target=\"_blank\" class=\"text-decoration-none\"><i class=\"fas fa-star me-1\"></i>Enterprise Version Available</a>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 78, "\"></script><!-- Custom JS --><script src=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 81, "</small></div></footer><!-- Bootstrap JS --><script src=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var54 string
|
||||
templ_7745c5c3_Var54, templ_7745c5c3_Err = templ.JoinStringErrs(string(view.P("/static/js/admin.js")))
|
||||
templ_7745c5c3_Var54, templ_7745c5c3_Err = templ.ResolveAttributeValue(string(view.P("/static/js/bootstrap.bundle.min.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 358, Col: 55}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 366, Col: 70}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var54))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var54)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 79, "\"></script><script src=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 82, "\"></script><!-- Modal Alerts JS (replaces native alert/confirm) --><script src=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var55 string
|
||||
templ_7745c5c3_Var55, templ_7745c5c3_Err = templ.JoinStringErrs(string(view.P("/static/js/iam-utils.js")))
|
||||
templ_7745c5c3_Var55, templ_7745c5c3_Err = templ.ResolveAttributeValue(string(view.P("/static/js/modal-alerts.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 359, Col: 59}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 368, Col: 62}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var55))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var55)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 80, "\"></script><script src=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 83, "\"></script><!-- Custom JS --><script src=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var56 string
|
||||
templ_7745c5c3_Var56, templ_7745c5c3_Err = templ.JoinStringErrs(string(view.P("/static/js/s3tables.js")))
|
||||
templ_7745c5c3_Var56, templ_7745c5c3_Err = templ.ResolveAttributeValue(string(view.P("/static/js/admin.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 360, Col: 58}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 370, Col: 55}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var56))
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var56)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 81, "\"></script></body></html>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 84, "\"></script><script src=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var57 string
|
||||
templ_7745c5c3_Var57, templ_7745c5c3_Err = templ.ResolveAttributeValue(string(view.P("/static/js/iam-utils.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 371, Col: 59}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var57)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 85, "\"></script><script src=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var58 string
|
||||
templ_7745c5c3_Var58, templ_7745c5c3_Err = templ.ResolveAttributeValue(string(view.P("/static/js/s3tables.js")))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 372, Col: 58}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var58)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "\"></script></body></html>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
@@ -933,140 +975,140 @@ func LoginForm(title string, errorMessage string, csrfToken string) templ.Compon
|
||||
}()
|
||||
}
|
||||
ctx = templ.InitializeContext(ctx)
|
||||
templ_7745c5c3_Var57 := templ.GetChildren(ctx)
|
||||
if templ_7745c5c3_Var57 == nil {
|
||||
templ_7745c5c3_Var57 = templ.NopComponent
|
||||
templ_7745c5c3_Var59 := templ.GetChildren(ctx)
|
||||
if templ_7745c5c3_Var59 == nil {
|
||||
templ_7745c5c3_Var59 = templ.NopComponent
|
||||
}
|
||||
ctx = templ.ClearChildren(ctx)
|
||||
prefix := dash.URLPrefixFromContext(ctx)
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 82, "<!doctype html><html lang=\"en\"><head><meta charset=\"UTF-8\"><title>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "<!doctype html><html lang=\"en\"><head><meta charset=\"UTF-8\"><title>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var58 string
|
||||
templ_7745c5c3_Var58, templ_7745c5c3_Err = templ.JoinStringErrs(title)
|
||||
var templ_7745c5c3_Var60 string
|
||||
templ_7745c5c3_Var60, templ_7745c5c3_Err = templ.JoinStringErrs(title)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 373, Col: 17}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var58))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 83, " - Login</title><link rel=\"icon\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var59 templ.SafeURL
|
||||
templ_7745c5c3_Var59, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/favicon.ico")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 374, Col: 58}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var59))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 84, "\" type=\"image/x-icon\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var60 templ.SafeURL
|
||||
templ_7745c5c3_Var60, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/css/bootstrap.min.css")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 376, Col: 57}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 385, Col: 17}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var60))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 85, "\" rel=\"stylesheet\"><link href=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, " - Login</title><link rel=\"icon\" href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var61 templ.SafeURL
|
||||
templ_7745c5c3_Var61, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/css/fontawesome.min.css")
|
||||
templ_7745c5c3_Var61, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/favicon.ico")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 377, Col: 59}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 386, Col: 58}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var61))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "\" rel=\"stylesheet\"></head><body class=\"bg-light\"><div class=\"container\"><div class=\"row justify-content-center min-vh-100 align-items-center\"><div class=\"col-md-6 col-lg-4\"><div class=\"card shadow\"><div class=\"card-body p-5\"><div class=\"text-center mb-4\"><i class=\"fas fa-server fa-3x text-primary mb-3\"></i><h4 class=\"card-title\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "\" type=\"image/x-icon\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var62 string
|
||||
templ_7745c5c3_Var62, templ_7745c5c3_Err = templ.JoinStringErrs(title)
|
||||
var templ_7745c5c3_Var62 templ.SafeURL
|
||||
templ_7745c5c3_Var62, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/css/bootstrap.min.css")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 387, Col: 57}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 388, Col: 57}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var62))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "</h4><p class=\"text-muted\">Please sign in to continue</p></div>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "\" rel=\"stylesheet\"><link href=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
if errorMessage != "" {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, "<div class=\"alert alert-danger\" role=\"alert\"><i class=\"fas fa-exclamation-triangle me-2\"></i> ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var63 string
|
||||
templ_7745c5c3_Var63, templ_7745c5c3_Err = templ.JoinStringErrs(errorMessage)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 394, Col: 45}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var63))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var63 templ.SafeURL
|
||||
templ_7745c5c3_Var63, templ_7745c5c3_Err = templ.JoinURLErrs(prefix + "/static/css/fontawesome.min.css")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 389, Col: 59}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "<form method=\"POST\" action=\"")
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var63))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var64 templ.SafeURL
|
||||
templ_7745c5c3_Var64, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(prefix + "/login"))
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "\" rel=\"stylesheet\"></head><body class=\"bg-light\"><div class=\"container\"><div class=\"row justify-content-center min-vh-100 align-items-center\"><div class=\"col-md-6 col-lg-4\"><div class=\"card shadow\"><div class=\"card-body p-5\"><div class=\"text-center mb-4\"><i class=\"fas fa-server fa-3x text-primary mb-3\"></i><h4 class=\"card-title\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 398, Col: 85}
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var64 string
|
||||
templ_7745c5c3_Var64, templ_7745c5c3_Err = templ.JoinStringErrs(title)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 399, Col: 57}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var64))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "\"><input type=\"hidden\" name=\"csrf_token\" value=\"")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "</h4><p class=\"text-muted\">Please sign in to continue</p></div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var65 string
|
||||
templ_7745c5c3_Var65, templ_7745c5c3_Err = templ.JoinStringErrs(csrfToken)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 399, Col: 84}
|
||||
if errorMessage != "" {
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, "<div class=\"alert alert-danger\" role=\"alert\"><i class=\"fas fa-exclamation-triangle me-2\"></i> ")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var65 string
|
||||
templ_7745c5c3_Var65, templ_7745c5c3_Err = templ.JoinStringErrs(errorMessage)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 406, Col: 45}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var65))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, "</div>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var65))
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "<form method=\"POST\" action=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "\"><div class=\"mb-3\"><label for=\"username\" class=\"form-label\">Username</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-user\"></i></span> <input type=\"text\" class=\"form-control\" id=\"username\" name=\"username\" required></div></div><div class=\"mb-4\"><label for=\"password\" class=\"form-label\">Password</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-lock\"></i></span> <input type=\"password\" class=\"form-control\" id=\"password\" name=\"password\" required></div></div><button type=\"submit\" class=\"btn btn-primary w-100\"><i class=\"fas fa-sign-in-alt me-2\"></i>Sign In</button></form></div></div></div></div></div><script src=\"")
|
||||
var templ_7745c5c3_Var66 templ.SafeURL
|
||||
templ_7745c5c3_Var66, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(prefix + "/login"))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var66 string
|
||||
templ_7745c5c3_Var66, templ_7745c5c3_Err = templ.JoinStringErrs(prefix + "/static/js/bootstrap.bundle.min.js")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 430, Col: 63}
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 410, Col: 85}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var66))
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, "\"></script></body></html>")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, "\"><input type=\"hidden\" name=\"csrf_token\" value=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var67 string
|
||||
templ_7745c5c3_Var67, templ_7745c5c3_Err = templ.ResolveAttributeValue(csrfToken)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 411, Col: 84}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var67)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 97, "\"><div class=\"mb-3\"><label for=\"username\" class=\"form-label\">Username</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-user\"></i></span> <input type=\"text\" class=\"form-control\" id=\"username\" name=\"username\" required></div></div><div class=\"mb-4\"><label for=\"password\" class=\"form-label\">Password</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-lock\"></i></span> <input type=\"password\" class=\"form-control\" id=\"password\" name=\"password\" required></div></div><button type=\"submit\" class=\"btn btn-primary w-100\"><i class=\"fas fa-sign-in-alt me-2\"></i>Sign In</button></form></div></div></div></div></div><script src=\"")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
var templ_7745c5c3_Var68 string
|
||||
templ_7745c5c3_Var68, templ_7745c5c3_Err = templ.ResolveAttributeValue(prefix + "/static/js/bootstrap.bundle.min.js")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 442, Col: 63}
|
||||
}
|
||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var68)
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 98, "\"></script></body></html>")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
|
||||
@@ -51,6 +51,7 @@ type AdminOptions struct {
|
||||
readOnlyPassword *string
|
||||
dataDir *string
|
||||
icebergPort *int
|
||||
lancePort *int
|
||||
urlPrefix *string
|
||||
metricsHttpPort *int
|
||||
metricsHttpIp *string
|
||||
@@ -74,6 +75,7 @@ func init() {
|
||||
a.readOnlyUser = cmdAdmin.Flag.String("readOnlyUser", "", "read-only user username (optional, for view-only access)")
|
||||
a.readOnlyPassword = cmdAdmin.Flag.String("readOnlyPassword", "", "read-only user password (optional, for view-only access; requires adminPassword to be set)")
|
||||
a.icebergPort = cmdAdmin.Flag.Int("iceberg.port", 8181, "Iceberg REST Catalog port (0 to hide in UI)")
|
||||
a.lancePort = cmdAdmin.Flag.Int("lance.port", 9101, "Lance Namespace port (0 to hide in UI)")
|
||||
a.urlPrefix = cmdAdmin.Flag.String("urlPrefix", "", "URL path prefix when running behind a reverse proxy under a subdirectory (e.g. /seaweedfs)")
|
||||
a.metricsHttpPort = cmdAdmin.Flag.Int("metricsPort", 0, "Prometheus metrics listen port")
|
||||
a.metricsHttpIp = cmdAdmin.Flag.String("metricsIp", "", "metrics listen ip. If empty, listens on all interfaces.")
|
||||
@@ -314,7 +316,7 @@ func runAdmin(cmd *Command, args []string) bool {
|
||||
}
|
||||
|
||||
// Start the admin server with all masters (UI enabled by default)
|
||||
err := startAdminServer(ctx, a, true, *a.icebergPort, urlPrefix)
|
||||
err := startAdminServer(ctx, a, true, *a.icebergPort, *a.lancePort, urlPrefix)
|
||||
if err != nil {
|
||||
fmt.Printf("Admin server error: %v\n", err)
|
||||
return false
|
||||
@@ -325,7 +327,7 @@ func runAdmin(cmd *Command, args []string) bool {
|
||||
}
|
||||
|
||||
// startAdminServer starts the actual admin server
|
||||
func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, icebergPort int, urlPrefix string) error {
|
||||
func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, icebergPort, lancePort int, urlPrefix string) error {
|
||||
// Create router
|
||||
r := mux.NewRouter()
|
||||
r.Use(loggingMiddleware)
|
||||
@@ -388,7 +390,7 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
|
||||
r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", admin.StaticHandler()))
|
||||
|
||||
// Create admin server (plugin is always enabled)
|
||||
adminServer := dash.NewAdminServer(*options.master, *options.filerGroup, nil, dataDir, icebergPort)
|
||||
adminServer := dash.NewAdminServer(*options.master, *options.filerGroup, nil, dataDir, icebergPort, lancePort)
|
||||
|
||||
if err := adminServer.ApplyPluginConfigFromToml(util.GetViper()); err != nil {
|
||||
return fmt.Errorf("apply admin.toml to plugin config: %w", err)
|
||||
|
||||
@@ -157,6 +157,7 @@ func init() {
|
||||
filerS3Options.cipher = cmdFiler.Flag.Bool("s3.encryptVolumeData", false, "encrypt data on volume servers for S3 uploads")
|
||||
filerS3Options.iamReadOnly = cmdFiler.Flag.Bool("s3.iam.readOnly", true, "disable IAM write operations on this server")
|
||||
filerS3Options.portIceberg = cmdFiler.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)")
|
||||
filerS3Options.portLance = cmdFiler.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)")
|
||||
filerS3Options.externalUrl = cmdFiler.Flag.String("s3.externalUrl", "", "the external URL clients use to connect (e.g. https://api.example.com:9000). Used for S3 signature verification behind a reverse proxy. Falls back to S3_EXTERNAL_URL env var.")
|
||||
filerS3Options.defaultFileMode = cmdFiler.Flag.String("s3.defaultFileMode", "", "default file mode for S3 uploaded objects, e.g. 0660, 0644, 0666")
|
||||
filerS3Options.cacheSizeMB = cmdFiler.Flag.Int64("s3.cacheCapacityMB", 0, "in-memory chunk cache capacity in MB for S3 GETs shared across requests (0 disables)")
|
||||
|
||||
+73
-9
@@ -212,6 +212,9 @@ func miniStartupServices() []string {
|
||||
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
|
||||
services = append(services, "Iceberg")
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
services = append(services, "Lance")
|
||||
}
|
||||
}
|
||||
services = append(services, "Admin")
|
||||
return services
|
||||
@@ -498,6 +501,7 @@ func initMiniS3Flags() {
|
||||
miniS3Options.portHttps = cmdMini.Flag.Int("s3.port.https", 0, "s3 server https listen port")
|
||||
miniS3Options.portGrpc = cmdMini.Flag.Int("s3.port.grpc", 0, "s3 server grpc listen port")
|
||||
miniS3Options.portIceberg = cmdMini.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)")
|
||||
miniS3Options.portLance = cmdMini.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)")
|
||||
miniS3Options.icebergCredentialRole = cmdMini.Flag.String("s3.iceberg.credentialRole", "", "IAM role ARN the Iceberg catalog assumes to vend table-scoped credentials (empty disables vending)")
|
||||
miniS3Options.icebergCredentialDuration = cmdMini.Flag.Int("s3.iceberg.credentialDurationSeconds", 3600, "lifetime of credentials vended by the Iceberg catalog")
|
||||
miniS3Options.domainName = cmdMini.Flag.String("s3.domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}")
|
||||
@@ -884,6 +888,14 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
|
||||
grpcPtr *int
|
||||
}{miniS3Options.portIceberg, "Iceberg", "s3.port.iceberg", nil})
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
portConfigs = append(portConfigs, struct {
|
||||
port *int
|
||||
name string
|
||||
flagName string
|
||||
grpcPtr *int
|
||||
}{miniS3Options.portLance, "Lance", "s3.port.lance", nil})
|
||||
}
|
||||
}
|
||||
portConfigs = append(portConfigs, struct {
|
||||
port *int
|
||||
@@ -935,9 +947,13 @@ func ensureAllPortsAvailableOnIP(bindIp string) error {
|
||||
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
|
||||
icebergPortStr = fmt.Sprintf("%d", *miniS3Options.portIceberg)
|
||||
}
|
||||
glog.V(1).Infof("Final port configuration - Master: %d, Filer: %d, Volume: %d, S3: %d, Iceberg: %s, WebDAV: %d, Admin: %d",
|
||||
lancePortStr := "disabled"
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
lancePortStr = fmt.Sprintf("%d", *miniS3Options.portLance)
|
||||
}
|
||||
glog.V(1).Infof("Final port configuration - Master: %d, Filer: %d, Volume: %d, S3: %d, Iceberg: %s, Lance: %s, WebDAV: %d, Admin: %d",
|
||||
*miniMasterOptions.port, *miniFilerOptions.port, *miniOptions.v.port,
|
||||
*miniS3Options.port, icebergPortStr, *miniWebDavOptions.port, *miniAdminOptions.port)
|
||||
*miniS3Options.port, icebergPortStr, lancePortStr, *miniWebDavOptions.port, *miniAdminOptions.port)
|
||||
|
||||
// Log gRPC ports too (now finalized)
|
||||
glog.V(1).Infof("gRPC port configuration - Master: %d, Filer: %d, Volume: %d, S3: %d, Admin: %d",
|
||||
@@ -967,6 +983,9 @@ func initializeGrpcPortsOnIP(bindIp string) {
|
||||
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
|
||||
allocatedPorts[*miniS3Options.portIceberg] = true
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
allocatedPorts[*miniS3Options.portLance] = true
|
||||
}
|
||||
}
|
||||
|
||||
grpcConfigs := []struct {
|
||||
@@ -1421,10 +1440,14 @@ func startMiniServices(miniWhiteList []string, allServicesReady chan struct{}) {
|
||||
go func() {
|
||||
defer done()
|
||||
defer reportMiniStopped("S3")
|
||||
// Iceberg lives inside the S3 server; report it stopped alongside.
|
||||
// Iceberg and Lance live inside the S3 server; report them stopped
|
||||
// alongside it.
|
||||
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
|
||||
defer reportMiniStopped("Iceberg")
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
defer reportMiniStopped("Lance")
|
||||
}
|
||||
startMiniService("S3", startS3Service, *miniS3Options.port)
|
||||
}()
|
||||
}
|
||||
@@ -1452,6 +1475,12 @@ func startMiniServices(miniWhiteList []string, allServicesReady chan struct{}) {
|
||||
}
|
||||
waitForServiceReady("Iceberg", *miniS3Options.portIceberg, bindIp)
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
if miniProgressBoard != nil {
|
||||
miniProgressBoard.starting("Lance")
|
||||
}
|
||||
waitForServiceReady("Lance", *miniS3Options.portLance, bindIp)
|
||||
}
|
||||
}
|
||||
if *miniEnableWebDAV {
|
||||
waitForServiceReady("WebDAV", *miniWebDavOptions.port, bindIp)
|
||||
@@ -1585,11 +1614,18 @@ func startMiniAdminWithWorker(allServicesReady chan struct{}) {
|
||||
go func() {
|
||||
defer done()
|
||||
defer reportMiniStopped("Admin")
|
||||
var icebergPort int
|
||||
if miniS3Options.portIceberg != nil {
|
||||
icebergPort = *miniS3Options.portIceberg
|
||||
// Only advertise a catalog port when S3 is actually running: with -s3=false
|
||||
// the admin UI would otherwise print an endpoint nothing is listening on.
|
||||
var icebergPort, lancePort int
|
||||
if miniEnableS3 != nil && *miniEnableS3 {
|
||||
if miniS3Options.portIceberg != nil {
|
||||
icebergPort = *miniS3Options.portIceberg
|
||||
}
|
||||
if miniS3Options.portLance != nil {
|
||||
lancePort = *miniS3Options.portLance
|
||||
}
|
||||
}
|
||||
if err := startAdminServer(ctx, miniAdminOptions, *miniEnableAdminUI, icebergPort, urlPrefix); err != nil {
|
||||
if err := startAdminServer(ctx, miniAdminOptions, *miniEnableAdminUI, icebergPort, lancePort, urlPrefix); err != nil {
|
||||
glog.Errorf("Admin server error: %v", err)
|
||||
}
|
||||
}()
|
||||
@@ -1856,6 +1892,9 @@ func printWelcomeMessage() {
|
||||
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
|
||||
fmt.Fprintf(&sb, " Iceberg Catalog: http://%s:%d\n", *miniIp, *miniS3Options.portIceberg)
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
fmt.Fprintf(&sb, " Lance Namespace: http://%s:%d\n", *miniIp, *miniS3Options.portLance)
|
||||
}
|
||||
}
|
||||
if *miniEnableAdminUI {
|
||||
fmt.Fprintf(&sb, " Admin UI: http://%s:%d\n", *miniIp, *miniAdminOptions.port)
|
||||
@@ -1984,6 +2023,15 @@ func ensureMiniTableBuckets(bucketSpec string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// A bucket holds one format, and the format decides which catalog serves it.
|
||||
// Creating one in a format this mini does not serve leaves a bucket no
|
||||
// client can reach, so take the format from the endpoint that is running.
|
||||
format := miniTableBucketFormat()
|
||||
if format == "" {
|
||||
glog.Warningf("not creating table buckets %q: neither the Iceberg nor the Lance endpoint is enabled, so nothing could reach them", bucketSpec)
|
||||
return nil
|
||||
}
|
||||
|
||||
filerAddress := pb.NewServerAddress(*miniIp, *miniFilerOptions.port, *miniFilerOptions.portGrpc)
|
||||
grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.client")
|
||||
|
||||
@@ -1992,12 +2040,12 @@ func ensureMiniTableBuckets(bucketSpec string) error {
|
||||
mgrClient := s3tables.NewManagerClient(client)
|
||||
for _, name := range names {
|
||||
ctx, cancel := context.WithTimeout(miniClientsCtx(), 5*time.Second)
|
||||
req := &s3tables.CreateTableBucketRequest{Name: name}
|
||||
req := &s3tables.CreateTableBucketRequest{Name: name, Format: format}
|
||||
var resp s3tables.CreateTableBucketResponse
|
||||
err := manager.Execute(ctx, mgrClient, "CreateTableBucket", req, &resp, s3tables.DefaultAccountID)
|
||||
cancel()
|
||||
if err == nil {
|
||||
glog.V(0).Infof("created table bucket %s", name)
|
||||
glog.V(0).Infof("created %s table bucket %s", format, name)
|
||||
continue
|
||||
}
|
||||
var s3Err *s3tables.S3TablesError
|
||||
@@ -2011,6 +2059,22 @@ func ensureMiniTableBuckets(bucketSpec string) error {
|
||||
})
|
||||
}
|
||||
|
||||
// miniTableBucketFormat is the format a pre-created table bucket should hold:
|
||||
// Iceberg when its catalog is running, else Lance, else none because neither
|
||||
// server is up.
|
||||
func miniTableBucketFormat() string {
|
||||
if miniEnableS3 == nil || !*miniEnableS3 {
|
||||
return ""
|
||||
}
|
||||
if miniS3Options.portIceberg != nil && *miniS3Options.portIceberg > 0 {
|
||||
return s3tables.FormatIceberg
|
||||
}
|
||||
if miniS3Options.portLance != nil && *miniS3Options.portLance > 0 {
|
||||
return s3tables.FormatLance
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// parseBucketList splits a comma-separated bucket spec into a deduplicated list
|
||||
// of trimmed, non-empty names, preserving the order they were given.
|
||||
func parseBucketList(spec string) []string {
|
||||
|
||||
@@ -24,7 +24,9 @@ import (
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/s3_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/iceberg"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/lance"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"github.com/seaweedfs/seaweedfs/weed/security"
|
||||
stats_collect "github.com/seaweedfs/seaweedfs/weed/stats"
|
||||
"github.com/seaweedfs/seaweedfs/weed/util"
|
||||
@@ -48,6 +50,7 @@ type S3Options struct {
|
||||
portHttps *int
|
||||
portGrpc *int
|
||||
portIceberg *int
|
||||
portLance *int
|
||||
icebergCredentialRole *string
|
||||
icebergCredentialDuration *int
|
||||
config *string
|
||||
@@ -93,6 +96,7 @@ func init() {
|
||||
s3StandaloneOptions.portHttps = cmdS3.Flag.Int("port.https", 0, "s3 server https listen port")
|
||||
s3StandaloneOptions.portGrpc = cmdS3.Flag.Int("port.grpc", 0, "s3 server grpc listen port")
|
||||
s3StandaloneOptions.portIceberg = cmdS3.Flag.Int("port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)")
|
||||
s3StandaloneOptions.portLance = cmdS3.Flag.Int("port.lance", 9101, "Lance Namespace server listen port (0 to disable); credential vending uses -iceberg.credentialRole")
|
||||
s3StandaloneOptions.icebergCredentialRole = cmdS3.Flag.String("iceberg.credentialRole", "", "IAM role ARN the Iceberg catalog assumes to vend table-scoped credentials (empty disables vending)")
|
||||
s3StandaloneOptions.icebergCredentialDuration = cmdS3.Flag.Int("iceberg.credentialDurationSeconds", 3600, "lifetime of credentials vended by the Iceberg catalog")
|
||||
s3StandaloneOptions.domainName = cmdS3.Flag.String("domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}")
|
||||
@@ -385,6 +389,11 @@ func (s3opt *S3Options) startS3Server() bool {
|
||||
go s3opt.startIcebergServer(s3ApiServer)
|
||||
}
|
||||
|
||||
// Start Lance Namespace server if enabled
|
||||
if s3opt.portLance != nil && *s3opt.portLance > 0 {
|
||||
go s3opt.startLanceServer(s3ApiServer)
|
||||
}
|
||||
|
||||
if runtime.GOOS != "windows" {
|
||||
localSocket := *s3opt.localSocket
|
||||
if localSocket == "" {
|
||||
@@ -588,6 +597,82 @@ func (s3opt *S3Options) startIcebergServer(s3ApiServer *s3api.S3ApiServer) {
|
||||
}
|
||||
}
|
||||
|
||||
// startLanceServer starts the Lance Namespace server on a separate port. It
|
||||
// shares the Iceberg catalog's credential role: one deployment vends table
|
||||
// credentials one way, whichever catalog the client speaks to.
|
||||
func (s3opt *S3Options) startLanceServer(s3ApiServer *s3api.S3ApiServer) {
|
||||
lanceRouter := mux.NewRouter().SkipClean(true)
|
||||
lanceRouter.Use(util_http.EscapeSemicolonsInQuery)
|
||||
|
||||
lanceServer := lance.NewServer(s3ApiServer, s3ApiServer)
|
||||
if s3opt.icebergCredentialRole != nil && *s3opt.icebergCredentialRole != "" {
|
||||
lanceServer.SetCredentialVendor(lanceCredentialVendor{s3ApiServer})
|
||||
}
|
||||
lanceServer.SetS3Endpoint(s3opt.deriveLanceStorageEndpoint())
|
||||
lanceServer.SetS3Region(s3tables.DefaultRegion)
|
||||
lanceServer.RegisterRoutes(lanceRouter)
|
||||
|
||||
listenAddress := fmt.Sprintf("%s:%d", *s3opt.bindIp, *s3opt.portLance)
|
||||
lanceListener, lanceLocalListener, err := util.NewIpAndLocalListeners(
|
||||
*s3opt.bindIp, *s3opt.portLance, time.Duration(*s3opt.idleTimeout)*time.Second)
|
||||
if err != nil {
|
||||
glog.Fatalf("Lance Namespace listener on %s error: %v", listenAddress, err)
|
||||
}
|
||||
|
||||
glog.V(0).Infof("Start Lance Namespace Server at http://%s", listenAddress)
|
||||
|
||||
httpS := newHttpServer(lanceRouter, nil)
|
||||
if s3opt.shutdownCtx != nil {
|
||||
go func() {
|
||||
<-s3opt.shutdownCtx.Done()
|
||||
httpS.Shutdown(context.Background())
|
||||
}()
|
||||
}
|
||||
if lanceLocalListener != nil {
|
||||
go func() {
|
||||
if err := httpS.Serve(lanceLocalListener); err != nil && err != http.ErrServerClosed {
|
||||
glog.V(0).Infof("Lance localhost listener error: %v", err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
if err = httpS.Serve(lanceListener); err != nil && err != http.ErrServerClosed {
|
||||
glog.Fatalf("Lance Namespace Server Fail to serve: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// deriveLanceStorageEndpoint picks the endpoint the Lance namespace puts in
|
||||
// storage_options. It falls back to the advertised -ip where the Iceberg
|
||||
// derivation gives up, because the two clients are not in the same position: a
|
||||
// Spark or Trino Iceberg client brings its own s3.endpoint and advertising the
|
||||
// wrong one hijacks it, whereas storage_options is the only place a Lance
|
||||
// client learns where the store is. Without one, object_store quietly falls
|
||||
// back to real AWS S3 and the failure reads like a credentials problem.
|
||||
func (s3opt *S3Options) deriveLanceStorageEndpoint() string {
|
||||
if endpoint := s3opt.deriveS3AdvertisedEndpoint(); endpoint != "" {
|
||||
return endpoint
|
||||
}
|
||||
host := ""
|
||||
if s3opt.ip != nil {
|
||||
host = *s3opt.ip
|
||||
}
|
||||
switch host {
|
||||
case "", "0.0.0.0", "::", "[::]":
|
||||
return ""
|
||||
}
|
||||
scheme := "http"
|
||||
port := 0
|
||||
if s3opt.port != nil {
|
||||
port = *s3opt.port
|
||||
}
|
||||
if s3opt.tlsPrivateKey != nil && *s3opt.tlsPrivateKey != "" {
|
||||
scheme = "https"
|
||||
if s3opt.portHttps != nil && *s3opt.portHttps > 0 {
|
||||
port = *s3opt.portHttps
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("%s://%s", scheme, util.JoinHostPort(host, port))
|
||||
}
|
||||
|
||||
// deriveS3AdvertisedEndpoint builds the S3 endpoint URL to advertise to
|
||||
// Iceberg catalog clients as part of LoadTable FileIO config. To avoid
|
||||
// hijacking correctly-configured clients (Spark/Trino/PyIceberg all bring
|
||||
@@ -646,3 +731,22 @@ func (v icebergCredentialVendor) VendTableCredentials(ctx context.Context, princ
|
||||
Expiration: credentials.Expiration,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// lanceCredentialVendor adapts the S3 gateway's STS-backed vending to the Lance
|
||||
// namespace's interface, keeping the two packages independent of each other.
|
||||
type lanceCredentialVendor struct {
|
||||
server *s3api.S3ApiServer
|
||||
}
|
||||
|
||||
func (v lanceCredentialVendor) VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*lance.VendedCredentials, error) {
|
||||
credentials, err := v.server.VendTableCredentials(ctx, principal, bucket, prefix)
|
||||
if err != nil || credentials == nil {
|
||||
return nil, err
|
||||
}
|
||||
return &lance.VendedCredentials{
|
||||
AccessKeyID: credentials.AccessKeyID,
|
||||
SecretAccessKey: credentials.SecretAccessKey,
|
||||
SessionToken: credentials.SessionToken,
|
||||
Expiration: credentials.Expiration,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -164,6 +164,7 @@ func init() {
|
||||
s3Options.portHttps = cmdServer.Flag.Int("s3.port.https", 0, "s3 server https listen port")
|
||||
s3Options.portGrpc = cmdServer.Flag.Int("s3.port.grpc", 0, "s3 server grpc listen port")
|
||||
s3Options.portIceberg = cmdServer.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)")
|
||||
s3Options.portLance = cmdServer.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)")
|
||||
s3Options.icebergCredentialRole = cmdServer.Flag.String("s3.iceberg.credentialRole", "", "IAM role ARN the Iceberg catalog assumes to vend table-scoped credentials (empty disables vending)")
|
||||
s3Options.icebergCredentialDuration = cmdServer.Flag.Int("s3.iceberg.credentialDurationSeconds", 3600, "lifetime of credentials vended by the Iceberg catalog")
|
||||
s3Options.domainName = cmdServer.Flag.String("s3.domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}")
|
||||
|
||||
@@ -27,6 +27,8 @@ message WorkerToAdminMessage {
|
||||
DetectionComplete detection_complete = 15;
|
||||
JobProgressUpdate job_progress_update = 16;
|
||||
JobCompleted job_completed = 17;
|
||||
WorkerObservations observations = 18;
|
||||
ObjectPreviewResponse object_preview_response = 19;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -42,6 +44,7 @@ message AdminToWorkerMessage {
|
||||
ExecuteJobRequest execute_job_request = 13;
|
||||
CancelRequest cancel_request = 14;
|
||||
AdminShutdown shutdown = 15;
|
||||
RequestObjectPreview request_object_preview = 16;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -452,3 +455,61 @@ enum ActivitySource {
|
||||
ACTIVITY_SOURCE_DETECTOR = 2;
|
||||
ACTIVITY_SOURCE_EXECUTOR = 3;
|
||||
}
|
||||
|
||||
// WorkerObservations reports what a worker learned about the objects it
|
||||
// inspected. A worker opens a table to decide whether it needs a job; what it
|
||||
// saw is worth keeping either way, because for a format the cluster cannot
|
||||
// read, the worker is the only thing that can describe it.
|
||||
//
|
||||
// Observations are not work. Admin caches them and serves them back, so a
|
||||
// worker that goes away leaves its last ones standing rather than blanking a
|
||||
// page.
|
||||
message WorkerObservations {
|
||||
string job_type = 1;
|
||||
repeated ObjectObservation observations = 2;
|
||||
}
|
||||
|
||||
// ObjectObservation is one object as a worker last saw it.
|
||||
message ObjectObservation {
|
||||
// Identifier of the object, e.g. ["bucket", "namespace", "table"].
|
||||
repeated string object_id = 1;
|
||||
// What kind of object this is, e.g. "table".
|
||||
string object_kind = 2;
|
||||
// The object's format, e.g. "LANCE", so a reader can tell whose observation
|
||||
// this is without parsing the attributes.
|
||||
string format = 3;
|
||||
// Whatever the worker can cheaply say: schema, row count, fragment count.
|
||||
// The keys are the worker's to choose.
|
||||
map<string, ConfigValue> attributes = 4;
|
||||
google.protobuf.Timestamp observed_at = 5;
|
||||
}
|
||||
|
||||
// RequestObjectPreview asks a worker for sample rows of an object admin cannot
|
||||
// read itself. Unlike an observation this is not cached: it is fetched when
|
||||
// someone opens the page, because rows are the object's data rather than a
|
||||
// description of it, and holding a copy in admin is neither fresh nor its
|
||||
// business.
|
||||
message RequestObjectPreview {
|
||||
repeated string object_id = 1;
|
||||
// The format admin believes this object is, so a worker that does not own it
|
||||
// can decline instead of guessing.
|
||||
string format = 2;
|
||||
int32 row_limit = 3;
|
||||
}
|
||||
|
||||
// ObjectPreviewResponse carries the sample back, already rendered as text. The
|
||||
// worker is the only thing that knows the object's types, so it formats them;
|
||||
// admin displays what it is given.
|
||||
message ObjectPreviewResponse {
|
||||
string request_id = 6;
|
||||
bool success = 1;
|
||||
string error_message = 2;
|
||||
repeated string columns = 3;
|
||||
repeated PreviewRow rows = 4;
|
||||
// Rows in the object, which is not the number sampled.
|
||||
int64 total_rows = 5;
|
||||
}
|
||||
|
||||
message PreviewRow {
|
||||
repeated string values = 1;
|
||||
}
|
||||
|
||||
+570
-128
@@ -453,6 +453,8 @@ type WorkerToAdminMessage struct {
|
||||
// *WorkerToAdminMessage_DetectionComplete
|
||||
// *WorkerToAdminMessage_JobProgressUpdate
|
||||
// *WorkerToAdminMessage_JobCompleted
|
||||
// *WorkerToAdminMessage_Observations
|
||||
// *WorkerToAdminMessage_ObjectPreviewResponse
|
||||
Body isWorkerToAdminMessage_Body `protobuf_oneof:"body"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -581,6 +583,24 @@ func (x *WorkerToAdminMessage) GetJobCompleted() *JobCompleted {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *WorkerToAdminMessage) GetObservations() *WorkerObservations {
|
||||
if x != nil {
|
||||
if x, ok := x.Body.(*WorkerToAdminMessage_Observations); ok {
|
||||
return x.Observations
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *WorkerToAdminMessage) GetObjectPreviewResponse() *ObjectPreviewResponse {
|
||||
if x != nil {
|
||||
if x, ok := x.Body.(*WorkerToAdminMessage_ObjectPreviewResponse); ok {
|
||||
return x.ObjectPreviewResponse
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isWorkerToAdminMessage_Body interface {
|
||||
isWorkerToAdminMessage_Body()
|
||||
}
|
||||
@@ -617,6 +637,14 @@ type WorkerToAdminMessage_JobCompleted struct {
|
||||
JobCompleted *JobCompleted `protobuf:"bytes,17,opt,name=job_completed,json=jobCompleted,proto3,oneof"`
|
||||
}
|
||||
|
||||
type WorkerToAdminMessage_Observations struct {
|
||||
Observations *WorkerObservations `protobuf:"bytes,18,opt,name=observations,proto3,oneof"`
|
||||
}
|
||||
|
||||
type WorkerToAdminMessage_ObjectPreviewResponse struct {
|
||||
ObjectPreviewResponse *ObjectPreviewResponse `protobuf:"bytes,19,opt,name=object_preview_response,json=objectPreviewResponse,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*WorkerToAdminMessage_Hello) isWorkerToAdminMessage_Body() {}
|
||||
|
||||
func (*WorkerToAdminMessage_Heartbeat) isWorkerToAdminMessage_Body() {}
|
||||
@@ -633,6 +661,10 @@ func (*WorkerToAdminMessage_JobProgressUpdate) isWorkerToAdminMessage_Body() {}
|
||||
|
||||
func (*WorkerToAdminMessage_JobCompleted) isWorkerToAdminMessage_Body() {}
|
||||
|
||||
func (*WorkerToAdminMessage_Observations) isWorkerToAdminMessage_Body() {}
|
||||
|
||||
func (*WorkerToAdminMessage_ObjectPreviewResponse) isWorkerToAdminMessage_Body() {}
|
||||
|
||||
// AdminToWorkerMessage carries commands and lifecycle notifications from admin.
|
||||
type AdminToWorkerMessage struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -646,6 +678,7 @@ type AdminToWorkerMessage struct {
|
||||
// *AdminToWorkerMessage_ExecuteJobRequest
|
||||
// *AdminToWorkerMessage_CancelRequest
|
||||
// *AdminToWorkerMessage_Shutdown
|
||||
// *AdminToWorkerMessage_RequestObjectPreview
|
||||
Body isAdminToWorkerMessage_Body `protobuf_oneof:"body"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -756,6 +789,15 @@ func (x *AdminToWorkerMessage) GetShutdown() *AdminShutdown {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AdminToWorkerMessage) GetRequestObjectPreview() *RequestObjectPreview {
|
||||
if x != nil {
|
||||
if x, ok := x.Body.(*AdminToWorkerMessage_RequestObjectPreview); ok {
|
||||
return x.RequestObjectPreview
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isAdminToWorkerMessage_Body interface {
|
||||
isAdminToWorkerMessage_Body()
|
||||
}
|
||||
@@ -784,6 +826,10 @@ type AdminToWorkerMessage_Shutdown struct {
|
||||
Shutdown *AdminShutdown `protobuf:"bytes,15,opt,name=shutdown,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AdminToWorkerMessage_RequestObjectPreview struct {
|
||||
RequestObjectPreview *RequestObjectPreview `protobuf:"bytes,16,opt,name=request_object_preview,json=requestObjectPreview,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*AdminToWorkerMessage_Hello) isAdminToWorkerMessage_Body() {}
|
||||
|
||||
func (*AdminToWorkerMessage_RequestConfigSchema) isAdminToWorkerMessage_Body() {}
|
||||
@@ -796,6 +842,8 @@ func (*AdminToWorkerMessage_CancelRequest) isAdminToWorkerMessage_Body() {}
|
||||
|
||||
func (*AdminToWorkerMessage_Shutdown) isAdminToWorkerMessage_Body() {}
|
||||
|
||||
func (*AdminToWorkerMessage_RequestObjectPreview) isAdminToWorkerMessage_Body() {}
|
||||
|
||||
type WorkerHello struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
WorkerId string `protobuf:"bytes,1,opt,name=worker_id,json=workerId,proto3" json:"worker_id,omitempty"`
|
||||
@@ -3928,11 +3976,353 @@ func (x *PersistedJobTypeConfig) GetUpdatedBy() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// WorkerObservations reports what a worker learned about the objects it
|
||||
// inspected. A worker opens a table to decide whether it needs a job; what it
|
||||
// saw is worth keeping either way, because for a format the cluster cannot
|
||||
// read, the worker is the only thing that can describe it.
|
||||
//
|
||||
// Observations are not work. Admin caches them and serves them back, so a
|
||||
// worker that goes away leaves its last ones standing rather than blanking a
|
||||
// page.
|
||||
type WorkerObservations struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobType string `protobuf:"bytes,1,opt,name=job_type,json=jobType,proto3" json:"job_type,omitempty"`
|
||||
Observations []*ObjectObservation `protobuf:"bytes,2,rep,name=observations,proto3" json:"observations,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *WorkerObservations) Reset() {
|
||||
*x = WorkerObservations{}
|
||||
mi := &file_plugin_proto_msgTypes[39]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *WorkerObservations) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*WorkerObservations) ProtoMessage() {}
|
||||
|
||||
func (x *WorkerObservations) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_plugin_proto_msgTypes[39]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use WorkerObservations.ProtoReflect.Descriptor instead.
|
||||
func (*WorkerObservations) Descriptor() ([]byte, []int) {
|
||||
return file_plugin_proto_rawDescGZIP(), []int{39}
|
||||
}
|
||||
|
||||
func (x *WorkerObservations) GetJobType() string {
|
||||
if x != nil {
|
||||
return x.JobType
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *WorkerObservations) GetObservations() []*ObjectObservation {
|
||||
if x != nil {
|
||||
return x.Observations
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ObjectObservation is one object as a worker last saw it.
|
||||
type ObjectObservation struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// Identifier of the object, e.g. ["bucket", "namespace", "table"].
|
||||
ObjectId []string `protobuf:"bytes,1,rep,name=object_id,json=objectId,proto3" json:"object_id,omitempty"`
|
||||
// What kind of object this is, e.g. "table".
|
||||
ObjectKind string `protobuf:"bytes,2,opt,name=object_kind,json=objectKind,proto3" json:"object_kind,omitempty"`
|
||||
// The object's format, e.g. "LANCE", so a reader can tell whose observation
|
||||
// this is without parsing the attributes.
|
||||
Format string `protobuf:"bytes,3,opt,name=format,proto3" json:"format,omitempty"`
|
||||
// Whatever the worker can cheaply say: schema, row count, fragment count.
|
||||
// The keys are the worker's to choose.
|
||||
Attributes map[string]*ConfigValue `protobuf:"bytes,4,rep,name=attributes,proto3" json:"attributes,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
ObservedAt *timestamppb.Timestamp `protobuf:"bytes,5,opt,name=observed_at,json=observedAt,proto3" json:"observed_at,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) Reset() {
|
||||
*x = ObjectObservation{}
|
||||
mi := &file_plugin_proto_msgTypes[40]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*ObjectObservation) ProtoMessage() {}
|
||||
|
||||
func (x *ObjectObservation) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_plugin_proto_msgTypes[40]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use ObjectObservation.ProtoReflect.Descriptor instead.
|
||||
func (*ObjectObservation) Descriptor() ([]byte, []int) {
|
||||
return file_plugin_proto_rawDescGZIP(), []int{40}
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) GetObjectId() []string {
|
||||
if x != nil {
|
||||
return x.ObjectId
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) GetObjectKind() string {
|
||||
if x != nil {
|
||||
return x.ObjectKind
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) GetFormat() string {
|
||||
if x != nil {
|
||||
return x.Format
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) GetAttributes() map[string]*ConfigValue {
|
||||
if x != nil {
|
||||
return x.Attributes
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ObjectObservation) GetObservedAt() *timestamppb.Timestamp {
|
||||
if x != nil {
|
||||
return x.ObservedAt
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RequestObjectPreview asks a worker for sample rows of an object admin cannot
|
||||
// read itself. Unlike an observation this is not cached: it is fetched when
|
||||
// someone opens the page, because rows are the object's data rather than a
|
||||
// description of it, and holding a copy in admin is neither fresh nor its
|
||||
// business.
|
||||
type RequestObjectPreview struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
ObjectId []string `protobuf:"bytes,1,rep,name=object_id,json=objectId,proto3" json:"object_id,omitempty"`
|
||||
// The format admin believes this object is, so a worker that does not own it
|
||||
// can decline instead of guessing.
|
||||
Format string `protobuf:"bytes,2,opt,name=format,proto3" json:"format,omitempty"`
|
||||
RowLimit int32 `protobuf:"varint,3,opt,name=row_limit,json=rowLimit,proto3" json:"row_limit,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *RequestObjectPreview) Reset() {
|
||||
*x = RequestObjectPreview{}
|
||||
mi := &file_plugin_proto_msgTypes[41]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *RequestObjectPreview) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*RequestObjectPreview) ProtoMessage() {}
|
||||
|
||||
func (x *RequestObjectPreview) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_plugin_proto_msgTypes[41]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use RequestObjectPreview.ProtoReflect.Descriptor instead.
|
||||
func (*RequestObjectPreview) Descriptor() ([]byte, []int) {
|
||||
return file_plugin_proto_rawDescGZIP(), []int{41}
|
||||
}
|
||||
|
||||
func (x *RequestObjectPreview) GetObjectId() []string {
|
||||
if x != nil {
|
||||
return x.ObjectId
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *RequestObjectPreview) GetFormat() string {
|
||||
if x != nil {
|
||||
return x.Format
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *RequestObjectPreview) GetRowLimit() int32 {
|
||||
if x != nil {
|
||||
return x.RowLimit
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ObjectPreviewResponse carries the sample back, already rendered as text. The
|
||||
// worker is the only thing that knows the object's types, so it formats them;
|
||||
// admin displays what it is given.
|
||||
type ObjectPreviewResponse struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
RequestId string `protobuf:"bytes,6,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
|
||||
Success bool `protobuf:"varint,1,opt,name=success,proto3" json:"success,omitempty"`
|
||||
ErrorMessage string `protobuf:"bytes,2,opt,name=error_message,json=errorMessage,proto3" json:"error_message,omitempty"`
|
||||
Columns []string `protobuf:"bytes,3,rep,name=columns,proto3" json:"columns,omitempty"`
|
||||
Rows []*PreviewRow `protobuf:"bytes,4,rep,name=rows,proto3" json:"rows,omitempty"`
|
||||
// Rows in the object, which is not the number sampled.
|
||||
TotalRows int64 `protobuf:"varint,5,opt,name=total_rows,json=totalRows,proto3" json:"total_rows,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) Reset() {
|
||||
*x = ObjectPreviewResponse{}
|
||||
mi := &file_plugin_proto_msgTypes[42]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*ObjectPreviewResponse) ProtoMessage() {}
|
||||
|
||||
func (x *ObjectPreviewResponse) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_plugin_proto_msgTypes[42]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use ObjectPreviewResponse.ProtoReflect.Descriptor instead.
|
||||
func (*ObjectPreviewResponse) Descriptor() ([]byte, []int) {
|
||||
return file_plugin_proto_rawDescGZIP(), []int{42}
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) GetRequestId() string {
|
||||
if x != nil {
|
||||
return x.RequestId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) GetSuccess() bool {
|
||||
if x != nil {
|
||||
return x.Success
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) GetErrorMessage() string {
|
||||
if x != nil {
|
||||
return x.ErrorMessage
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) GetColumns() []string {
|
||||
if x != nil {
|
||||
return x.Columns
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) GetRows() []*PreviewRow {
|
||||
if x != nil {
|
||||
return x.Rows
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ObjectPreviewResponse) GetTotalRows() int64 {
|
||||
if x != nil {
|
||||
return x.TotalRows
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
type PreviewRow struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Values []string `protobuf:"bytes,1,rep,name=values,proto3" json:"values,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *PreviewRow) Reset() {
|
||||
*x = PreviewRow{}
|
||||
mi := &file_plugin_proto_msgTypes[43]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *PreviewRow) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*PreviewRow) ProtoMessage() {}
|
||||
|
||||
func (x *PreviewRow) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_plugin_proto_msgTypes[43]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use PreviewRow.ProtoReflect.Descriptor instead.
|
||||
func (*PreviewRow) Descriptor() ([]byte, []int) {
|
||||
return file_plugin_proto_rawDescGZIP(), []int{43}
|
||||
}
|
||||
|
||||
func (x *PreviewRow) GetValues() []string {
|
||||
if x != nil {
|
||||
return x.Values
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var File_plugin_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_plugin_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\fplugin.proto\x12\x06plugin\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"\x90\x05\n" +
|
||||
"\fplugin.proto\x12\x06plugin\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"\xab\x06\n" +
|
||||
"\x14WorkerToAdminMessage\x12\x1b\n" +
|
||||
"\tworker_id\x18\x01 \x01(\tR\bworkerId\x123\n" +
|
||||
"\asent_at\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\x06sentAt\x12+\n" +
|
||||
@@ -3944,8 +4334,10 @@ const file_plugin_proto_rawDesc = "" +
|
||||
"\x13detection_proposals\x18\x0e \x01(\v2\x1a.plugin.DetectionProposalsH\x00R\x12detectionProposals\x12J\n" +
|
||||
"\x12detection_complete\x18\x0f \x01(\v2\x19.plugin.DetectionCompleteH\x00R\x11detectionComplete\x12K\n" +
|
||||
"\x13job_progress_update\x18\x10 \x01(\v2\x19.plugin.JobProgressUpdateH\x00R\x11jobProgressUpdate\x12;\n" +
|
||||
"\rjob_completed\x18\x11 \x01(\v2\x14.plugin.JobCompletedH\x00R\fjobCompletedB\x06\n" +
|
||||
"\x04body\"\x86\x04\n" +
|
||||
"\rjob_completed\x18\x11 \x01(\v2\x14.plugin.JobCompletedH\x00R\fjobCompleted\x12@\n" +
|
||||
"\fobservations\x18\x12 \x01(\v2\x1a.plugin.WorkerObservationsH\x00R\fobservations\x12W\n" +
|
||||
"\x17object_preview_response\x18\x13 \x01(\v2\x1d.plugin.ObjectPreviewResponseH\x00R\x15objectPreviewResponseB\x06\n" +
|
||||
"\x04body\"\xdc\x04\n" +
|
||||
"\x14AdminToWorkerMessage\x12\x1d\n" +
|
||||
"\n" +
|
||||
"request_id\x18\x01 \x01(\tR\trequestId\x123\n" +
|
||||
@@ -3956,7 +4348,8 @@ const file_plugin_proto_rawDesc = "" +
|
||||
"\x15run_detection_request\x18\f \x01(\v2\x1b.plugin.RunDetectionRequestH\x00R\x13runDetectionRequest\x12K\n" +
|
||||
"\x13execute_job_request\x18\r \x01(\v2\x19.plugin.ExecuteJobRequestH\x00R\x11executeJobRequest\x12>\n" +
|
||||
"\x0ecancel_request\x18\x0e \x01(\v2\x15.plugin.CancelRequestH\x00R\rcancelRequest\x123\n" +
|
||||
"\bshutdown\x18\x0f \x01(\v2\x15.plugin.AdminShutdownH\x00R\bshutdownB\x06\n" +
|
||||
"\bshutdown\x18\x0f \x01(\v2\x15.plugin.AdminShutdownH\x00R\bshutdown\x12T\n" +
|
||||
"\x16request_object_preview\x18\x10 \x01(\v2\x1c.plugin.RequestObjectPreviewH\x00R\x14requestObjectPreviewB\x06\n" +
|
||||
"\x04body\"\xff\x02\n" +
|
||||
"\vWorkerHello\x12\x1b\n" +
|
||||
"\tworker_id\x18\x01 \x01(\tR\bworkerId\x12,\n" +
|
||||
@@ -4319,7 +4712,39 @@ const file_plugin_proto_rawDesc = "" +
|
||||
"\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01\x1aZ\n" +
|
||||
"\x17WorkerConfigValuesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12)\n" +
|
||||
"\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01*W\n" +
|
||||
"\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01\"n\n" +
|
||||
"\x12WorkerObservations\x12\x19\n" +
|
||||
"\bjob_type\x18\x01 \x01(\tR\ajobType\x12=\n" +
|
||||
"\fobservations\x18\x02 \x03(\v2\x19.plugin.ObjectObservationR\fobservations\"\xc5\x02\n" +
|
||||
"\x11ObjectObservation\x12\x1b\n" +
|
||||
"\tobject_id\x18\x01 \x03(\tR\bobjectId\x12\x1f\n" +
|
||||
"\vobject_kind\x18\x02 \x01(\tR\n" +
|
||||
"objectKind\x12\x16\n" +
|
||||
"\x06format\x18\x03 \x01(\tR\x06format\x12I\n" +
|
||||
"\n" +
|
||||
"attributes\x18\x04 \x03(\v2).plugin.ObjectObservation.AttributesEntryR\n" +
|
||||
"attributes\x12;\n" +
|
||||
"\vobserved_at\x18\x05 \x01(\v2\x1a.google.protobuf.TimestampR\n" +
|
||||
"observedAt\x1aR\n" +
|
||||
"\x0fAttributesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12)\n" +
|
||||
"\x05value\x18\x02 \x01(\v2\x13.plugin.ConfigValueR\x05value:\x028\x01\"h\n" +
|
||||
"\x14RequestObjectPreview\x12\x1b\n" +
|
||||
"\tobject_id\x18\x01 \x03(\tR\bobjectId\x12\x16\n" +
|
||||
"\x06format\x18\x02 \x01(\tR\x06format\x12\x1b\n" +
|
||||
"\trow_limit\x18\x03 \x01(\x05R\browLimit\"\xd6\x01\n" +
|
||||
"\x15ObjectPreviewResponse\x12\x1d\n" +
|
||||
"\n" +
|
||||
"request_id\x18\x06 \x01(\tR\trequestId\x12\x18\n" +
|
||||
"\asuccess\x18\x01 \x01(\bR\asuccess\x12#\n" +
|
||||
"\rerror_message\x18\x02 \x01(\tR\ferrorMessage\x12\x18\n" +
|
||||
"\acolumns\x18\x03 \x03(\tR\acolumns\x12&\n" +
|
||||
"\x04rows\x18\x04 \x03(\v2\x12.plugin.PreviewRowR\x04rows\x12\x1d\n" +
|
||||
"\n" +
|
||||
"total_rows\x18\x05 \x01(\x03R\ttotalRows\"$\n" +
|
||||
"\n" +
|
||||
"PreviewRow\x12\x16\n" +
|
||||
"\x06values\x18\x01 \x03(\tR\x06values*W\n" +
|
||||
"\bWorkKind\x12\x19\n" +
|
||||
"\x15WORK_KIND_UNSPECIFIED\x10\x00\x12\x17\n" +
|
||||
"\x13WORK_KIND_DETECTION\x10\x01\x12\x17\n" +
|
||||
@@ -4388,7 +4813,7 @@ func file_plugin_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_plugin_proto_enumTypes = make([]protoimpl.EnumInfo, 7)
|
||||
var file_plugin_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
|
||||
var file_plugin_proto_msgTypes = make([]protoimpl.MessageInfo, 65)
|
||||
var file_plugin_proto_goTypes = []any{
|
||||
(WorkKind)(0), // 0: plugin.WorkKind
|
||||
(JobPriority)(0), // 1: plugin.JobPriority
|
||||
@@ -4436,31 +4861,37 @@ var file_plugin_proto_goTypes = []any{
|
||||
(*CancelRequest)(nil), // 43: plugin.CancelRequest
|
||||
(*AdminShutdown)(nil), // 44: plugin.AdminShutdown
|
||||
(*PersistedJobTypeConfig)(nil), // 45: plugin.PersistedJobTypeConfig
|
||||
nil, // 46: plugin.WorkerHello.MetadataEntry
|
||||
nil, // 47: plugin.WorkerHeartbeat.QueuedJobsByTypeEntry
|
||||
nil, // 48: plugin.WorkerHeartbeat.MetadataEntry
|
||||
nil, // 49: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry
|
||||
nil, // 50: plugin.ConfigForm.DefaultValuesEntry
|
||||
nil, // 51: plugin.ValueMap.FieldsEntry
|
||||
nil, // 52: plugin.RunDetectionRequest.AdminConfigValuesEntry
|
||||
nil, // 53: plugin.RunDetectionRequest.WorkerConfigValuesEntry
|
||||
nil, // 54: plugin.JobProposal.ParametersEntry
|
||||
nil, // 55: plugin.JobProposal.LabelsEntry
|
||||
nil, // 56: plugin.ExecuteJobRequest.AdminConfigValuesEntry
|
||||
nil, // 57: plugin.ExecuteJobRequest.WorkerConfigValuesEntry
|
||||
nil, // 58: plugin.JobSpec.ParametersEntry
|
||||
nil, // 59: plugin.JobSpec.LabelsEntry
|
||||
nil, // 60: plugin.JobProgressUpdate.MetricsEntry
|
||||
nil, // 61: plugin.JobResult.OutputValuesEntry
|
||||
nil, // 62: plugin.ClusterContext.MetadataEntry
|
||||
nil, // 63: plugin.ActivityEvent.DetailsEntry
|
||||
nil, // 64: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry
|
||||
nil, // 65: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry
|
||||
(*timestamppb.Timestamp)(nil), // 66: google.protobuf.Timestamp
|
||||
(*durationpb.Duration)(nil), // 67: google.protobuf.Duration
|
||||
(*WorkerObservations)(nil), // 46: plugin.WorkerObservations
|
||||
(*ObjectObservation)(nil), // 47: plugin.ObjectObservation
|
||||
(*RequestObjectPreview)(nil), // 48: plugin.RequestObjectPreview
|
||||
(*ObjectPreviewResponse)(nil), // 49: plugin.ObjectPreviewResponse
|
||||
(*PreviewRow)(nil), // 50: plugin.PreviewRow
|
||||
nil, // 51: plugin.WorkerHello.MetadataEntry
|
||||
nil, // 52: plugin.WorkerHeartbeat.QueuedJobsByTypeEntry
|
||||
nil, // 53: plugin.WorkerHeartbeat.MetadataEntry
|
||||
nil, // 54: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry
|
||||
nil, // 55: plugin.ConfigForm.DefaultValuesEntry
|
||||
nil, // 56: plugin.ValueMap.FieldsEntry
|
||||
nil, // 57: plugin.RunDetectionRequest.AdminConfigValuesEntry
|
||||
nil, // 58: plugin.RunDetectionRequest.WorkerConfigValuesEntry
|
||||
nil, // 59: plugin.JobProposal.ParametersEntry
|
||||
nil, // 60: plugin.JobProposal.LabelsEntry
|
||||
nil, // 61: plugin.ExecuteJobRequest.AdminConfigValuesEntry
|
||||
nil, // 62: plugin.ExecuteJobRequest.WorkerConfigValuesEntry
|
||||
nil, // 63: plugin.JobSpec.ParametersEntry
|
||||
nil, // 64: plugin.JobSpec.LabelsEntry
|
||||
nil, // 65: plugin.JobProgressUpdate.MetricsEntry
|
||||
nil, // 66: plugin.JobResult.OutputValuesEntry
|
||||
nil, // 67: plugin.ClusterContext.MetadataEntry
|
||||
nil, // 68: plugin.ActivityEvent.DetailsEntry
|
||||
nil, // 69: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry
|
||||
nil, // 70: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry
|
||||
nil, // 71: plugin.ObjectObservation.AttributesEntry
|
||||
(*timestamppb.Timestamp)(nil), // 72: google.protobuf.Timestamp
|
||||
(*durationpb.Duration)(nil), // 73: google.protobuf.Duration
|
||||
}
|
||||
var file_plugin_proto_depIdxs = []int32{
|
||||
66, // 0: plugin.WorkerToAdminMessage.sent_at:type_name -> google.protobuf.Timestamp
|
||||
72, // 0: plugin.WorkerToAdminMessage.sent_at:type_name -> google.protobuf.Timestamp
|
||||
9, // 1: plugin.WorkerToAdminMessage.hello:type_name -> plugin.WorkerHello
|
||||
11, // 2: plugin.WorkerToAdminMessage.heartbeat:type_name -> plugin.WorkerHeartbeat
|
||||
12, // 3: plugin.WorkerToAdminMessage.acknowledge:type_name -> plugin.WorkerAcknowledge
|
||||
@@ -4469,104 +4900,112 @@ var file_plugin_proto_depIdxs = []int32{
|
||||
34, // 6: plugin.WorkerToAdminMessage.detection_complete:type_name -> plugin.DetectionComplete
|
||||
38, // 7: plugin.WorkerToAdminMessage.job_progress_update:type_name -> plugin.JobProgressUpdate
|
||||
39, // 8: plugin.WorkerToAdminMessage.job_completed:type_name -> plugin.JobCompleted
|
||||
66, // 9: plugin.AdminToWorkerMessage.sent_at:type_name -> google.protobuf.Timestamp
|
||||
10, // 10: plugin.AdminToWorkerMessage.hello:type_name -> plugin.AdminHello
|
||||
15, // 11: plugin.AdminToWorkerMessage.request_config_schema:type_name -> plugin.RequestConfigSchema
|
||||
32, // 12: plugin.AdminToWorkerMessage.run_detection_request:type_name -> plugin.RunDetectionRequest
|
||||
36, // 13: plugin.AdminToWorkerMessage.execute_job_request:type_name -> plugin.ExecuteJobRequest
|
||||
43, // 14: plugin.AdminToWorkerMessage.cancel_request:type_name -> plugin.CancelRequest
|
||||
44, // 15: plugin.AdminToWorkerMessage.shutdown:type_name -> plugin.AdminShutdown
|
||||
14, // 16: plugin.WorkerHello.capabilities:type_name -> plugin.JobTypeCapability
|
||||
46, // 17: plugin.WorkerHello.metadata:type_name -> plugin.WorkerHello.MetadataEntry
|
||||
13, // 18: plugin.WorkerHeartbeat.running_work:type_name -> plugin.RunningWork
|
||||
47, // 19: plugin.WorkerHeartbeat.queued_jobs_by_type:type_name -> plugin.WorkerHeartbeat.QueuedJobsByTypeEntry
|
||||
48, // 20: plugin.WorkerHeartbeat.metadata:type_name -> plugin.WorkerHeartbeat.MetadataEntry
|
||||
0, // 21: plugin.RunningWork.kind:type_name -> plugin.WorkKind
|
||||
2, // 22: plugin.RunningWork.state:type_name -> plugin.JobState
|
||||
17, // 23: plugin.ConfigSchemaResponse.job_type_descriptor:type_name -> plugin.JobTypeDescriptor
|
||||
18, // 24: plugin.JobTypeDescriptor.admin_config_form:type_name -> plugin.ConfigForm
|
||||
18, // 25: plugin.JobTypeDescriptor.worker_config_form:type_name -> plugin.ConfigForm
|
||||
30, // 26: plugin.JobTypeDescriptor.admin_runtime_defaults:type_name -> plugin.AdminRuntimeDefaults
|
||||
49, // 27: plugin.JobTypeDescriptor.worker_default_values:type_name -> plugin.JobTypeDescriptor.WorkerDefaultValuesEntry
|
||||
19, // 28: plugin.ConfigForm.sections:type_name -> plugin.ConfigSection
|
||||
50, // 29: plugin.ConfigForm.default_values:type_name -> plugin.ConfigForm.DefaultValuesEntry
|
||||
20, // 30: plugin.ConfigSection.fields:type_name -> plugin.ConfigField
|
||||
3, // 31: plugin.ConfigField.field_type:type_name -> plugin.ConfigFieldType
|
||||
4, // 32: plugin.ConfigField.widget:type_name -> plugin.ConfigWidget
|
||||
23, // 33: plugin.ConfigField.min_value:type_name -> plugin.ConfigValue
|
||||
23, // 34: plugin.ConfigField.max_value:type_name -> plugin.ConfigValue
|
||||
21, // 35: plugin.ConfigField.options:type_name -> plugin.ConfigOption
|
||||
22, // 36: plugin.ConfigField.validation_rules:type_name -> plugin.ValidationRule
|
||||
23, // 37: plugin.ConfigField.visible_when_equals:type_name -> plugin.ConfigValue
|
||||
5, // 38: plugin.ValidationRule.type:type_name -> plugin.ValidationRuleType
|
||||
67, // 39: plugin.ConfigValue.duration_value:type_name -> google.protobuf.Duration
|
||||
24, // 40: plugin.ConfigValue.string_list:type_name -> plugin.StringList
|
||||
25, // 41: plugin.ConfigValue.int64_list:type_name -> plugin.Int64List
|
||||
26, // 42: plugin.ConfigValue.double_list:type_name -> plugin.DoubleList
|
||||
27, // 43: plugin.ConfigValue.bool_list:type_name -> plugin.BoolList
|
||||
28, // 44: plugin.ConfigValue.list_value:type_name -> plugin.ValueList
|
||||
29, // 45: plugin.ConfigValue.map_value:type_name -> plugin.ValueMap
|
||||
23, // 46: plugin.ValueList.values:type_name -> plugin.ConfigValue
|
||||
51, // 47: plugin.ValueMap.fields:type_name -> plugin.ValueMap.FieldsEntry
|
||||
31, // 48: plugin.RunDetectionRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig
|
||||
52, // 49: plugin.RunDetectionRequest.admin_config_values:type_name -> plugin.RunDetectionRequest.AdminConfigValuesEntry
|
||||
53, // 50: plugin.RunDetectionRequest.worker_config_values:type_name -> plugin.RunDetectionRequest.WorkerConfigValuesEntry
|
||||
41, // 51: plugin.RunDetectionRequest.cluster_context:type_name -> plugin.ClusterContext
|
||||
66, // 52: plugin.RunDetectionRequest.last_successful_run:type_name -> google.protobuf.Timestamp
|
||||
35, // 53: plugin.DetectionProposals.proposals:type_name -> plugin.JobProposal
|
||||
1, // 54: plugin.JobProposal.priority:type_name -> plugin.JobPriority
|
||||
54, // 55: plugin.JobProposal.parameters:type_name -> plugin.JobProposal.ParametersEntry
|
||||
55, // 56: plugin.JobProposal.labels:type_name -> plugin.JobProposal.LabelsEntry
|
||||
66, // 57: plugin.JobProposal.not_before:type_name -> google.protobuf.Timestamp
|
||||
66, // 58: plugin.JobProposal.expires_at:type_name -> google.protobuf.Timestamp
|
||||
37, // 59: plugin.ExecuteJobRequest.job:type_name -> plugin.JobSpec
|
||||
31, // 60: plugin.ExecuteJobRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig
|
||||
56, // 61: plugin.ExecuteJobRequest.admin_config_values:type_name -> plugin.ExecuteJobRequest.AdminConfigValuesEntry
|
||||
57, // 62: plugin.ExecuteJobRequest.worker_config_values:type_name -> plugin.ExecuteJobRequest.WorkerConfigValuesEntry
|
||||
41, // 63: plugin.ExecuteJobRequest.cluster_context:type_name -> plugin.ClusterContext
|
||||
1, // 64: plugin.JobSpec.priority:type_name -> plugin.JobPriority
|
||||
58, // 65: plugin.JobSpec.parameters:type_name -> plugin.JobSpec.ParametersEntry
|
||||
59, // 66: plugin.JobSpec.labels:type_name -> plugin.JobSpec.LabelsEntry
|
||||
66, // 67: plugin.JobSpec.created_at:type_name -> google.protobuf.Timestamp
|
||||
66, // 68: plugin.JobSpec.scheduled_at:type_name -> google.protobuf.Timestamp
|
||||
2, // 69: plugin.JobProgressUpdate.state:type_name -> plugin.JobState
|
||||
60, // 70: plugin.JobProgressUpdate.metrics:type_name -> plugin.JobProgressUpdate.MetricsEntry
|
||||
42, // 71: plugin.JobProgressUpdate.activities:type_name -> plugin.ActivityEvent
|
||||
66, // 72: plugin.JobProgressUpdate.updated_at:type_name -> google.protobuf.Timestamp
|
||||
40, // 73: plugin.JobCompleted.result:type_name -> plugin.JobResult
|
||||
42, // 74: plugin.JobCompleted.activities:type_name -> plugin.ActivityEvent
|
||||
66, // 75: plugin.JobCompleted.completed_at:type_name -> google.protobuf.Timestamp
|
||||
61, // 76: plugin.JobResult.output_values:type_name -> plugin.JobResult.OutputValuesEntry
|
||||
62, // 77: plugin.ClusterContext.metadata:type_name -> plugin.ClusterContext.MetadataEntry
|
||||
6, // 78: plugin.ActivityEvent.source:type_name -> plugin.ActivitySource
|
||||
63, // 79: plugin.ActivityEvent.details:type_name -> plugin.ActivityEvent.DetailsEntry
|
||||
66, // 80: plugin.ActivityEvent.created_at:type_name -> google.protobuf.Timestamp
|
||||
0, // 81: plugin.CancelRequest.target_kind:type_name -> plugin.WorkKind
|
||||
64, // 82: plugin.PersistedJobTypeConfig.admin_config_values:type_name -> plugin.PersistedJobTypeConfig.AdminConfigValuesEntry
|
||||
65, // 83: plugin.PersistedJobTypeConfig.worker_config_values:type_name -> plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry
|
||||
31, // 84: plugin.PersistedJobTypeConfig.admin_runtime:type_name -> plugin.AdminRuntimeConfig
|
||||
66, // 85: plugin.PersistedJobTypeConfig.updated_at:type_name -> google.protobuf.Timestamp
|
||||
23, // 86: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 87: plugin.ConfigForm.DefaultValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 88: plugin.ValueMap.FieldsEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 89: plugin.RunDetectionRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 90: plugin.RunDetectionRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 91: plugin.JobProposal.ParametersEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 92: plugin.ExecuteJobRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 93: plugin.ExecuteJobRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 94: plugin.JobSpec.ParametersEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 95: plugin.JobProgressUpdate.MetricsEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 96: plugin.JobResult.OutputValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 97: plugin.ActivityEvent.DetailsEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 98: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 99: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
7, // 100: plugin.PluginControlService.WorkerStream:input_type -> plugin.WorkerToAdminMessage
|
||||
8, // 101: plugin.PluginControlService.WorkerStream:output_type -> plugin.AdminToWorkerMessage
|
||||
101, // [101:102] is the sub-list for method output_type
|
||||
100, // [100:101] is the sub-list for method input_type
|
||||
100, // [100:100] is the sub-list for extension type_name
|
||||
100, // [100:100] is the sub-list for extension extendee
|
||||
0, // [0:100] is the sub-list for field type_name
|
||||
46, // 9: plugin.WorkerToAdminMessage.observations:type_name -> plugin.WorkerObservations
|
||||
49, // 10: plugin.WorkerToAdminMessage.object_preview_response:type_name -> plugin.ObjectPreviewResponse
|
||||
72, // 11: plugin.AdminToWorkerMessage.sent_at:type_name -> google.protobuf.Timestamp
|
||||
10, // 12: plugin.AdminToWorkerMessage.hello:type_name -> plugin.AdminHello
|
||||
15, // 13: plugin.AdminToWorkerMessage.request_config_schema:type_name -> plugin.RequestConfigSchema
|
||||
32, // 14: plugin.AdminToWorkerMessage.run_detection_request:type_name -> plugin.RunDetectionRequest
|
||||
36, // 15: plugin.AdminToWorkerMessage.execute_job_request:type_name -> plugin.ExecuteJobRequest
|
||||
43, // 16: plugin.AdminToWorkerMessage.cancel_request:type_name -> plugin.CancelRequest
|
||||
44, // 17: plugin.AdminToWorkerMessage.shutdown:type_name -> plugin.AdminShutdown
|
||||
48, // 18: plugin.AdminToWorkerMessage.request_object_preview:type_name -> plugin.RequestObjectPreview
|
||||
14, // 19: plugin.WorkerHello.capabilities:type_name -> plugin.JobTypeCapability
|
||||
51, // 20: plugin.WorkerHello.metadata:type_name -> plugin.WorkerHello.MetadataEntry
|
||||
13, // 21: plugin.WorkerHeartbeat.running_work:type_name -> plugin.RunningWork
|
||||
52, // 22: plugin.WorkerHeartbeat.queued_jobs_by_type:type_name -> plugin.WorkerHeartbeat.QueuedJobsByTypeEntry
|
||||
53, // 23: plugin.WorkerHeartbeat.metadata:type_name -> plugin.WorkerHeartbeat.MetadataEntry
|
||||
0, // 24: plugin.RunningWork.kind:type_name -> plugin.WorkKind
|
||||
2, // 25: plugin.RunningWork.state:type_name -> plugin.JobState
|
||||
17, // 26: plugin.ConfigSchemaResponse.job_type_descriptor:type_name -> plugin.JobTypeDescriptor
|
||||
18, // 27: plugin.JobTypeDescriptor.admin_config_form:type_name -> plugin.ConfigForm
|
||||
18, // 28: plugin.JobTypeDescriptor.worker_config_form:type_name -> plugin.ConfigForm
|
||||
30, // 29: plugin.JobTypeDescriptor.admin_runtime_defaults:type_name -> plugin.AdminRuntimeDefaults
|
||||
54, // 30: plugin.JobTypeDescriptor.worker_default_values:type_name -> plugin.JobTypeDescriptor.WorkerDefaultValuesEntry
|
||||
19, // 31: plugin.ConfigForm.sections:type_name -> plugin.ConfigSection
|
||||
55, // 32: plugin.ConfigForm.default_values:type_name -> plugin.ConfigForm.DefaultValuesEntry
|
||||
20, // 33: plugin.ConfigSection.fields:type_name -> plugin.ConfigField
|
||||
3, // 34: plugin.ConfigField.field_type:type_name -> plugin.ConfigFieldType
|
||||
4, // 35: plugin.ConfigField.widget:type_name -> plugin.ConfigWidget
|
||||
23, // 36: plugin.ConfigField.min_value:type_name -> plugin.ConfigValue
|
||||
23, // 37: plugin.ConfigField.max_value:type_name -> plugin.ConfigValue
|
||||
21, // 38: plugin.ConfigField.options:type_name -> plugin.ConfigOption
|
||||
22, // 39: plugin.ConfigField.validation_rules:type_name -> plugin.ValidationRule
|
||||
23, // 40: plugin.ConfigField.visible_when_equals:type_name -> plugin.ConfigValue
|
||||
5, // 41: plugin.ValidationRule.type:type_name -> plugin.ValidationRuleType
|
||||
73, // 42: plugin.ConfigValue.duration_value:type_name -> google.protobuf.Duration
|
||||
24, // 43: plugin.ConfigValue.string_list:type_name -> plugin.StringList
|
||||
25, // 44: plugin.ConfigValue.int64_list:type_name -> plugin.Int64List
|
||||
26, // 45: plugin.ConfigValue.double_list:type_name -> plugin.DoubleList
|
||||
27, // 46: plugin.ConfigValue.bool_list:type_name -> plugin.BoolList
|
||||
28, // 47: plugin.ConfigValue.list_value:type_name -> plugin.ValueList
|
||||
29, // 48: plugin.ConfigValue.map_value:type_name -> plugin.ValueMap
|
||||
23, // 49: plugin.ValueList.values:type_name -> plugin.ConfigValue
|
||||
56, // 50: plugin.ValueMap.fields:type_name -> plugin.ValueMap.FieldsEntry
|
||||
31, // 51: plugin.RunDetectionRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig
|
||||
57, // 52: plugin.RunDetectionRequest.admin_config_values:type_name -> plugin.RunDetectionRequest.AdminConfigValuesEntry
|
||||
58, // 53: plugin.RunDetectionRequest.worker_config_values:type_name -> plugin.RunDetectionRequest.WorkerConfigValuesEntry
|
||||
41, // 54: plugin.RunDetectionRequest.cluster_context:type_name -> plugin.ClusterContext
|
||||
72, // 55: plugin.RunDetectionRequest.last_successful_run:type_name -> google.protobuf.Timestamp
|
||||
35, // 56: plugin.DetectionProposals.proposals:type_name -> plugin.JobProposal
|
||||
1, // 57: plugin.JobProposal.priority:type_name -> plugin.JobPriority
|
||||
59, // 58: plugin.JobProposal.parameters:type_name -> plugin.JobProposal.ParametersEntry
|
||||
60, // 59: plugin.JobProposal.labels:type_name -> plugin.JobProposal.LabelsEntry
|
||||
72, // 60: plugin.JobProposal.not_before:type_name -> google.protobuf.Timestamp
|
||||
72, // 61: plugin.JobProposal.expires_at:type_name -> google.protobuf.Timestamp
|
||||
37, // 62: plugin.ExecuteJobRequest.job:type_name -> plugin.JobSpec
|
||||
31, // 63: plugin.ExecuteJobRequest.admin_runtime:type_name -> plugin.AdminRuntimeConfig
|
||||
61, // 64: plugin.ExecuteJobRequest.admin_config_values:type_name -> plugin.ExecuteJobRequest.AdminConfigValuesEntry
|
||||
62, // 65: plugin.ExecuteJobRequest.worker_config_values:type_name -> plugin.ExecuteJobRequest.WorkerConfigValuesEntry
|
||||
41, // 66: plugin.ExecuteJobRequest.cluster_context:type_name -> plugin.ClusterContext
|
||||
1, // 67: plugin.JobSpec.priority:type_name -> plugin.JobPriority
|
||||
63, // 68: plugin.JobSpec.parameters:type_name -> plugin.JobSpec.ParametersEntry
|
||||
64, // 69: plugin.JobSpec.labels:type_name -> plugin.JobSpec.LabelsEntry
|
||||
72, // 70: plugin.JobSpec.created_at:type_name -> google.protobuf.Timestamp
|
||||
72, // 71: plugin.JobSpec.scheduled_at:type_name -> google.protobuf.Timestamp
|
||||
2, // 72: plugin.JobProgressUpdate.state:type_name -> plugin.JobState
|
||||
65, // 73: plugin.JobProgressUpdate.metrics:type_name -> plugin.JobProgressUpdate.MetricsEntry
|
||||
42, // 74: plugin.JobProgressUpdate.activities:type_name -> plugin.ActivityEvent
|
||||
72, // 75: plugin.JobProgressUpdate.updated_at:type_name -> google.protobuf.Timestamp
|
||||
40, // 76: plugin.JobCompleted.result:type_name -> plugin.JobResult
|
||||
42, // 77: plugin.JobCompleted.activities:type_name -> plugin.ActivityEvent
|
||||
72, // 78: plugin.JobCompleted.completed_at:type_name -> google.protobuf.Timestamp
|
||||
66, // 79: plugin.JobResult.output_values:type_name -> plugin.JobResult.OutputValuesEntry
|
||||
67, // 80: plugin.ClusterContext.metadata:type_name -> plugin.ClusterContext.MetadataEntry
|
||||
6, // 81: plugin.ActivityEvent.source:type_name -> plugin.ActivitySource
|
||||
68, // 82: plugin.ActivityEvent.details:type_name -> plugin.ActivityEvent.DetailsEntry
|
||||
72, // 83: plugin.ActivityEvent.created_at:type_name -> google.protobuf.Timestamp
|
||||
0, // 84: plugin.CancelRequest.target_kind:type_name -> plugin.WorkKind
|
||||
69, // 85: plugin.PersistedJobTypeConfig.admin_config_values:type_name -> plugin.PersistedJobTypeConfig.AdminConfigValuesEntry
|
||||
70, // 86: plugin.PersistedJobTypeConfig.worker_config_values:type_name -> plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry
|
||||
31, // 87: plugin.PersistedJobTypeConfig.admin_runtime:type_name -> plugin.AdminRuntimeConfig
|
||||
72, // 88: plugin.PersistedJobTypeConfig.updated_at:type_name -> google.protobuf.Timestamp
|
||||
47, // 89: plugin.WorkerObservations.observations:type_name -> plugin.ObjectObservation
|
||||
71, // 90: plugin.ObjectObservation.attributes:type_name -> plugin.ObjectObservation.AttributesEntry
|
||||
72, // 91: plugin.ObjectObservation.observed_at:type_name -> google.protobuf.Timestamp
|
||||
50, // 92: plugin.ObjectPreviewResponse.rows:type_name -> plugin.PreviewRow
|
||||
23, // 93: plugin.JobTypeDescriptor.WorkerDefaultValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 94: plugin.ConfigForm.DefaultValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 95: plugin.ValueMap.FieldsEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 96: plugin.RunDetectionRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 97: plugin.RunDetectionRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 98: plugin.JobProposal.ParametersEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 99: plugin.ExecuteJobRequest.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 100: plugin.ExecuteJobRequest.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 101: plugin.JobSpec.ParametersEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 102: plugin.JobProgressUpdate.MetricsEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 103: plugin.JobResult.OutputValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 104: plugin.ActivityEvent.DetailsEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 105: plugin.PersistedJobTypeConfig.AdminConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 106: plugin.PersistedJobTypeConfig.WorkerConfigValuesEntry.value:type_name -> plugin.ConfigValue
|
||||
23, // 107: plugin.ObjectObservation.AttributesEntry.value:type_name -> plugin.ConfigValue
|
||||
7, // 108: plugin.PluginControlService.WorkerStream:input_type -> plugin.WorkerToAdminMessage
|
||||
8, // 109: plugin.PluginControlService.WorkerStream:output_type -> plugin.AdminToWorkerMessage
|
||||
109, // [109:110] is the sub-list for method output_type
|
||||
108, // [108:109] is the sub-list for method input_type
|
||||
108, // [108:108] is the sub-list for extension type_name
|
||||
108, // [108:108] is the sub-list for extension extendee
|
||||
0, // [0:108] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_plugin_proto_init() }
|
||||
@@ -4583,6 +5022,8 @@ func file_plugin_proto_init() {
|
||||
(*WorkerToAdminMessage_DetectionComplete)(nil),
|
||||
(*WorkerToAdminMessage_JobProgressUpdate)(nil),
|
||||
(*WorkerToAdminMessage_JobCompleted)(nil),
|
||||
(*WorkerToAdminMessage_Observations)(nil),
|
||||
(*WorkerToAdminMessage_ObjectPreviewResponse)(nil),
|
||||
}
|
||||
file_plugin_proto_msgTypes[1].OneofWrappers = []any{
|
||||
(*AdminToWorkerMessage_Hello)(nil),
|
||||
@@ -4591,6 +5032,7 @@ func file_plugin_proto_init() {
|
||||
(*AdminToWorkerMessage_ExecuteJobRequest)(nil),
|
||||
(*AdminToWorkerMessage_CancelRequest)(nil),
|
||||
(*AdminToWorkerMessage_Shutdown)(nil),
|
||||
(*AdminToWorkerMessage_RequestObjectPreview)(nil),
|
||||
}
|
||||
file_plugin_proto_msgTypes[16].OneofWrappers = []any{
|
||||
(*ConfigValue_BoolValue)(nil),
|
||||
@@ -4612,7 +5054,7 @@ func file_plugin_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_plugin_proto_rawDesc), len(file_plugin_proto_rawDesc)),
|
||||
NumEnums: 7,
|
||||
NumMessages: 59,
|
||||
NumMessages: 65,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -69,6 +69,11 @@ func (s3a *S3ApiServer) validateTableBucketObjectPath(bucket, object string) err
|
||||
return err
|
||||
}
|
||||
parts := strings.SplitN(cleanObject, "/", 4)
|
||||
// A table's marker files sit at namespace/table/<marker>, one level above
|
||||
// everything else, so they are three parts rather than four.
|
||||
if len(parts) == 3 && s3tables.IsTableMarkerFile(parts[2]) {
|
||||
return nil
|
||||
}
|
||||
if len(parts) < 4 {
|
||||
return &s3tables.IcebergLayoutError{
|
||||
Code: s3tables.ErrCodeInvalidIcebergLayout,
|
||||
|
||||
@@ -169,14 +169,38 @@ func writeManagerError(w http.ResponseWriter, err error) {
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", err.Error())
|
||||
return
|
||||
}
|
||||
// A missing table bucket means the catalog the client selected does not
|
||||
// exist, not a server fault. The storage-layer message names the resolved
|
||||
// bucket, which for a client that sent no warehouse at all is the default
|
||||
// one it never asked for, so say how to select a real table bucket.
|
||||
// Storage-layer failures are mostly the client's, not the server's. Reporting
|
||||
// a missing namespace or a name conflict as a 500 makes the catalog look
|
||||
// broken and gives the client nothing to act on.
|
||||
var tableErr *s3tables.S3TablesError
|
||||
if errors.As(err, &tableErr) && tableErr.Type == s3tables.ErrCodeNoSuchBucket {
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException",
|
||||
fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3://<table-bucket>/ or /v1/<table-bucket>/", tableErr.Message))
|
||||
if errors.As(err, &tableErr) {
|
||||
switch tableErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket:
|
||||
// The storage-layer message names the resolved bucket, which for a
|
||||
// client that sent no warehouse at all is the default one it never
|
||||
// asked for, so say how to select a real table bucket.
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException",
|
||||
fmt.Sprintf("%s: each table bucket is a separate catalog, select one with warehouse=s3://<table-bucket>/ or /v1/<table-bucket>/", tableErr.Message))
|
||||
case s3tables.ErrCodeNoSuchNamespace:
|
||||
writeError(w, http.StatusNotFound, "NoSuchNamespaceException", tableErr.Message)
|
||||
case s3tables.ErrCodeNoSuchTable:
|
||||
writeError(w, http.StatusNotFound, "NoSuchTableException", tableErr.Message)
|
||||
case s3tables.ErrCodeNoSuchView:
|
||||
writeError(w, http.StatusNotFound, "NoSuchViewException", tableErr.Message)
|
||||
case s3tables.ErrCodeNamespaceAlreadyExists, s3tables.ErrCodeBucketAlreadyExists,
|
||||
s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
|
||||
case s3tables.ErrCodeNamespaceNotEmpty, s3tables.ErrCodeBucketNotEmpty:
|
||||
writeError(w, http.StatusConflict, "AlreadyExistsException", tableErr.Message)
|
||||
case s3tables.ErrCodeConflict:
|
||||
writeError(w, http.StatusConflict, "CommitFailedException", tableErr.Message)
|
||||
case s3tables.ErrCodeAccessDenied:
|
||||
writeError(w, http.StatusForbidden, "ForbiddenException", tableErr.Message)
|
||||
case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout:
|
||||
writeError(w, http.StatusBadRequest, "BadRequestException", tableErr.Message)
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "InternalServerError", err.Error())
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// Lance Namespace error codes. The spec numbers them, so a client switches on
|
||||
// the code rather than parsing the message.
|
||||
const (
|
||||
codeUnsupported = 0
|
||||
codeNamespaceNotFound = 1
|
||||
codeNamespaceAlreadyExists = 2
|
||||
codeNamespaceNotEmpty = 3
|
||||
codeTableNotFound = 4
|
||||
codeTableAlreadyExists = 5
|
||||
codeTableVersionNotFound = 11
|
||||
codeInvalidInput = 13
|
||||
codeConcurrentModification = 14
|
||||
codePermissionDenied = 15
|
||||
codeUnauthenticated = 16
|
||||
codeInternal = 18
|
||||
)
|
||||
|
||||
type errorResponse struct {
|
||||
Error string `json:"error,omitempty"`
|
||||
Code int `json:"code"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
Instance string `json:"instance,omitempty"`
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, body interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
if body == nil {
|
||||
return
|
||||
}
|
||||
if err := json.NewEncoder(w).Encode(body); err != nil {
|
||||
glog.Warningf("lance: failed to encode response: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, r *http.Request, status, code int, message string) {
|
||||
instance := ""
|
||||
if r != nil {
|
||||
instance = r.URL.Path
|
||||
}
|
||||
writeJSON(w, status, errorResponse{Error: message, Code: code, Instance: instance})
|
||||
}
|
||||
|
||||
// writeStorageError translates an S3 Tables storage error into the Lance error
|
||||
// model. A table bucket is the first namespace level here, so a missing bucket
|
||||
// is a missing namespace rather than a missing catalog.
|
||||
func writeStorageError(w http.ResponseWriter, r *http.Request, err error) {
|
||||
var storageErr *s3tables.S3TablesError
|
||||
if !errors.As(err, &storageErr) {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
status, code := http.StatusInternalServerError, codeInternal
|
||||
switch storageErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket, s3tables.ErrCodeNoSuchNamespace:
|
||||
status, code = http.StatusNotFound, codeNamespaceNotFound
|
||||
case s3tables.ErrCodeNoSuchTable, s3tables.ErrCodeNoSuchView:
|
||||
status, code = http.StatusNotFound, codeTableNotFound
|
||||
case s3tables.ErrCodeBucketAlreadyExists, s3tables.ErrCodeNamespaceAlreadyExists:
|
||||
status, code = http.StatusConflict, codeNamespaceAlreadyExists
|
||||
case s3tables.ErrCodeTableAlreadyExists, s3tables.ErrCodeViewAlreadyExists:
|
||||
status, code = http.StatusConflict, codeTableAlreadyExists
|
||||
case s3tables.ErrCodeBucketNotEmpty, s3tables.ErrCodeNamespaceNotEmpty:
|
||||
status, code = http.StatusConflict, codeNamespaceNotEmpty
|
||||
case s3tables.ErrCodeConflict:
|
||||
status, code = http.StatusConflict, codeConcurrentModification
|
||||
case s3tables.ErrCodeAccessDenied:
|
||||
status, code = http.StatusForbidden, codePermissionDenied
|
||||
case s3tables.ErrCodeInvalidRequest, s3tables.ErrCodeInvalidIcebergLayout:
|
||||
status, code = http.StatusBadRequest, codeInvalidInput
|
||||
}
|
||||
writeError(w, r, status, code, storageErr.Message)
|
||||
}
|
||||
|
||||
// isNotFound reports whether a storage error means the object is absent, so
|
||||
// exists-style handlers can answer without a second lookup.
|
||||
func isNotFound(err error) bool {
|
||||
var storageErr *s3tables.S3TablesError
|
||||
if !errors.As(err, &storageErr) {
|
||||
return false
|
||||
}
|
||||
switch storageErr.Type {
|
||||
case s3tables.ErrCodeNoSuchBucket, s3tables.ErrCodeNoSuchNamespace,
|
||||
s3tables.ErrCodeNoSuchTable, s3tables.ErrCodeNoSuchView:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,341 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleCreateNamespace creates a table bucket for a one-part identifier and a
|
||||
// namespace inside one for anything deeper. The root cannot be created.
|
||||
func (s *Server) handleCreateNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
var req CreateNamespaceRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
if len(id) == 0 {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "the root namespace always exists and cannot be created")
|
||||
return
|
||||
}
|
||||
|
||||
mode := normalizeMode(req.Mode, modeCreate)
|
||||
switch mode {
|
||||
case modeCreate, modeExistOk, modeOverwrite:
|
||||
default:
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "mode must be Create, ExistOk or Overwrite")
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
exists, err := s.namespaceExists(r, bucket, ns)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if exists {
|
||||
switch mode {
|
||||
case modeExistOk:
|
||||
writeJSON(w, http.StatusOK, CreateNamespaceResponse{Properties: req.Properties})
|
||||
return
|
||||
case modeCreate:
|
||||
writeError(w, r, http.StatusConflict, codeNamespaceAlreadyExists, "namespace already exists")
|
||||
return
|
||||
}
|
||||
// Overwrite replaces the namespace with an empty one, so the drop has to
|
||||
// succeed first. A namespace holding tables refuses, which is the point.
|
||||
if err := s.dropNamespace(r, bucket, ns); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.createNamespace(r, bucket, ns, req.Properties); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, CreateNamespaceResponse{Properties: normalizeProperties(req.Properties)})
|
||||
}
|
||||
|
||||
// handleListNamespaces lists the children of a namespace: table buckets at the
|
||||
// root, and the next path component below that.
|
||||
func (s *Server) handleListNamespaces(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
bucket, ns := id.namespace()
|
||||
|
||||
if bucket == "" {
|
||||
var resp s3tables.ListTableBucketsResponse
|
||||
req := &s3tables.ListTableBucketsRequest{
|
||||
ContinuationToken: r.URL.Query().Get("page_token"),
|
||||
MaxBuckets: pageSize(r),
|
||||
}
|
||||
if err := s.execute(r, "ListTableBuckets", req, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
names := make([]string, 0, len(resp.TableBuckets))
|
||||
for _, b := range resp.TableBuckets {
|
||||
names = append(names, b.Name)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListNamespacesResponse{Namespaces: names, PageToken: resp.ContinuationToken})
|
||||
return
|
||||
}
|
||||
|
||||
var resp s3tables.ListNamespacesResponse
|
||||
req := &s3tables.ListNamespacesRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
ContinuationToken: r.URL.Query().Get("page_token"),
|
||||
MaxNamespaces: pageSize(r),
|
||||
}
|
||||
if len(ns) > 0 {
|
||||
req.Prefix = strings.Join(ns, ".") + "."
|
||||
}
|
||||
if err := s.execute(r, "ListNamespaces", req, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Storage namespaces are full paths; the spec wants the child name relative
|
||||
// to the parent, so take the next component and drop repeats.
|
||||
children := make([]string, 0, len(resp.Namespaces))
|
||||
seen := make(map[string]struct{}, len(resp.Namespaces))
|
||||
for _, summary := range resp.Namespaces {
|
||||
if len(summary.Namespace) <= len(ns) {
|
||||
continue
|
||||
}
|
||||
child := summary.Namespace[len(ns)]
|
||||
if _, done := seen[child]; done {
|
||||
continue
|
||||
}
|
||||
seen[child] = struct{}{}
|
||||
children = append(children, child)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListNamespacesResponse{Namespaces: children, PageToken: resp.ContinuationToken})
|
||||
}
|
||||
|
||||
// handleDescribeNamespace returns a namespace's properties.
|
||||
func (s *Server) handleDescribeNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DescribeNamespaceRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
if bucket == "" {
|
||||
writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: map[string]string{}})
|
||||
return
|
||||
}
|
||||
if len(ns) == 0 {
|
||||
var resp s3tables.GetTableBucketResponse
|
||||
if err := s.execute(r, "GetTableBucket", &s3tables.GetTableBucketRequest{TableBucketARN: bucketARN(bucket)}, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: map[string]string{}})
|
||||
return
|
||||
}
|
||||
|
||||
var resp s3tables.GetNamespaceResponse
|
||||
req2 := &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}
|
||||
if err := s.execute(r, "GetNamespace", req2, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, DescribeNamespaceResponse{Properties: normalizeProperties(resp.Properties)})
|
||||
}
|
||||
|
||||
// handleDropNamespace removes a namespace or table bucket.
|
||||
func (s *Server) handleDropNamespace(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DropNamespaceRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
if len(id) == 0 {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "the root namespace cannot be dropped")
|
||||
return
|
||||
}
|
||||
|
||||
if normalizeMode(req.Behavior, behaviorRestrict) == behaviorCascade {
|
||||
writeError(w, r, http.StatusNotImplemented, codeUnsupported,
|
||||
"cascade drop is not supported; drop the tables in the namespace first")
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
err := s.dropNamespace(r, bucket, ns)
|
||||
if err == nil {
|
||||
writeJSON(w, http.StatusOK, DropNamespaceResponse{})
|
||||
return
|
||||
}
|
||||
if isNotFound(err) {
|
||||
// Skip reports success on a missing namespace; Fail reports 400 rather
|
||||
// than 404, which is what the spec asks for on this operation alone.
|
||||
if normalizeMode(req.Mode, modeFail) == modeSkip {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
writeError(w, r, http.StatusBadRequest, codeNamespaceNotFound, "namespace does not exist")
|
||||
return
|
||||
}
|
||||
writeStorageError(w, r, err)
|
||||
}
|
||||
|
||||
// handleNamespaceExists answers with the status code and no body.
|
||||
func (s *Server) handleNamespaceExists(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req NamespaceExistsRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
|
||||
bucket, ns := id.namespace()
|
||||
if bucket == "" {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
exists, err := s.namespaceExists(r, bucket, ns)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if !exists {
|
||||
writeError(w, r, http.StatusNotFound, codeNamespaceNotFound, "namespace does not exist")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
func (s *Server) namespaceExists(r *http.Request, bucket string, ns []string) (bool, error) {
|
||||
var err error
|
||||
if len(ns) == 0 {
|
||||
var resp s3tables.GetTableBucketResponse
|
||||
err = s.execute(r, "GetTableBucket", &s3tables.GetTableBucketRequest{TableBucketARN: bucketARN(bucket)}, &resp)
|
||||
} else {
|
||||
var resp s3tables.GetNamespaceResponse
|
||||
err = s.execute(r, "GetNamespace", &s3tables.GetNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}, &resp)
|
||||
}
|
||||
if err == nil {
|
||||
return true, nil
|
||||
}
|
||||
if isNotFound(err) {
|
||||
return false, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
|
||||
func (s *Server) createNamespace(r *http.Request, bucket string, ns []string, properties map[string]string) error {
|
||||
if len(ns) == 0 {
|
||||
// A bucket made through this surface holds Lance tables. Saying so is
|
||||
// what stops it from being described to a client as an Iceberg catalog.
|
||||
var resp s3tables.CreateTableBucketResponse
|
||||
return s.execute(r, "CreateTableBucket", &s3tables.CreateTableBucketRequest{
|
||||
Name: bucket,
|
||||
Format: s3tables.FormatLance,
|
||||
}, &resp)
|
||||
}
|
||||
// A table bucket is a tenant resource with its own policy and lifecycle, so
|
||||
// it is created deliberately, never as a side effect of naming a namespace
|
||||
// inside it.
|
||||
// The immediate parent has to exist too. Storage keeps a namespace's parts
|
||||
// flattened, so creating "a.b" without "a" leaves an intermediate that
|
||||
// listing derives from the name and describe then denies exists. The spec
|
||||
// asks for NamespaceNotFound here, which also keeps the two consistent.
|
||||
if len(ns) > 1 {
|
||||
parent := ns[:len(ns)-1]
|
||||
if exists, err := s.namespaceExists(r, bucket, parent); err != nil {
|
||||
return err
|
||||
} else if !exists {
|
||||
return &s3tables.S3TablesError{
|
||||
Type: s3tables.ErrCodeNoSuchNamespace,
|
||||
Message: "parent namespace " + strings.Join(parent, ".") + " does not exist",
|
||||
}
|
||||
}
|
||||
}
|
||||
if exists, err := s.namespaceExists(r, bucket, nil); err != nil {
|
||||
return err
|
||||
} else if !exists {
|
||||
return &s3tables.S3TablesError{
|
||||
Type: s3tables.ErrCodeNoSuchBucket,
|
||||
Message: "table bucket " + bucket + " does not exist",
|
||||
}
|
||||
}
|
||||
var resp s3tables.CreateNamespaceResponse
|
||||
req := &s3tables.CreateNamespaceRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Properties: properties,
|
||||
}
|
||||
return s.execute(r, "CreateNamespace", req, &resp)
|
||||
}
|
||||
|
||||
func (s *Server) dropNamespace(r *http.Request, bucket string, ns []string) error {
|
||||
if len(ns) == 0 {
|
||||
return s.execute(r, "DeleteTableBucket", &s3tables.DeleteTableBucketRequest{TableBucketARN: bucketARN(bucket)}, nil)
|
||||
}
|
||||
req := &s3tables.DeleteNamespaceRequest{TableBucketARN: bucketARN(bucket), Namespace: ns}
|
||||
return s.execute(r, "DeleteNamespace", req, nil)
|
||||
}
|
||||
|
||||
// normalizeMode folds a spec mode or behavior value, which is case-insensitive
|
||||
// and spelled either PascalCase or snake_case, onto its lowercase form.
|
||||
func normalizeMode(value, fallback string) string {
|
||||
value = strings.ToLower(strings.ReplaceAll(strings.TrimSpace(value), "_", ""))
|
||||
if value == "" {
|
||||
return fallback
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func normalizeProperties(properties map[string]string) map[string]string {
|
||||
if properties == nil {
|
||||
return map[string]string{}
|
||||
}
|
||||
return properties
|
||||
}
|
||||
@@ -0,0 +1,566 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// handleListTables lists the Lance tables under a namespace. The spec asks for
|
||||
// full string identifiers, not bare names, so a recursive listing stays
|
||||
// unambiguous.
|
||||
func (s *Server) handleListTables(w http.ResponseWriter, r *http.Request) {
|
||||
id, delimiter, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
bucket, ns := id.namespace()
|
||||
if bucket == "" {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "listing tables needs a namespace, not the root")
|
||||
return
|
||||
}
|
||||
|
||||
req := &s3tables.ListTablesRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
ContinuationToken: r.URL.Query().Get("page_token"),
|
||||
MaxTables: pageSize(r),
|
||||
}
|
||||
var resp s3tables.ListTablesResponse
|
||||
if err := s.execute(r, "ListTables", req, &resp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
includeDeclared := true
|
||||
if raw := r.URL.Query().Get("include_declared"); raw != "" {
|
||||
includeDeclared = boolQuery(r, "include_declared")
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, ListTablesResponse{
|
||||
Tables: s.lanceTableIDs(r, bucket, resp.Tables, delimiter, includeDeclared),
|
||||
PageToken: resp.ContinuationToken,
|
||||
})
|
||||
}
|
||||
|
||||
// handleListAllTables lists every Lance table the caller can see, across every
|
||||
// table bucket.
|
||||
func (s *Server) handleListAllTables(w http.ResponseWriter, r *http.Request) {
|
||||
delimiter := requestDelimiter(r)
|
||||
|
||||
var buckets s3tables.ListTableBucketsResponse
|
||||
if err := s.execute(r, "ListTableBuckets", &s3tables.ListTableBucketsRequest{MaxBuckets: pageSize(r)}, &buckets); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
// The spec marks `tables` required, so an empty catalog answers with an
|
||||
// empty list rather than null.
|
||||
all := []string{}
|
||||
for _, bucket := range buckets.TableBuckets {
|
||||
var tables s3tables.ListTablesResponse
|
||||
req := &s3tables.ListTablesRequest{TableBucketARN: bucketARN(bucket.Name), MaxTables: pageSize(r)}
|
||||
if err := s.execute(r, "ListTables", req, &tables); err != nil {
|
||||
// One unreadable bucket must not hide the rest; a caller with access
|
||||
// to some buckets still gets those.
|
||||
glog.V(2).Infof("lance: skipping bucket %s in ListAllTables: %v", bucket.Name, err)
|
||||
continue
|
||||
}
|
||||
all = append(all, s.lanceTableIDs(r, bucket.Name, tables.Tables, delimiter, true)...)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, ListTablesResponse{Tables: all})
|
||||
}
|
||||
|
||||
// lanceTableIDs keeps the Lance tables out of a listing that also carries
|
||||
// Iceberg tables, and drops deregistered ones because they are meant to be
|
||||
// invisible until re-registered.
|
||||
func (s *Server) lanceTableIDs(r *http.Request, bucket string, summaries []s3tables.TableSummary, delimiter string, includeDeclared bool) []string {
|
||||
ids := make([]string, 0, len(summaries))
|
||||
for _, summary := range summaries {
|
||||
if summary.Format != s3tables.FormatLance {
|
||||
continue
|
||||
}
|
||||
location := summary.MetadataLocation
|
||||
if location == "" {
|
||||
location = tableLocation(bucket, summary.Namespace, summary.Name)
|
||||
}
|
||||
deregistered, hasData, err := s.datasetState(r, location)
|
||||
if err != nil {
|
||||
glog.V(2).Infof("lance: cannot read dataset state for %s: %v", location, err)
|
||||
continue
|
||||
}
|
||||
if deregistered {
|
||||
continue
|
||||
}
|
||||
if !hasData && !includeDeclared {
|
||||
continue
|
||||
}
|
||||
parts := append([]string{bucket}, summary.Namespace...)
|
||||
ids = append(ids, identifier(append(parts, summary.Name)).String(delimiter))
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// handleDeclareTable records a table that does not exist on storage yet. This
|
||||
// is what a Lance client calls on CREATE TABLE, before it writes any data.
|
||||
func (s *Server) handleDeclareTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DeclareTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
location := strings.TrimSuffix(req.Location, "/")
|
||||
if location == "" {
|
||||
location = tableLocation(bucket, ns, name)
|
||||
}
|
||||
|
||||
if err := s.createTable(r, bucket, ns, name, location); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if err := s.writeMarker(r, location, reservedMarker); err != nil {
|
||||
// The catalog entry is the authority; the marker only mirrors it for
|
||||
// clients that read the storage prefix directly.
|
||||
glog.V(1).Infof("lance: could not write %s for %s: %v", reservedMarker, location, err)
|
||||
}
|
||||
// Declaring a name that was deregistered brings it back, the same way
|
||||
// registering it does.
|
||||
if err := s.removeMarker(r, location, deregisteredMarker); err != nil {
|
||||
glog.V(2).Infof("lance: could not clear %s for %s: %v", deregisteredMarker, location, err)
|
||||
}
|
||||
|
||||
options, err := s.storageOptions(r, bucket, location, wants(r, "vend_credentials", req.VendCredentials))
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
// Properties are not persisted for a table, so neither response carries
|
||||
// them: null says "this catalog does not keep them", where echoing the
|
||||
// request back or answering {} would say they were stored and are empty.
|
||||
writeJSON(w, http.StatusOK, DeclareTableResponse{
|
||||
Location: location,
|
||||
StorageOptions: options,
|
||||
})
|
||||
}
|
||||
|
||||
// handleDescribeTable resolves a table to a location, and to credentials when
|
||||
// the caller asks for them.
|
||||
func (s *Server) handleDescribeTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DescribeTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if req.Version != nil || req.Tag != "" || req.Branch != "" {
|
||||
writeError(w, r, http.StatusNotImplemented, codeUnsupported,
|
||||
"this namespace does not resolve versions, tags or branches; the dataset owns them")
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
location := table.location
|
||||
|
||||
deregistered, hasData, err := s.datasetState(r, location)
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
if deregistered {
|
||||
writeError(w, r, http.StatusNotFound, codeTableNotFound, "table is deregistered")
|
||||
return
|
||||
}
|
||||
|
||||
options, err := s.storageOptions(r, bucket, location, wants(r, "vend_credentials", req.VendCredentials))
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
resp := DescribeTableResponse{
|
||||
Location: location,
|
||||
}
|
||||
if len(options) > 0 {
|
||||
resp.StorageOptions = options
|
||||
}
|
||||
if wants(r, "with_table_uri", req.WithTableURI) {
|
||||
resp.TableURI = location
|
||||
}
|
||||
if wants(r, "check_declared", req.CheckDeclared) {
|
||||
onlyDeclared := !hasData
|
||||
resp.IsOnlyDeclared = &onlyDeclared
|
||||
}
|
||||
if wants(r, "load_detailed_metadata", req.LoadDetailedMetadata) {
|
||||
resp.Table = name
|
||||
resp.Namespace = ns
|
||||
}
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// handleTableExists answers with the status code and no body.
|
||||
func (s *Server) handleTableExists(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req TableExistsRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
if isNotFound(err) {
|
||||
writeError(w, r, http.StatusNotFound, codeTableNotFound, "table does not exist")
|
||||
return
|
||||
}
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
deregistered, _, err := s.datasetState(r, table.location)
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, err.Error())
|
||||
return
|
||||
}
|
||||
if deregistered {
|
||||
writeError(w, r, http.StatusNotFound, codeTableNotFound, "table is deregistered")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// handleRegisterTable points a table name at an existing dataset.
|
||||
func (s *Server) handleRegisterTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req RegisterTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
location := strings.TrimSuffix(req.Location, "/")
|
||||
if location == "" {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "location is required")
|
||||
return
|
||||
}
|
||||
|
||||
mode := normalizeMode(req.Mode, modeCreate)
|
||||
if mode != modeCreate && mode != modeOverwrite {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "mode must be Create or Overwrite")
|
||||
return
|
||||
}
|
||||
|
||||
existing, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
switch {
|
||||
case err != nil && !isNotFound(err):
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
case err == nil:
|
||||
// A deregistered table is absent as far as the spec is concerned, so
|
||||
// registering over it is a re-registration rather than a conflict.
|
||||
deregistered, _, stateErr := s.datasetState(r, existing.location)
|
||||
if stateErr != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal, stateErr.Error())
|
||||
return
|
||||
}
|
||||
if !deregistered && mode == modeCreate {
|
||||
writeError(w, r, http.StatusConflict, codeTableAlreadyExists, "table already exists")
|
||||
return
|
||||
}
|
||||
if existing.location != location {
|
||||
// Repointing the name at another dataset is an update. Dropping and
|
||||
// recreating the entry would take the old dataset's files with it,
|
||||
// because the entry is the directory holding them.
|
||||
if err := s.repointTable(r, bucket, ns, name, location, existing.versionToken); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
default:
|
||||
if err := s.createTable(r, bucket, ns, name, location); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Registering a deregistered dataset brings it back.
|
||||
if err := s.removeMarker(r, location, deregisteredMarker); err != nil {
|
||||
glog.V(2).Infof("lance: could not clear %s for %s: %v", deregisteredMarker, location, err)
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, RegisterTableResponse{
|
||||
Location: location,
|
||||
Properties: normalizeProperties(req.Properties),
|
||||
})
|
||||
}
|
||||
|
||||
// handleDeregisterTable forgets a table without touching its data.
|
||||
func (s *Server) handleDeregisterTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DeregisterTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Deregistering is a state, not a deletion. Dropping the catalog entry would
|
||||
// take the dataset with it, because the entry is the dataset directory and
|
||||
// DeleteTable purges what it holds. The marker is what hides the table, and
|
||||
// it is also what a directory-catalog client reads.
|
||||
if err := s.writeMarker(r, table.location, deregisteredMarker); err != nil {
|
||||
writeError(w, r, http.StatusInternalServerError, codeInternal,
|
||||
"could not mark the table deregistered: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, DeregisterTableResponse{
|
||||
ID: append(append([]string{bucket}, ns...), name),
|
||||
Location: table.location,
|
||||
Properties: map[string]string{},
|
||||
})
|
||||
}
|
||||
|
||||
// handleDropTable removes the table and its data.
|
||||
func (s *Server) handleDropTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req DropTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
table, err := s.loadLanceTable(r, bucket, ns, name)
|
||||
if err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
if err := s.dropTableEntry(r, bucket, ns, name); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, DropTableResponse{
|
||||
ID: append(append([]string{bucket}, ns...), name),
|
||||
Location: table.location,
|
||||
Properties: map[string]string{},
|
||||
})
|
||||
}
|
||||
|
||||
// handleRenameTable moves a table's catalog entry. The dataset stays put, which
|
||||
// is what the storage layer already does for a renamed Iceberg table.
|
||||
func (s *Server) handleRenameTable(w http.ResponseWriter, r *http.Request) {
|
||||
id, _, ok := routeIdentifier(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req RenameTableRequest
|
||||
if err := decodeBody(r, &req); err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
if len(id) == 0 && len(req.ID) > 0 {
|
||||
id = req.ID
|
||||
}
|
||||
if !checkBodyIdentifier(w, r, id, req.ID) {
|
||||
return
|
||||
}
|
||||
bucket, ns, name, err := id.table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return
|
||||
}
|
||||
destBucket, destNS, destName, err := identifier(req.NewID).table()
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, "new_id: "+err.Error())
|
||||
return
|
||||
}
|
||||
if destBucket != bucket {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput,
|
||||
"a table cannot move between table buckets")
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := s.loadLanceTable(r, bucket, ns, name); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
|
||||
renameReq := &s3tables.RenameTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
SourceNamespace: ns,
|
||||
SourceName: name,
|
||||
DestNamespace: destNS,
|
||||
DestName: destName,
|
||||
}
|
||||
var renameResp s3tables.RenameTableResponse
|
||||
if err := s.execute(r, "RenameTable", renameReq, &renameResp); err != nil {
|
||||
writeStorageError(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, RenameTableResponse{})
|
||||
}
|
||||
|
||||
// lanceTable is the catalog's record of one Lance table.
|
||||
type lanceTable struct {
|
||||
location string
|
||||
versionToken string
|
||||
}
|
||||
|
||||
// loadLanceTable reads a table and refuses one that is not a Lance table, so a
|
||||
// Lance client never resolves an Iceberg table's location and writes over it.
|
||||
func (s *Server) loadLanceTable(r *http.Request, bucket string, ns []string, name string) (*lanceTable, error) {
|
||||
req := &s3tables.GetTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
}
|
||||
var resp s3tables.GetTableResponse
|
||||
if err := s.execute(r, "GetTable", req, &resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.Format != s3tables.FormatLance {
|
||||
return nil, &s3tables.S3TablesError{
|
||||
Type: s3tables.ErrCodeNoSuchTable,
|
||||
Message: "table " + name + " is not a lance table",
|
||||
}
|
||||
}
|
||||
location := resp.MetadataLocation
|
||||
if location == "" {
|
||||
location = tableLocation(bucket, resp.Namespace, resp.Name)
|
||||
}
|
||||
return &lanceTable{location: location, versionToken: resp.VersionToken}, nil
|
||||
}
|
||||
|
||||
func (s *Server) createTable(r *http.Request, bucket string, ns []string, name, location string) error {
|
||||
req := &s3tables.CreateTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
Format: s3tables.FormatLance,
|
||||
// A Lance table has no metadata file, so the location the catalog stores
|
||||
// is the dataset root itself.
|
||||
MetadataLocation: location,
|
||||
}
|
||||
var resp s3tables.CreateTableResponse
|
||||
return s.execute(r, "CreateTable", req, &resp)
|
||||
}
|
||||
|
||||
// repointTable moves an existing entry to another dataset location without
|
||||
// touching either dataset's files.
|
||||
func (s *Server) repointTable(r *http.Request, bucket string, ns []string, name, location, versionToken string) error {
|
||||
req := &s3tables.UpdateTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
VersionToken: versionToken,
|
||||
MetadataLocation: location,
|
||||
}
|
||||
var resp s3tables.UpdateTableResponse
|
||||
return s.execute(r, "UpdateTable", req, &resp)
|
||||
}
|
||||
|
||||
// dropTableEntry removes the catalog entry and the dataset under it. Only
|
||||
// DropTable wants this; deregistering and repointing must leave the files.
|
||||
func (s *Server) dropTableEntry(r *http.Request, bucket string, ns []string, name string) error {
|
||||
req := &s3tables.DeleteTableRequest{
|
||||
TableBucketARN: bucketARN(bucket),
|
||||
Namespace: ns,
|
||||
Name: name,
|
||||
}
|
||||
return s.execute(r, "DeleteTable", req, nil)
|
||||
}
|
||||
@@ -0,0 +1,359 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
)
|
||||
|
||||
// openAuthenticator stands in for a gateway with no IAM configured, which is the
|
||||
// mode the namespace falls open in.
|
||||
type openAuthenticator struct{}
|
||||
|
||||
func (openAuthenticator) AuthenticateRequest(*http.Request) (string, interface{}, s3err.ErrorCode) {
|
||||
return s3tables.DefaultAccountID, nil, s3err.ErrNone
|
||||
}
|
||||
|
||||
func (openAuthenticator) DefaultAllow() bool { return true }
|
||||
|
||||
type testHarness struct {
|
||||
router *mux.Router
|
||||
filer *s3tablestest.MemFiler
|
||||
admin *s3tables.Manager
|
||||
server *Server
|
||||
}
|
||||
|
||||
func newTestHarness(t *testing.T) *testHarness {
|
||||
t.Helper()
|
||||
filer := s3tablestest.Start(t)
|
||||
server := NewServer(s3tables.NewManagerClient(filer.Client), openAuthenticator{})
|
||||
server.SetS3Endpoint("http://127.0.0.1:8333")
|
||||
server.SetS3Region(s3tables.DefaultRegion)
|
||||
router := mux.NewRouter().SkipClean(true)
|
||||
server.RegisterRoutes(router)
|
||||
|
||||
// Table buckets are created by an operator, not by a namespace client, so
|
||||
// the harness seeds them the way the shell and admin console would.
|
||||
admin := s3tables.NewManager()
|
||||
admin.SetTrusted(true)
|
||||
return &testHarness{router: router, filer: filer, admin: admin, server: server}
|
||||
}
|
||||
|
||||
// createBucket seeds a table bucket the Lance namespace can then be pointed at.
|
||||
// An empty format strips the declaration afterwards, which is the only way to
|
||||
// get the shape a bucket made before formats existed has: one that still
|
||||
// accepts either format.
|
||||
func (h *testHarness) createBucket(t *testing.T, name, format string) {
|
||||
t.Helper()
|
||||
var resp s3tables.CreateTableBucketResponse
|
||||
declared := format
|
||||
if declared == "" {
|
||||
declared = s3tables.FormatIceberg
|
||||
}
|
||||
err := h.admin.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTableBucket",
|
||||
&s3tables.CreateTableBucketRequest{Name: name, Format: declared}, &resp, "")
|
||||
if err != nil {
|
||||
t.Fatalf("create table bucket %s: %v", name, err)
|
||||
}
|
||||
if format == "" {
|
||||
h.undeclareBucket(t, name)
|
||||
}
|
||||
}
|
||||
|
||||
// undeclareBucket removes a bucket's format from its metadata, ageing it back to
|
||||
// what the filer holds for a bucket created before the field existed.
|
||||
func (h *testHarness) undeclareBucket(t *testing.T, name string) {
|
||||
t.Helper()
|
||||
entry := h.filer.Get(s3tables.TablesPath, name)
|
||||
if entry == nil {
|
||||
t.Fatalf("table bucket %s is not in the filer", name)
|
||||
}
|
||||
var metadata map[string]any
|
||||
if err := json.Unmarshal(entry.Extended[s3tables.ExtendedKeyMetadata], &metadata); err != nil {
|
||||
t.Fatalf("read bucket metadata: %v", err)
|
||||
}
|
||||
delete(metadata, "format")
|
||||
updated, err := json.Marshal(metadata)
|
||||
if err != nil {
|
||||
t.Fatalf("write bucket metadata: %v", err)
|
||||
}
|
||||
extended := map[string][]byte{}
|
||||
for key, value := range entry.Extended {
|
||||
extended[key] = value
|
||||
}
|
||||
extended[s3tables.ExtendedKeyMetadata] = updated
|
||||
h.filer.Put(s3tables.TablesPath, name, extended)
|
||||
}
|
||||
|
||||
func (h *testHarness) bucketARN(t *testing.T, name string) string {
|
||||
t.Helper()
|
||||
arn, err := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3tables.DefaultAccountID, name)
|
||||
if err != nil {
|
||||
t.Fatalf("build arn: %v", err)
|
||||
}
|
||||
return arn
|
||||
}
|
||||
|
||||
func (h *testHarness) do(t *testing.T, method, target, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
var reader *strings.Reader
|
||||
if body == "" {
|
||||
reader = strings.NewReader("")
|
||||
} else {
|
||||
reader = strings.NewReader(body)
|
||||
}
|
||||
req := httptest.NewRequest(method, target, reader)
|
||||
recorder := httptest.NewRecorder()
|
||||
h.router.ServeHTTP(recorder, req)
|
||||
return recorder
|
||||
}
|
||||
|
||||
func (h *testHarness) mustDo(t *testing.T, method, target, body string, want int) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
recorder := h.do(t, method, target, body)
|
||||
if recorder.Code != want {
|
||||
t.Fatalf("%s %s = %d (%s), want %d", method, target, recorder.Code, recorder.Body.String(), want)
|
||||
}
|
||||
return recorder
|
||||
}
|
||||
|
||||
func decode[T any](t *testing.T, recorder *httptest.ResponseRecorder) T {
|
||||
t.Helper()
|
||||
var out T
|
||||
if err := json.Unmarshal(recorder.Body.Bytes(), &out); err != nil {
|
||||
t.Fatalf("decode %s: %v", recorder.Body.String(), err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The lifecycle a Lance client drives: create the namespace, declare the table
|
||||
// before any data exists, resolve it to a location, then deregister and bring it
|
||||
// back by registering the same location.
|
||||
func TestTableLifecycle(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
|
||||
declared := decode[DeclareTableResponse](t,
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK))
|
||||
if declared.Location != "s3://analytics/sales/orders" {
|
||||
t.Fatalf("declared location = %q", declared.Location)
|
||||
}
|
||||
// The endpoint is plaintext, so a client that does not get allow_http fails
|
||||
// with what looks like a credential error.
|
||||
if declared.StorageOptions["aws_endpoint"] != "http://127.0.0.1:8333" ||
|
||||
declared.StorageOptions["allow_http"] != "true" {
|
||||
t.Fatalf("storage options = %v", declared.StorageOptions)
|
||||
}
|
||||
|
||||
described := decode[DescribeTableResponse](t,
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe?check_declared=true", `{}`, http.StatusOK))
|
||||
if described.Location != declared.Location {
|
||||
t.Fatalf("described location = %q, want %q", described.Location, declared.Location)
|
||||
}
|
||||
if described.IsOnlyDeclared == nil || !*described.IsOnlyDeclared {
|
||||
t.Fatalf("is_only_declared = %v, want true for a table with no data", described.IsOnlyDeclared)
|
||||
}
|
||||
|
||||
listed := decode[ListTablesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
|
||||
if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$orders" {
|
||||
t.Fatalf("listed tables = %v, want the full identifier", listed.Tables)
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/deregister", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusNotFound)
|
||||
afterDrop := decode[ListTablesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
|
||||
if len(afterDrop.Tables) != 0 {
|
||||
t.Fatalf("deregistered table still listed: %v", afterDrop.Tables)
|
||||
}
|
||||
// Deregistering preserves the data. The catalog entry is the dataset
|
||||
// directory, so dropping it would take the dataset with it.
|
||||
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil {
|
||||
t.Fatal("deregister deleted the dataset directory")
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register",
|
||||
`{"location":"s3://analytics/sales/orders"}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/exists", `{}`, http.StatusOK)
|
||||
|
||||
// Dropping is the operation that does remove the data.
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/drop", `{}`, http.StatusOK)
|
||||
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") != nil {
|
||||
t.Fatal("drop left the dataset directory behind")
|
||||
}
|
||||
}
|
||||
|
||||
// Repointing a registered name at another dataset must not take the dataset it
|
||||
// used to name with it.
|
||||
func TestRegisterOverwriteKeepsTheOldDataset(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$archive/declare", `{}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/register",
|
||||
`{"location":"s3://analytics/sales/archive","mode":"Overwrite"}`, http.StatusOK)
|
||||
|
||||
described := decode[DescribeTableResponse](t,
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/describe", `{}`, http.StatusOK))
|
||||
if described.Location != "s3://analytics/sales/archive" {
|
||||
t.Fatalf("location after repointing = %q", described.Location)
|
||||
}
|
||||
if h.filer.Get(s3tables.GetNamespacePath("analytics", "sales"), "orders") == nil {
|
||||
t.Fatal("repointing deleted the dataset the name used to hold")
|
||||
}
|
||||
}
|
||||
|
||||
// A required list field answers empty rather than null, which a generated
|
||||
// client may decode differently.
|
||||
func TestListAllTablesIsNeverNull(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
|
||||
body := h.mustDo(t, http.MethodGet, "/v1/table", "", http.StatusOK).Body.String()
|
||||
if strings.Contains(body, `"tables":null`) {
|
||||
t.Fatalf("ListAllTables returned null for a required field: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNamespaceListing(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$finance/create", `{}`, http.StatusOK)
|
||||
|
||||
roots := decode[ListNamespacesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/$/list", "", http.StatusOK))
|
||||
if len(roots.Namespaces) != 1 || roots.Namespaces[0] != "analytics" {
|
||||
t.Fatalf("root listing = %v, want the table buckets", roots.Namespaces)
|
||||
}
|
||||
|
||||
children := decode[ListNamespacesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK))
|
||||
if len(children.Namespaces) != 2 {
|
||||
t.Fatalf("bucket listing = %v, want two namespaces", children.Namespaces)
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/exists", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/exists", `{}`, http.StatusNotFound)
|
||||
|
||||
// Fail mode reports a missing namespace as 400 on this operation, which is
|
||||
// what the spec asks for; Skip reports success.
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{}`, http.StatusBadRequest)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$nope/drop", `{"mode":"Skip"}`, http.StatusNoContent)
|
||||
}
|
||||
|
||||
// Storage flattens a namespace's parts, so creating a$b without a would leave
|
||||
// an intermediate that listing derives from the name and describe denies
|
||||
// exists. The spec asks for NamespaceNotFound, which keeps them consistent.
|
||||
func TestCreateNamespaceRequiresItsParent(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$missing$child/create", `{}`, http.StatusNotFound)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeNamespaceNotFound {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeNamespaceNotFound)
|
||||
}
|
||||
|
||||
// With the parent in place the child is fine, and both are then listed and
|
||||
// describable.
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$parent$child/create", `{}`, http.StatusOK)
|
||||
|
||||
listed := decode[ListNamespacesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics/list", "", http.StatusOK))
|
||||
for _, name := range listed.Namespaces {
|
||||
path := "/v1/namespace/analytics$" + name
|
||||
h.mustDo(t, http.MethodPost, path+"/exists", `{}`, http.StatusOK)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateNamespaceModes(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusConflict)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"ExistOk"}`, http.StatusOK)
|
||||
// snake_case is the other spelling the spec accepts.
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{"mode":"exist_ok"}`, http.StatusOK)
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/$/create", `{}`, http.StatusBadRequest)
|
||||
}
|
||||
|
||||
// A Lance client must never resolve an Iceberg table's location, or it would
|
||||
// write a dataset over a table another engine owns.
|
||||
func TestIcebergTablesAreInvisible(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
// Undeclared, because a bucket that declares one format cannot hold the
|
||||
// other - and mixing is exactly what this test needs to prove is hidden.
|
||||
h.createBucket(t, "analytics", "")
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$vectors/declare", `{}`, http.StatusOK)
|
||||
|
||||
manager := h.admin
|
||||
var created s3tables.CreateTableResponse
|
||||
err := manager.Execute(t.Context(), s3tables.NewManagerClient(h.filer.Client), "CreateTable", &s3tables.CreateTableRequest{
|
||||
TableBucketARN: h.bucketARN(t, "analytics"),
|
||||
Namespace: []string{"sales"},
|
||||
Name: "ledger",
|
||||
Format: s3tables.FormatIceberg,
|
||||
MetadataLocation: "s3://analytics/sales/ledger/metadata/v1.metadata.json",
|
||||
}, &created, "")
|
||||
if err != nil {
|
||||
t.Fatalf("create iceberg table: %v", err)
|
||||
}
|
||||
|
||||
listed := decode[ListTablesResponse](t,
|
||||
h.mustDo(t, http.MethodGet, "/v1/namespace/analytics$sales/table/list", "", http.StatusOK))
|
||||
if len(listed.Tables) != 1 || listed.Tables[0] != "analytics$sales$vectors" {
|
||||
t.Fatalf("listing = %v, want only the lance table", listed.Tables)
|
||||
}
|
||||
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/describe", `{}`, http.StatusNotFound)
|
||||
h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/exists", `{}`, http.StatusNotFound)
|
||||
|
||||
// Declaring over the Iceberg table must not quietly succeed and hand the
|
||||
// Lance client a directory another format owns.
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$ledger/declare", `{}`, http.StatusConflict)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeTableAlreadyExists {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeTableAlreadyExists)
|
||||
}
|
||||
}
|
||||
|
||||
// The route and the body naming different objects is a bad request, not a silent
|
||||
// preference for one of them.
|
||||
func TestRouteAndBodyMustAgree(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
h.createBucket(t, "analytics", s3tables.FormatLance)
|
||||
h.mustDo(t, http.MethodPost, "/v1/namespace/analytics$sales/create", `{}`, http.StatusOK)
|
||||
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/declare",
|
||||
`{"id":["analytics","sales","other"]}`, http.StatusBadRequest)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeInvalidInput {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeInvalidInput)
|
||||
}
|
||||
}
|
||||
|
||||
// The data plane needs Lance format support that does not exist in Go, so it
|
||||
// answers with the spec's own Unsupported code rather than a bare 404.
|
||||
func TestDataPlaneIsUnsupported(t *testing.T) {
|
||||
h := newTestHarness(t)
|
||||
recorder := h.mustDo(t, http.MethodPost, "/v1/table/analytics$sales$orders/query", `{}`, http.StatusNotImplemented)
|
||||
if got := decode[errorResponse](t, recorder); got.Code != codeUnsupported {
|
||||
t.Fatalf("error code = %d, want %d", got.Code, codeUnsupported)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// defaultDelimiter joins the parts of a Lance string identifier when the caller
|
||||
// does not pass ?delimiter=. An id equal to the delimiter is the root namespace,
|
||||
// so /v1/namespace/$/list lists the root's children.
|
||||
const defaultDelimiter = "$"
|
||||
|
||||
// identifier is a Lance object identifier: zero parts is the root namespace, one
|
||||
// part names a table bucket, and the rest are namespace parts with a table name
|
||||
// last. The mapping onto storage is bucket / namespace / table, which is the
|
||||
// three-level shape Lance clients already use.
|
||||
type identifier []string
|
||||
|
||||
func requestDelimiter(r *http.Request) string {
|
||||
if d := r.URL.Query().Get("delimiter"); d != "" {
|
||||
return d
|
||||
}
|
||||
return defaultDelimiter
|
||||
}
|
||||
|
||||
// parseIdentifier decodes the {id} route variable. Empty parts are rejected
|
||||
// rather than dropped so that "a$$b" cannot silently resolve to "a$b".
|
||||
func parseIdentifier(encoded, delimiter string) (identifier, error) {
|
||||
if encoded == "" || encoded == delimiter {
|
||||
return nil, nil
|
||||
}
|
||||
parts := strings.Split(encoded, delimiter)
|
||||
for _, part := range parts {
|
||||
if part == "" {
|
||||
return nil, fmt.Errorf("identifier %q has an empty part", encoded)
|
||||
}
|
||||
}
|
||||
return parts, nil
|
||||
}
|
||||
|
||||
func (id identifier) String(delimiter string) string {
|
||||
if len(id) == 0 {
|
||||
return delimiter
|
||||
}
|
||||
return strings.Join(id, delimiter)
|
||||
}
|
||||
|
||||
// namespace splits a namespace identifier into the table bucket and the parts of
|
||||
// the namespace inside it. The root and a bare bucket both return an empty
|
||||
// namespace, so callers check the bucket to tell them apart.
|
||||
func (id identifier) namespace() (bucket string, ns []string) {
|
||||
if len(id) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
return id[0], id[1:]
|
||||
}
|
||||
|
||||
// table splits a table identifier. A table needs a bucket, at least one
|
||||
// namespace part and a name, because storage has no unnamespaced tables.
|
||||
func (id identifier) table() (bucket string, ns []string, name string, err error) {
|
||||
if len(id) < 3 {
|
||||
return "", nil, "", fmt.Errorf("table identifier needs a bucket, a namespace and a name")
|
||||
}
|
||||
return id[0], id[1 : len(id)-1], id[len(id)-1], nil
|
||||
}
|
||||
|
||||
// matchesBody reports whether an identifier carried in the request body agrees
|
||||
// with the one in the route. The spec requires 400 when both are present and
|
||||
// differ, and requires the route to win when the body omits it.
|
||||
func (id identifier) matchesBody(body []string) bool {
|
||||
if len(body) == 0 {
|
||||
return true
|
||||
}
|
||||
if len(body) != len(id) {
|
||||
return false
|
||||
}
|
||||
for i := range body {
|
||||
if body[i] != id[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package lance
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseIdentifier(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
encoded string
|
||||
delimiter string
|
||||
want []string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "root is the delimiter", encoded: "$", delimiter: "$"},
|
||||
{name: "empty is the root", encoded: "", delimiter: "$"},
|
||||
{name: "bucket", encoded: "analytics", delimiter: "$", want: []string{"analytics"}},
|
||||
{name: "namespace", encoded: "analytics$sales", delimiter: "$", want: []string{"analytics", "sales"}},
|
||||
{name: "table", encoded: "analytics$sales$orders", delimiter: "$", want: []string{"analytics", "sales", "orders"}},
|
||||
{name: "custom delimiter", encoded: "a.b.c", delimiter: ".", want: []string{"a", "b", "c"}},
|
||||
{name: "empty part is rejected", encoded: "a$$b", delimiter: "$", wantErr: true},
|
||||
{name: "trailing delimiter is rejected", encoded: "a$", delimiter: "$", wantErr: true},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got, err := parseIdentifier(c.encoded, c.delimiter)
|
||||
if c.wantErr {
|
||||
if err == nil {
|
||||
t.Fatalf("parseIdentifier(%q) error = nil, want an error", c.encoded)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("parseIdentifier(%q) error = %v", c.encoded, err)
|
||||
}
|
||||
if len(got) != len(c.want) {
|
||||
t.Fatalf("parseIdentifier(%q) = %v, want %v", c.encoded, got, c.want)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != c.want[i] {
|
||||
t.Fatalf("parseIdentifier(%q) = %v, want %v", c.encoded, got, c.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentifierNamespace(t *testing.T) {
|
||||
bucket, ns := identifier(nil).namespace()
|
||||
if bucket != "" || len(ns) != 0 {
|
||||
t.Fatalf("root namespace() = %q %v, want empty", bucket, ns)
|
||||
}
|
||||
|
||||
bucket, ns = identifier{"analytics"}.namespace()
|
||||
if bucket != "analytics" || len(ns) != 0 {
|
||||
t.Fatalf("bucket namespace() = %q %v", bucket, ns)
|
||||
}
|
||||
|
||||
bucket, ns = identifier{"analytics", "sales", "eu"}.namespace()
|
||||
if bucket != "analytics" || len(ns) != 2 || ns[0] != "sales" || ns[1] != "eu" {
|
||||
t.Fatalf("nested namespace() = %q %v", bucket, ns)
|
||||
}
|
||||
}
|
||||
|
||||
// A table always needs a bucket, a namespace and a name: storage has no
|
||||
// unnamespaced tables, so a two-part identifier is a client error rather than a
|
||||
// table at the top of a bucket.
|
||||
func TestIdentifierTable(t *testing.T) {
|
||||
if _, _, _, err := (identifier{"analytics", "orders"}).table(); err == nil {
|
||||
t.Fatal("table() on a two-part identifier error = nil, want an error")
|
||||
}
|
||||
|
||||
bucket, ns, name, err := identifier{"analytics", "sales", "eu", "orders"}.table()
|
||||
if err != nil {
|
||||
t.Fatalf("table() error = %v", err)
|
||||
}
|
||||
if bucket != "analytics" || name != "orders" || len(ns) != 2 || ns[0] != "sales" || ns[1] != "eu" {
|
||||
t.Fatalf("table() = %q %v %q", bucket, ns, name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentifierMatchesBody(t *testing.T) {
|
||||
id := identifier{"a", "b", "c"}
|
||||
if !id.matchesBody(nil) {
|
||||
t.Fatal("an absent body identifier must defer to the route")
|
||||
}
|
||||
if !id.matchesBody([]string{"a", "b", "c"}) {
|
||||
t.Fatal("an equal body identifier must match")
|
||||
}
|
||||
if id.matchesBody([]string{"a", "b"}) {
|
||||
t.Fatal("a shorter body identifier must not match")
|
||||
}
|
||||
if id.matchesBody([]string{"a", "b", "d"}) {
|
||||
t.Fatal("a different body identifier must not match")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentifierString(t *testing.T) {
|
||||
if got := identifier(nil).String("$"); got != "$" {
|
||||
t.Fatalf("root String() = %q, want %q", got, "$")
|
||||
}
|
||||
if got := (identifier{"a", "b"}).String("$"); got != "a$b" {
|
||||
t.Fatalf("String() = %q, want %q", got, "a$b")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// Marker files the Lance Directory Catalog reads. Writing them beside the
|
||||
// dataset keeps a client that bypasses this namespace, and lists the storage
|
||||
// prefix directly, seeing the same table states the catalog reports.
|
||||
const (
|
||||
reservedMarker = ".lance-reserved"
|
||||
deregisteredMarker = ".lance-deregistered"
|
||||
versionsDir = "_versions"
|
||||
)
|
||||
|
||||
// datasetDir maps a table location onto the filer directory holding it.
|
||||
func datasetDir(location string) string {
|
||||
return s3tables.TableDataDirFromMetadataLocation(location)
|
||||
}
|
||||
|
||||
func (s *Server) entryExists(r *http.Request, dir, name string) (bool, error) {
|
||||
found := false
|
||||
err := s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
_, lookupErr := filer_pb.LookupEntry(r.Context(), client, &filer_pb.LookupDirectoryEntryRequest{
|
||||
Directory: dir,
|
||||
Name: name,
|
||||
})
|
||||
if lookupErr == filer_pb.ErrNotFound {
|
||||
return nil
|
||||
}
|
||||
if lookupErr != nil {
|
||||
return lookupErr
|
||||
}
|
||||
found = true
|
||||
return nil
|
||||
})
|
||||
return found, err
|
||||
}
|
||||
|
||||
// writeMarker drops a zero-length marker into the dataset directory. The
|
||||
// directory has to exist already: a table declared through this namespace has
|
||||
// one, and a table registered at a location that does not yet exist has no
|
||||
// storage to mark.
|
||||
func (s *Server) writeMarker(r *http.Request, location, name string) error {
|
||||
dir := datasetDir(location)
|
||||
if dir == "" {
|
||||
return nil
|
||||
}
|
||||
now := time.Now().Unix()
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return filer_pb.CreateEntry(r.Context(), client, &filer_pb.CreateEntryRequest{
|
||||
Directory: dir,
|
||||
Entry: &filer_pb.Entry{
|
||||
Name: name,
|
||||
Attributes: &filer_pb.FuseAttributes{
|
||||
Mtime: now,
|
||||
Crtime: now,
|
||||
FileMode: uint32(0644),
|
||||
},
|
||||
},
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) removeMarker(r *http.Request, location, name string) error {
|
||||
dir := datasetDir(location)
|
||||
if dir == "" {
|
||||
return nil
|
||||
}
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return filer_pb.DoRemove(r.Context(), client, dir, name, true, false, true, false, nil)
|
||||
})
|
||||
}
|
||||
|
||||
// datasetState reports the two things the spec asks about a table's storage:
|
||||
// whether it has been deregistered, and whether it holds data yet. A dataset
|
||||
// holds data once it has a version manifest, which is how a directory catalog
|
||||
// decides the same question.
|
||||
func (s *Server) datasetState(r *http.Request, location string) (deregistered, hasData bool, err error) {
|
||||
dir := datasetDir(location)
|
||||
if dir == "" {
|
||||
return false, false, nil
|
||||
}
|
||||
if deregistered, err = s.entryExists(r, dir, deregisteredMarker); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
hasData, err = s.entryExists(r, dir, versionsDir)
|
||||
return deregistered, hasData, err
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/glog"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
// FilerClient provides access to the filer for storage operations.
|
||||
type FilerClient interface {
|
||||
WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error
|
||||
}
|
||||
|
||||
type S3Authenticator interface {
|
||||
AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode)
|
||||
DefaultAllow() bool
|
||||
}
|
||||
|
||||
// VendedCredentials are short-lived S3 credentials scoped to one table.
|
||||
type VendedCredentials struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
SessionToken string
|
||||
Expiration time.Time
|
||||
}
|
||||
|
||||
// CredentialVendor mints credentials limited to a single table's prefix for a
|
||||
// caller the catalog has already authenticated and authorized. A nil result with
|
||||
// no error means the deployment has vending switched off.
|
||||
type CredentialVendor interface {
|
||||
VendTableCredentials(ctx context.Context, principal, bucket, prefix string) (*VendedCredentials, error)
|
||||
}
|
||||
|
||||
// Server implements the Lance Namespace REST spec.
|
||||
type Server struct {
|
||||
filerClient FilerClient
|
||||
tablesManager *s3tables.Manager
|
||||
authenticator S3Authenticator
|
||||
credentialVendor CredentialVendor
|
||||
s3Endpoint string
|
||||
s3Region string
|
||||
}
|
||||
|
||||
// NewServer creates a Lance namespace server over the given filer.
|
||||
func NewServer(filerClient FilerClient, authenticator S3Authenticator) *Server {
|
||||
manager := s3tables.NewManager()
|
||||
// Mirror the S3 port: fall open by default only when the gateway itself is
|
||||
// open, so an authenticated caller still passes the normal permission check.
|
||||
if authenticator != nil {
|
||||
manager.SetDefaultAllow(authenticator.DefaultAllow())
|
||||
}
|
||||
return &Server{
|
||||
filerClient: filerClient,
|
||||
tablesManager: manager,
|
||||
authenticator: authenticator,
|
||||
}
|
||||
}
|
||||
|
||||
// SetCredentialVendor enables storage_options credential vending for clients
|
||||
// that ask for it with vend_credentials.
|
||||
func (s *Server) SetCredentialVendor(vendor CredentialVendor) {
|
||||
s.credentialVendor = vendor
|
||||
}
|
||||
|
||||
// SetS3Endpoint configures the S3 endpoint advertised in storage_options so a
|
||||
// client can reach the dataset without separately discovering the S3 address.
|
||||
func (s *Server) SetS3Endpoint(endpoint string) {
|
||||
s.s3Endpoint = endpoint
|
||||
}
|
||||
|
||||
// SetS3Region configures the region advertised in storage_options.
|
||||
func (s *Server) SetS3Region(region string) {
|
||||
s.s3Region = region
|
||||
}
|
||||
|
||||
// RegisterRoutes registers the Lance Namespace REST routes.
|
||||
//
|
||||
// The spec puts the identifier in the path rather than the body so a reverse
|
||||
// proxy can route and authorize without deserializing the request.
|
||||
func (s *Server) RegisterRoutes(router *mux.Router) {
|
||||
router.Use(loggingMiddleware)
|
||||
|
||||
router.HandleFunc("/v1/namespace/{id}/create", s.Auth(s.handleCreateNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/list", s.Auth(s.handleListNamespaces)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/namespace/{id}/describe", s.Auth(s.handleDescribeNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/drop", s.Auth(s.handleDropNamespace)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/exists", s.Auth(s.handleNamespaceExists)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/namespace/{id}/table/list", s.Auth(s.handleListTables)).Methods(http.MethodGet)
|
||||
|
||||
router.HandleFunc("/v1/table", s.Auth(s.handleListAllTables)).Methods(http.MethodGet)
|
||||
router.HandleFunc("/v1/table/{id}/declare", s.Auth(s.handleDeclareTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/describe", s.Auth(s.handleDescribeTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/exists", s.Auth(s.handleTableExists)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/register", s.Auth(s.handleRegisterTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/deregister", s.Auth(s.handleDeregisterTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/drop", s.Auth(s.handleDropTable)).Methods(http.MethodPost)
|
||||
router.HandleFunc("/v1/table/{id}/rename", s.Auth(s.handleRenameTable)).Methods(http.MethodPost)
|
||||
|
||||
// The data plane needs Lance format support that does not exist in Go. Say
|
||||
// so with the spec's own code instead of returning a bare 404.
|
||||
for _, action := range []string{"create", "insert", "merge_insert", "update", "delete",
|
||||
"query", "count_rows", "explain_plan", "analyze_plan", "restore",
|
||||
"add_columns", "alter_columns", "drop_columns", "backfill_column",
|
||||
"create_index", "create_scalar_index", "stats", "schema_metadata/update"} {
|
||||
router.HandleFunc("/v1/table/{id}/"+action, s.Auth(s.handleUnsupported)).Methods(http.MethodPost)
|
||||
}
|
||||
// Version ops exist in the spec for stores that cannot order commits
|
||||
// themselves. Ours can: a Lance commit is a put-if-not-exists, and this S3
|
||||
// evaluates that precondition at the object's owner filer under a per-path
|
||||
// lock, so the dataset keeps its own version history and the catalog stays
|
||||
// out of the commit path.
|
||||
for _, action := range []string{
|
||||
"version/create", "version/list", "version/describe", "version/delete",
|
||||
"index/list", "tags/list", "tags/version", "tags/create", "tags/delete", "tags/update",
|
||||
"branches/list", "branches/create", "branches/delete"} {
|
||||
router.HandleFunc("/v1/table/{id}/"+action, s.Auth(s.handleUnsupported)).Methods(http.MethodPost)
|
||||
}
|
||||
|
||||
router.PathPrefix("/").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
glog.V(2).Infof("lance: no route for %s %s", r.Method, r.RequestURI)
|
||||
writeError(w, r, http.StatusNotFound, codeUnsupported, "no such operation")
|
||||
})
|
||||
|
||||
glog.V(2).Infof("Registered Lance Namespace routes")
|
||||
}
|
||||
|
||||
func (s *Server) handleUnsupported(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, r, http.StatusNotImplemented, codeUnsupported,
|
||||
"this namespace records table metadata only; run data operations through a Lance client against the table location")
|
||||
}
|
||||
|
||||
func loggingMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
glog.V(2).Infof("lance request: %s %s from %s", r.Method, r.RequestURI, r.RemoteAddr)
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// Auth authenticates the caller and puts the identity in the request context.
|
||||
// The Lance spec maps identity onto the same headers the S3 authenticator
|
||||
// already understands, so SigV4 and bearer tokens both keep working.
|
||||
func (s *Server) Auth(handler http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if s.authenticator == nil {
|
||||
writeError(w, r, http.StatusUnauthorized, codeUnauthenticated, "authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
identityName, identity, errCode := s.authenticator.AuthenticateRequest(r)
|
||||
if errCode != s3err.ErrNone {
|
||||
if !s.authenticator.DefaultAllow() {
|
||||
apiErr := s3err.GetAPIError(errCode)
|
||||
code := codeInternal
|
||||
switch apiErr.HTTPStatusCode {
|
||||
case http.StatusForbidden:
|
||||
code = codePermissionDenied
|
||||
case http.StatusUnauthorized:
|
||||
code = codeUnauthenticated
|
||||
case http.StatusBadRequest:
|
||||
code = codeInvalidInput
|
||||
}
|
||||
writeError(w, r, apiErr.HTTPStatusCode, code, apiErr.Description)
|
||||
return
|
||||
}
|
||||
glog.V(2).Infof("lance: authentication failed (%v) but the gateway is open, proceeding", errCode)
|
||||
}
|
||||
|
||||
if identityName != "" || identity != nil {
|
||||
ctx := r.Context()
|
||||
if identityName != "" {
|
||||
ctx = s3_constants.SetIdentityNameInContext(ctx, identityName)
|
||||
}
|
||||
if identity != nil {
|
||||
ctx = s3_constants.SetIdentityInContext(ctx, identity)
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
}
|
||||
|
||||
handler(w, r)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
package lance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
||||
)
|
||||
|
||||
const defaultPageSize = 1000
|
||||
|
||||
// execute runs one S3 Tables operation as the request's authenticated caller.
|
||||
func (s *Server) execute(r *http.Request, operation string, req, resp interface{}) error {
|
||||
identityName := s3_constants.GetIdentityNameFromContext(r)
|
||||
return s.filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return s.tablesManager.Execute(r.Context(), s3tables.NewManagerClient(client), operation, req, resp, identityName)
|
||||
})
|
||||
}
|
||||
|
||||
func bucketARN(bucket string) string {
|
||||
arn, _ := s3tables.BuildBucketARN(s3tables.DefaultRegion, s3_constants.AccountAdminId, bucket)
|
||||
return arn
|
||||
}
|
||||
|
||||
// maxRequestBody bounds what one call can make the catalog hold. Every request
|
||||
// this surface takes is a small JSON envelope; the largest carries a set of
|
||||
// properties, not data.
|
||||
const maxRequestBody = 4 << 20
|
||||
|
||||
// decodeBody reads an optional JSON request body. Every Lance operation carries
|
||||
// one, but the fields that matter are also in the route, so an empty body is
|
||||
// not an error.
|
||||
func decodeBody(r *http.Request, into interface{}) error {
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, maxRequestBody+1))
|
||||
if err != nil {
|
||||
return fmt.Errorf("read request body: %w", err)
|
||||
}
|
||||
if len(body) > maxRequestBody {
|
||||
return fmt.Errorf("request body is larger than %d bytes", maxRequestBody)
|
||||
}
|
||||
if len(strings.TrimSpace(string(body))) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := json.Unmarshal(body, into); err != nil {
|
||||
return fmt.Errorf("invalid request body: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// routeIdentifier parses the {id} route variable, writing the error response
|
||||
// itself when the identifier is malformed.
|
||||
func routeIdentifier(w http.ResponseWriter, r *http.Request) (identifier, string, bool) {
|
||||
delimiter := requestDelimiter(r)
|
||||
id, err := parseIdentifier(mux.Vars(r)["id"], delimiter)
|
||||
if err != nil {
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput, err.Error())
|
||||
return nil, delimiter, false
|
||||
}
|
||||
return id, delimiter, true
|
||||
}
|
||||
|
||||
// checkBodyIdentifier enforces the spec rule that a route and a body naming
|
||||
// different objects is a bad request rather than a silent preference.
|
||||
func checkBodyIdentifier(w http.ResponseWriter, r *http.Request, id identifier, body []string) bool {
|
||||
if id.matchesBody(body) {
|
||||
return true
|
||||
}
|
||||
writeError(w, r, http.StatusBadRequest, codeInvalidInput,
|
||||
"the identifier in the request body does not match the one in the route")
|
||||
return false
|
||||
}
|
||||
|
||||
func pageSize(r *http.Request) int {
|
||||
if raw := r.URL.Query().Get("limit"); raw != "" {
|
||||
if parsed, err := strconv.Atoi(raw); err == nil && parsed > 0 {
|
||||
return parsed
|
||||
}
|
||||
}
|
||||
return defaultPageSize
|
||||
}
|
||||
|
||||
func boolQuery(r *http.Request, name string) bool {
|
||||
value, err := strconv.ParseBool(r.URL.Query().Get(name))
|
||||
return err == nil && value
|
||||
}
|
||||
|
||||
// wants resolves a tri-state request flag: the query parameter the REST spec
|
||||
// adds, else the body field, else the implementation's own choice.
|
||||
func wants(r *http.Request, name string, body *bool) bool {
|
||||
if boolQuery(r, name) {
|
||||
return true
|
||||
}
|
||||
return body != nil && *body
|
||||
}
|
||||
|
||||
// tableLocation is where a table's dataset lives when the caller does not name
|
||||
// a location. It mirrors the Iceberg catalog's layout so both catalogs put a
|
||||
// table of the same name in the same place.
|
||||
func tableLocation(bucket string, ns []string, name string) string {
|
||||
return fmt.Sprintf("s3://%s/%s/%s", bucket, strings.Join(ns, "."), name)
|
||||
}
|
||||
|
||||
// storageOptions builds the object_store settings a Lance client needs to reach
|
||||
// the dataset. The key names are the aws_-prefixed forms Lance clients pass
|
||||
// through to object_store.
|
||||
func (s *Server) storageOptions(r *http.Request, bucket, location string, vend bool) (map[string]string, error) {
|
||||
options := map[string]string{}
|
||||
if s.s3Endpoint != "" {
|
||||
options["aws_endpoint"] = s.s3Endpoint
|
||||
if strings.HasPrefix(s.s3Endpoint, "http://") {
|
||||
// object_store refuses a plaintext endpoint unless told to allow it,
|
||||
// and the resulting failure reads like a credential problem.
|
||||
options["allow_http"] = "true"
|
||||
}
|
||||
}
|
||||
if s.s3Region != "" {
|
||||
options["aws_region"] = s.s3Region
|
||||
}
|
||||
|
||||
if !vend || s.credentialVendor == nil {
|
||||
return options, nil
|
||||
}
|
||||
|
||||
principal := s3_constants.GetIdentityNameFromContext(r)
|
||||
credentials, err := s.credentialVendor.VendTableCredentials(r.Context(), principal, bucket, locationPrefix(location))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if credentials == nil {
|
||||
return options, nil
|
||||
}
|
||||
options["aws_access_key_id"] = credentials.AccessKeyID
|
||||
options["aws_secret_access_key"] = credentials.SecretAccessKey
|
||||
if credentials.SessionToken != "" {
|
||||
options["aws_session_token"] = credentials.SessionToken
|
||||
}
|
||||
if !credentials.Expiration.IsZero() {
|
||||
options["expires_at_millis"] = strconv.FormatInt(credentials.Expiration.UnixMilli(), 10)
|
||||
}
|
||||
return options, nil
|
||||
}
|
||||
|
||||
// locationPrefix strips the s3://bucket/ part of a location, leaving the key
|
||||
// prefix a credential is scoped to.
|
||||
func locationPrefix(location string) string {
|
||||
trimmed := strings.TrimPrefix(location, "s3://")
|
||||
if _, prefix, found := strings.Cut(trimmed, "/"); found {
|
||||
return prefix
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
// Package lance serves the Lance Namespace REST spec over the same table
|
||||
// buckets the Iceberg REST catalog uses. A Lance table is a catalog entry with
|
||||
// format LANCE: the namespace records where the dataset lives and vends
|
||||
// credentials for it, and the Lance client owns everything under that location.
|
||||
package lance
|
||||
|
||||
// Modes shared by create and register. The spec matches them case-insensitively
|
||||
// and accepts both PascalCase and snake_case.
|
||||
const (
|
||||
modeCreate = "create"
|
||||
modeExistOk = "existok"
|
||||
modeOverwrite = "overwrite"
|
||||
modeFail = "fail"
|
||||
modeSkip = "skip"
|
||||
|
||||
behaviorRestrict = "restrict"
|
||||
behaviorCascade = "cascade"
|
||||
)
|
||||
|
||||
type CreateNamespaceRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type CreateNamespaceResponse struct {
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type ListNamespacesResponse struct {
|
||||
Namespaces []string `json:"namespaces"`
|
||||
PageToken string `json:"page_token,omitempty"`
|
||||
}
|
||||
|
||||
type DescribeNamespaceRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type DescribeNamespaceResponse struct {
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type DropNamespaceRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Behavior string `json:"behavior,omitempty"`
|
||||
}
|
||||
|
||||
type DropNamespaceResponse struct {
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type NamespaceExistsRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type ListTablesResponse struct {
|
||||
Tables []string `json:"tables"`
|
||||
PageToken string `json:"page_token,omitempty"`
|
||||
}
|
||||
|
||||
type DeclareTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Location string `json:"location,omitempty"`
|
||||
VendCredentials *bool `json:"vend_credentials,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type DeclareTableResponse struct {
|
||||
Location string `json:"location"`
|
||||
StorageOptions map[string]string `json:"storage_options,omitempty"`
|
||||
// Properties are null rather than {}: the catalog does not keep a table's
|
||||
// properties, and {} would claim it kept them and found none.
|
||||
Properties map[string]string `json:"properties"`
|
||||
// ManagedVersioning stays false: the dataset owns its version history,
|
||||
// because this store can order commits without the catalog in the path.
|
||||
ManagedVersioning bool `json:"managed_versioning"`
|
||||
}
|
||||
|
||||
type DescribeTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Version *int64 `json:"version,omitempty"`
|
||||
Tag string `json:"tag,omitempty"`
|
||||
Branch string `json:"branch,omitempty"`
|
||||
// The REST spec carries these as query parameters, but clients also put
|
||||
// them in the body, so honour both.
|
||||
WithTableURI *bool `json:"with_table_uri,omitempty"`
|
||||
LoadDetailedMetadata *bool `json:"load_detailed_metadata,omitempty"`
|
||||
CheckDeclared *bool `json:"check_declared,omitempty"`
|
||||
VendCredentials *bool `json:"vend_credentials,omitempty"`
|
||||
}
|
||||
|
||||
type DescribeTableResponse struct {
|
||||
Table string `json:"table,omitempty"`
|
||||
Namespace []string `json:"namespace,omitempty"`
|
||||
Version *int64 `json:"version,omitempty"`
|
||||
Location string `json:"location"`
|
||||
TableURI string `json:"table_uri,omitempty"`
|
||||
StorageOptions map[string]string `json:"storage_options,omitempty"`
|
||||
// Null, not {}: this catalog does not keep a table's properties.
|
||||
Properties map[string]string `json:"properties"`
|
||||
// ManagedVersioning stays false: the dataset owns its version history,
|
||||
// because this store can order commits without the catalog in the path.
|
||||
ManagedVersioning bool `json:"managed_versioning"`
|
||||
IsOnlyDeclared *bool `json:"is_only_declared,omitempty"`
|
||||
}
|
||||
|
||||
type TableExistsRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type RegisterTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
Location string `json:"location"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Properties map[string]string `json:"properties,omitempty"`
|
||||
}
|
||||
|
||||
type RegisterTableResponse struct {
|
||||
Location string `json:"location"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type DeregisterTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type DeregisterTableResponse struct {
|
||||
ID []string `json:"id"`
|
||||
Location string `json:"location"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type DropTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
}
|
||||
|
||||
type DropTableResponse struct {
|
||||
ID []string `json:"id"`
|
||||
Location string `json:"location"`
|
||||
Properties map[string]string `json:"properties"`
|
||||
}
|
||||
|
||||
type RenameTableRequest struct {
|
||||
ID []string `json:"id,omitempty"`
|
||||
NewID []string `json:"new_id"`
|
||||
}
|
||||
|
||||
type RenameTableResponse struct{}
|
||||
@@ -3,6 +3,7 @@ package s3tables
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -247,6 +248,31 @@ func (h *S3TablesHandler) getExtendedAttribute(ctx context.Context, client filer
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// loadNamespaceMetadata resolves a namespace to its metadata. A directory that
|
||||
// carries no namespace metadata is not a namespace, so a missing attribute
|
||||
// reports the same absence as a missing entry and every caller tests one
|
||||
// condition instead of forgetting the second.
|
||||
func (h *S3TablesHandler) loadNamespaceMetadata(ctx context.Context, filerClient FilerClient, bucketName, namespaceName string) (*namespaceMetadata, error) {
|
||||
var metadata namespaceMetadata
|
||||
err := filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(ctx, client, GetNamespacePath(bucketName, namespaceName), ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
if errors.Is(err, ErrAttributeNotFound) {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(data, &metadata); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal namespace metadata: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &metadata, nil
|
||||
}
|
||||
|
||||
// lookupEntry returns the filer entry at the given path.
|
||||
func (h *S3TablesHandler) lookupEntry(ctx context.Context, client filer_pb.SeaweedFilerClient, path string) (*filer_pb.Entry, error) {
|
||||
dir, name := splitPath(path)
|
||||
|
||||
@@ -26,6 +26,19 @@ func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http
|
||||
return err
|
||||
}
|
||||
|
||||
// A bucket is a catalog, and a catalog serves one protocol. Saying which one
|
||||
// at creation is what lets everything downstream - the endpoint the UI
|
||||
// shows, the tables the bucket accepts - be answered without opening a table.
|
||||
bucketFormat := FormatIceberg
|
||||
if req.Format != "" {
|
||||
normalized, ok := NormalizeFormat(req.Format)
|
||||
if !ok {
|
||||
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, fmt.Sprintf("unsupported format %q", req.Format))
|
||||
return fmt.Errorf("invalid format")
|
||||
}
|
||||
bucketFormat = normalized
|
||||
}
|
||||
|
||||
principal := h.getAccountID(r)
|
||||
identityActions := getIdentityActions(r)
|
||||
identityPolicyNames := getIdentityPolicyNames(r)
|
||||
@@ -115,6 +128,7 @@ func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http
|
||||
Name: req.Name,
|
||||
CreatedAt: now,
|
||||
OwnerAccountID: principal,
|
||||
Format: bucketFormat,
|
||||
}
|
||||
|
||||
metadataBytes, err := json.Marshal(metadata)
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
package s3tables
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const formatTestBucket = "formats"
|
||||
|
||||
// bucketWithFormat lays down a table bucket holding one namespace, declared as
|
||||
// the given format. An empty format is a bucket from before the declaration
|
||||
// existed.
|
||||
func bucketWithFormat(t *testing.T, format string) (*s3tablestest.MemFiler, *Manager) {
|
||||
t.Helper()
|
||||
fs := s3tablestest.Start(t)
|
||||
m := NewManager()
|
||||
|
||||
bucketMeta, _ := json.Marshal(tableBucketMetadata{
|
||||
Name: formatTestBucket,
|
||||
OwnerAccountID: DefaultAccountID,
|
||||
Format: format,
|
||||
})
|
||||
fs.Put(TablesPath, formatTestBucket, map[string][]byte{
|
||||
ExtendedKeyTableBucket: []byte("{}"),
|
||||
ExtendedKeyMetadata: bucketMeta,
|
||||
})
|
||||
|
||||
nsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"ns"}, OwnerAccountID: DefaultAccountID})
|
||||
fs.Put(GetTableBucketPath(formatTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta})
|
||||
|
||||
return fs, m
|
||||
}
|
||||
|
||||
func createTableOfFormat(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, name, format string) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "CreateTable", &CreateTableRequest{
|
||||
TableBucketARN: "arn:aws:s3tables:::bucket/" + formatTestBucket,
|
||||
Namespace: []string{"ns"},
|
||||
Name: name,
|
||||
Format: format,
|
||||
}, nil, "")
|
||||
}
|
||||
|
||||
// The declaration is the point: a bucket that says it holds Lance cannot be
|
||||
// handed an Iceberg table, because the catalog serving it would never show one.
|
||||
func TestCreateTableRefusesAForeignFormat(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, FormatLance)
|
||||
|
||||
require.NoError(t, createTableOfFormat(t, m, fs, "vectors", FormatLance))
|
||||
|
||||
err := createTableOfFormat(t, m, fs, "events", FormatIceberg)
|
||||
require.Error(t, err, "an Iceberg table in a Lance bucket should be refused")
|
||||
require.Contains(t, err.Error(), "holds LANCE")
|
||||
}
|
||||
|
||||
func TestCreateTableRefusesLanceInAnIcebergBucket(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, FormatIceberg)
|
||||
|
||||
err := createTableOfFormat(t, m, fs, "vectors", FormatLance)
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "holds ICEBERG")
|
||||
}
|
||||
|
||||
// A bucket made before formats were declared keeps taking anything. Nothing is
|
||||
// migrated, so nothing that worked stops working.
|
||||
func TestUndeclaredBucketAcceptsEitherFormat(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, "")
|
||||
|
||||
require.NoError(t, createTableOfFormat(t, m, fs, "events", FormatIceberg))
|
||||
require.NoError(t, createTableOfFormat(t, m, fs, "vectors", FormatLance))
|
||||
}
|
||||
|
||||
// A view is Iceberg metadata, so it has no meaning in a bucket of another format.
|
||||
func TestCreateViewRefusedInALanceBucket(t *testing.T) {
|
||||
fs, m := bucketWithFormat(t, FormatLance)
|
||||
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "CreateView", &CreateViewRequest{
|
||||
TableBucketARN: "arn:aws:s3tables:::bucket/" + formatTestBucket,
|
||||
Namespace: []string{"ns"},
|
||||
Name: "v",
|
||||
}, nil, "")
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "cannot hold views")
|
||||
}
|
||||
|
||||
func TestNormalizeFormat(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
want string
|
||||
ok bool
|
||||
}{
|
||||
{"ICEBERG", FormatIceberg, true},
|
||||
{"iceberg", FormatIceberg, true},
|
||||
{" Lance ", FormatLance, true},
|
||||
{"delta", "", false},
|
||||
{"", "", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := NormalizeFormat(c.in)
|
||||
if got != c.want || ok != c.ok {
|
||||
t.Errorf("NormalizeFormat(%q) = %q,%v; want %q,%v", c.in, got, ok, c.want, c.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -83,6 +83,7 @@ func (h *S3TablesHandler) handleGetTableBucket(w http.ResponseWriter, r *http.Re
|
||||
Name: metadata.Name,
|
||||
OwnerAccountID: metadata.OwnerAccountID,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
Format: metadata.Format,
|
||||
}
|
||||
|
||||
h.writeJSON(w, http.StatusOK, resp)
|
||||
@@ -202,6 +203,7 @@ func (h *S3TablesHandler) handleListTableBuckets(w http.ResponseWriter, r *http.
|
||||
ARN: bucketARN,
|
||||
Name: entry.Entry.Name,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
Format: metadata.Format,
|
||||
})
|
||||
|
||||
if len(buckets) >= maxBuckets {
|
||||
|
||||
@@ -5,13 +5,15 @@ import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func runDeleteTable(t *testing.T, m *Manager, fs *memFilerServer, namespace, name string) error {
|
||||
func runDeleteTable(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, namespace, name string) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.client), "DeleteTable", &DeleteTableRequest{
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "DeleteTable", &DeleteTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
Namespace: []string{namespace},
|
||||
Name: name,
|
||||
@@ -33,24 +35,24 @@ func TestDeleteTableDecoupledKeepsReusedNamePath(t *testing.T) {
|
||||
MetadataLocation: "s3://" + renameTestBucket + "/ns/newt-x/metadata/v1.metadata.json",
|
||||
})
|
||||
markerKeys := []string{ExtendedKeyMetadata, ExtendedKeyMetadataVersion, ExtendedKeyPolicy, ExtendedKeyTags, ExtendedKeyEntryType}
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "newt", map[string][]byte{
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "newt", map[string][]byte{
|
||||
ExtendedKeyMetadata: newtMeta,
|
||||
ExtendedKeyMetadataVersion: []byte("v1"),
|
||||
ExtendedKeyPolicy: []byte(`{"Version":"2012-10-17"}`),
|
||||
ExtendedKeyTags: []byte(`{"k":"v"}`),
|
||||
ExtendedKeyEntryType: []byte(EntryTypeTable),
|
||||
})
|
||||
fs.putEntry(GetTablePath(renameTestBucket, "ns", "newt"), "leftover", nil) // another table's data under the name path
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "newt-x", nil) // this table's own (decoupled) data
|
||||
fs.putEntry(GetTablePath(renameTestBucket, "ns", "newt-x"), "metadata", nil)
|
||||
fs.Put(GetTablePath(renameTestBucket, "ns", "newt"), "leftover", nil) // another table's data under the name path
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "newt-x", nil) // this table's own (decoupled) data
|
||||
fs.Put(GetTablePath(renameTestBucket, "ns", "newt-x"), "metadata", nil)
|
||||
|
||||
require.NoError(t, runDeleteTable(t, m, fs, "ns", "newt"))
|
||||
|
||||
assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "newt-x"),
|
||||
assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "newt-x"),
|
||||
"the table's own data location must be purged")
|
||||
assert.NotNil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "newt"), "leftover"),
|
||||
assert.NotNil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "newt"), "leftover"),
|
||||
"data under the reused name path must survive")
|
||||
marker := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "newt")
|
||||
marker := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "newt")
|
||||
require.NotNil(t, marker)
|
||||
for _, key := range markerKeys {
|
||||
_, present := marker.Extended[key]
|
||||
@@ -72,14 +74,14 @@ func TestDeleteTableRefusesAncestorDataPath(t *testing.T) {
|
||||
OwnerAccountID: DefaultAccountID,
|
||||
MetadataLocation: "s3://" + renameTestBucket + "/ns/metadata/v1.metadata.json",
|
||||
})
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "badt", map[string][]byte{ExtendedKeyMetadata: badMeta})
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "badt", map[string][]byte{ExtendedKeyMetadata: badMeta})
|
||||
|
||||
require.Error(t, runDeleteTable(t, m, fs, "ns", "badt"))
|
||||
|
||||
// The sibling table seeded by startRenameManager and its data must survive.
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"),
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"),
|
||||
"sibling table marker must survive a refused delete")
|
||||
assert.NotNil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "t"), "data"),
|
||||
assert.NotNil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "t"), "data"),
|
||||
"sibling table data must survive a refused delete")
|
||||
}
|
||||
|
||||
@@ -89,6 +91,6 @@ func TestDeleteTableColocatedRemovesData(t *testing.T) {
|
||||
|
||||
require.NoError(t, runDeleteTable(t, m, fs, "ns", "t"))
|
||||
|
||||
assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "colocated table entry must be deleted")
|
||||
assert.Nil(t, fs.getEntry(GetTablePath(renameTestBucket, "ns", "t"), "metadata"), "colocated table data must be deleted")
|
||||
assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "colocated table entry must be deleted")
|
||||
assert.Nil(t, fs.Get(GetTablePath(renameTestBucket, "ns", "t"), "metadata"), "colocated table data must be deleted")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package s3tables
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
)
|
||||
|
||||
// A directory under a table bucket that carries no namespace metadata is not a
|
||||
// namespace. Reporting that as an internal error, which is what the three
|
||||
// callers of this did while each testing only for a missing entry, turns a
|
||||
// client mistake into a 500 and hides it behind "attribute not found".
|
||||
func TestLoadNamespaceMetadataTreatsMissingAttributeAsAbsent(t *testing.T) {
|
||||
filer := s3tablestest.Start(t)
|
||||
handler := NewS3TablesHandler()
|
||||
client := NewManagerClient(filer.Client)
|
||||
ctx := context.Background()
|
||||
|
||||
filer.Put(TablesPath, "bkt", map[string][]byte{ExtendedKeyTableBucket: []byte("{}")})
|
||||
|
||||
// A stray directory where a namespace would live, with no metadata on it.
|
||||
filer.Put(GetTableBucketPath("bkt"), "stray", nil)
|
||||
|
||||
if _, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "stray"); !errors.Is(err, filer_pb.ErrNotFound) {
|
||||
t.Fatalf("loadNamespaceMetadata on a directory without metadata = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
if _, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "absent"); !errors.Is(err, filer_pb.ErrNotFound) {
|
||||
t.Fatalf("loadNamespaceMetadata on a missing directory = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
filer.Put(GetTableBucketPath("bkt"), "real", map[string][]byte{
|
||||
ExtendedKeyMetadata: []byte(`{"namespace":["real"],"ownerAccountId":"000000000000"}`),
|
||||
})
|
||||
metadata, err := handler.loadNamespaceMetadata(ctx, client, "bkt", "real")
|
||||
if err != nil {
|
||||
t.Fatalf("loadNamespaceMetadata on a real namespace: %v", err)
|
||||
}
|
||||
if metadata.OwnerAccountID != "000000000000" {
|
||||
t.Fatalf("owner = %q, want the stored one", metadata.OwnerAccountID)
|
||||
}
|
||||
}
|
||||
@@ -1,165 +1,18 @@
|
||||
package s3tables
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
// memFilerServer is an in-memory filer used to drive Manager operations
|
||||
// end-to-end without a live cluster.
|
||||
type memFilerServer struct {
|
||||
filer_pb.UnimplementedSeaweedFilerServer
|
||||
entries map[string]map[string]*filer_pb.Entry // dir -> name -> entry
|
||||
client filer_pb.SeaweedFilerClient
|
||||
// beforeUpdate runs once, at the start of the next UpdateEntry, so a test
|
||||
// can land a competing write in a handler's read-to-write window.
|
||||
beforeUpdate func()
|
||||
}
|
||||
|
||||
func newMemFilerServer() *memFilerServer {
|
||||
return &memFilerServer{entries: make(map[string]map[string]*filer_pb.Entry)}
|
||||
}
|
||||
|
||||
func (f *memFilerServer) getEntry(dir, name string) *filer_pb.Entry {
|
||||
if d, ok := f.entries[dir]; ok {
|
||||
return d[name]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) putEntry(dir, name string, extended map[string][]byte) {
|
||||
if _, ok := f.entries[dir]; !ok {
|
||||
f.entries[dir] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[dir][name] = &filer_pb.Entry{Name: name, IsDirectory: true, Extended: extended}
|
||||
}
|
||||
|
||||
func (f *memFilerServer) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
|
||||
if e := f.getEntry(req.Directory, req.Name); e != nil {
|
||||
return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
|
||||
}
|
||||
// Carry the sentinel text so filer_pb.LookupEntry maps it to ErrNotFound.
|
||||
return nil, status.Errorf(codes.NotFound, "%s: %s/%s", filer_pb.ErrNotFound.Error(), req.Directory, req.Name)
|
||||
}
|
||||
|
||||
func (f *memFilerServer) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error {
|
||||
d, ok := f.entries[req.Directory]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
names := make([]string, 0, len(d))
|
||||
for name := range d {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: d[name]}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest) (*filer_pb.CreateEntryResponse, error) {
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.CreateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) UpdateEntry(_ context.Context, req *filer_pb.UpdateEntryRequest) (*filer_pb.UpdateEntryResponse, error) {
|
||||
if hook := f.beforeUpdate; hook != nil {
|
||||
f.beforeUpdate = nil
|
||||
hook()
|
||||
}
|
||||
// The real filer validates ExpectedExtended under the per-path lock; without
|
||||
// it here a lost update would look like a success.
|
||||
for key, expected := range req.ExpectedExtended {
|
||||
var actual []byte
|
||||
if existing := f.getEntry(req.Directory, req.Entry.Name); existing != nil {
|
||||
actual = existing.Extended[key]
|
||||
}
|
||||
if !bytes.Equal(actual, expected) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "extended attribute %q changed", key)
|
||||
}
|
||||
}
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.UpdateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) DeleteEntry(_ context.Context, req *filer_pb.DeleteEntryRequest) (*filer_pb.DeleteEntryResponse, error) {
|
||||
if d, ok := f.entries[req.Directory]; ok {
|
||||
delete(d, req.Name)
|
||||
}
|
||||
// Honor recursive data deletion so a regression that wipes the table directory
|
||||
// also drops its metadata/ and data/ children (the data-loss this guards against).
|
||||
if req.IsRecursive && req.IsDeleteData {
|
||||
child := path.Join(req.Directory, req.Name)
|
||||
for dir := range f.entries {
|
||||
if dir == child || strings.HasPrefix(dir, child+"/") {
|
||||
delete(f.entries, dir)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &filer_pb.DeleteEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *memFilerServer) Ping(_ context.Context, _ *filer_pb.PingRequest) (*filer_pb.PingResponse, error) {
|
||||
now := time.Now().UnixNano()
|
||||
return &filer_pb.PingResponse{StartTimeNs: now, RemoteTimeNs: now, StopTimeNs: now}, nil
|
||||
}
|
||||
|
||||
func startMemFiler(t *testing.T) *memFilerServer {
|
||||
t.Helper()
|
||||
fs := newMemFilerServer()
|
||||
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
require.NoError(t, err)
|
||||
|
||||
server := grpc.NewServer()
|
||||
filer_pb.RegisterSeaweedFilerServer(server, fs)
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(server.GracefulStop)
|
||||
|
||||
conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
|
||||
fs.client = filer_pb.NewSeaweedFilerClient(conn)
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
pingCtx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
_, err := fs.client.Ping(pingCtx, &filer_pb.PingRequest{})
|
||||
cancel()
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
require.False(t, time.Now().After(deadline), "filer not ready: %v", err)
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return fs
|
||||
}
|
||||
|
||||
const renameTestBucket = "renamebkt"
|
||||
|
||||
func mustBucketARN(t *testing.T) string {
|
||||
@@ -170,18 +23,18 @@ func mustBucketARN(t *testing.T) string {
|
||||
}
|
||||
|
||||
// startRenameManager seeds a bucket/namespace/table and returns a trusted Manager.
|
||||
func startRenameManager(t *testing.T) (*memFilerServer, *Manager) {
|
||||
func startRenameManager(t *testing.T) (*s3tablestest.MemFiler, *Manager) {
|
||||
t.Helper()
|
||||
fs := startMemFiler(t)
|
||||
fs := s3tablestest.Start(t)
|
||||
|
||||
bucketMeta, _ := json.Marshal(tableBucketMetadata{Name: renameTestBucket, OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(TablesPath, renameTestBucket, map[string][]byte{
|
||||
fs.Put(TablesPath, renameTestBucket, map[string][]byte{
|
||||
ExtendedKeyTableBucket: []byte("{}"),
|
||||
ExtendedKeyMetadata: bucketMeta,
|
||||
})
|
||||
|
||||
nsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"ns"}, OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(GetTableBucketPath(renameTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta})
|
||||
fs.Put(GetTableBucketPath(renameTestBucket), "ns", map[string][]byte{ExtendedKeyMetadata: nsMeta})
|
||||
|
||||
tableMeta, _ := json.Marshal(tableMetadataInternal{
|
||||
Name: "t",
|
||||
@@ -191,31 +44,31 @@ func startRenameManager(t *testing.T) (*memFilerServer, *Manager) {
|
||||
MetadataVersion: 3,
|
||||
MetadataLocation: "s3://" + renameTestBucket + "/ns/t/metadata/v3.metadata.json",
|
||||
})
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "t", map[string][]byte{
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "t", map[string][]byte{
|
||||
ExtendedKeyMetadata: tableMeta,
|
||||
ExtendedKeyMetadataVersion: []byte("3"),
|
||||
})
|
||||
|
||||
// Physical metadata.json and data files live under the table directory.
|
||||
tablePath := GetTablePath(renameTestBucket, "ns", "t")
|
||||
fs.putEntry(tablePath, "metadata", nil)
|
||||
fs.putEntry(tablePath, "data", nil)
|
||||
fs.putEntry(path.Join(tablePath, "metadata"), "v3.metadata.json", nil)
|
||||
fs.Put(tablePath, "metadata", nil)
|
||||
fs.Put(tablePath, "data", nil)
|
||||
fs.Put(path.Join(tablePath, "metadata"), "v3.metadata.json", nil)
|
||||
|
||||
m := NewManager()
|
||||
m.SetTrusted(true)
|
||||
return fs, m
|
||||
}
|
||||
|
||||
func runRename(t *testing.T, m *Manager, fs *memFilerServer, req *RenameTableRequest) error {
|
||||
func runRename(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, req *RenameTableRequest) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.client), "RenameTable", req, nil, "")
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameTable", req, nil, "")
|
||||
}
|
||||
|
||||
func runGetTable(t *testing.T, m *Manager, fs *memFilerServer, namespace, name string) (*GetTableResponse, error) {
|
||||
func runGetTable(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, namespace, name string) (*GetTableResponse, error) {
|
||||
t.Helper()
|
||||
resp := &GetTableResponse{}
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "GetTable", &GetTableRequest{
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "GetTable", &GetTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
Namespace: []string{namespace},
|
||||
Name: name,
|
||||
@@ -238,12 +91,12 @@ func TestRenameTablePreservesData(t *testing.T) {
|
||||
// The source directory and its metadata.json/data children must survive: rename
|
||||
// is catalog-only and the destination still points at the original location.
|
||||
srcPath := GetTablePath(renameTestBucket, "ns", "t")
|
||||
assert.NotNil(t, fs.getEntry(srcPath, "metadata"), "source metadata dir must survive")
|
||||
assert.NotNil(t, fs.getEntry(srcPath, "data"), "source data dir must survive")
|
||||
assert.NotNil(t, fs.getEntry(path.Join(srcPath, "metadata"), "v3.metadata.json"), "metadata.json must survive")
|
||||
assert.NotNil(t, fs.Get(srcPath, "metadata"), "source metadata dir must survive")
|
||||
assert.NotNil(t, fs.Get(srcPath, "data"), "source data dir must survive")
|
||||
assert.NotNil(t, fs.Get(path.Join(srcPath, "metadata"), "v3.metadata.json"), "metadata.json must survive")
|
||||
|
||||
// Source catalog xattrs are dropped so the name stops resolving.
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, src, "source directory must remain to hold the data children")
|
||||
_, hasMeta := src.Extended[ExtendedKeyMetadata]
|
||||
assert.False(t, hasMeta, "source table-metadata xattr must be removed")
|
||||
@@ -260,7 +113,7 @@ func TestRenameTablePreservesData(t *testing.T) {
|
||||
assert.Equal(t, "t2", got.Name)
|
||||
assert.Equal(t, "s3://"+renameTestBucket+"/ns/t/metadata/v3.metadata.json", got.MetadataLocation)
|
||||
|
||||
dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
require.NotNil(t, dest)
|
||||
assert.Equal(t, []byte("3"), dest.Extended[ExtendedKeyMetadataVersion])
|
||||
}
|
||||
@@ -283,7 +136,7 @@ func TestRenameTableSourceMissing(t *testing.T) {
|
||||
func TestRenameTableDestExists(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
existing, _ := json.Marshal(tableMetadataInternal{Name: "t2", Namespace: "ns", OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), "t2", map[string][]byte{ExtendedKeyMetadata: existing})
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), "t2", map[string][]byte{ExtendedKeyMetadata: existing})
|
||||
|
||||
err := runRename(t, m, fs, &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
@@ -296,7 +149,7 @@ func TestRenameTableDestExists(t *testing.T) {
|
||||
var s3Err *S3TablesError
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeTableAlreadyExists, s3Err.Type)
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched on conflict")
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched on conflict")
|
||||
}
|
||||
|
||||
func TestRenameTableDestNamespaceMissing(t *testing.T) {
|
||||
@@ -312,7 +165,7 @@ func TestRenameTableDestNamespaceMissing(t *testing.T) {
|
||||
var s3Err *S3TablesError
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeNoSuchNamespace, s3Err.Type)
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
}
|
||||
|
||||
// A principal allowed to rename the source must still be denied when it cannot
|
||||
@@ -332,16 +185,16 @@ func TestRenameTableDestNamespaceUnauthorized(t *testing.T) {
|
||||
"Resource": "*",
|
||||
}},
|
||||
})
|
||||
srcEntry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
srcEntry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, srcEntry)
|
||||
srcEntry.Extended[ExtendedKeyPolicy] = srcPolicy
|
||||
|
||||
destNsMeta, _ := json.Marshal(namespaceMetadata{Namespace: []string{"dest"}, OwnerAccountID: DefaultAccountID})
|
||||
fs.putEntry(GetTableBucketPath(renameTestBucket), "dest", map[string][]byte{ExtendedKeyMetadata: destNsMeta})
|
||||
fs.Put(GetTableBucketPath(renameTestBucket), "dest", map[string][]byte{ExtendedKeyMetadata: destNsMeta})
|
||||
|
||||
mover := &testIdentity{Name: "mover", Account: &testIdentityAccount{Id: "mover"}}
|
||||
ctx := s3_constants.SetIdentityInContext(context.Background(), mover)
|
||||
err := m.Execute(ctx, NewManagerClient(fs.client), "RenameTable", &RenameTableRequest{
|
||||
err := m.Execute(ctx, NewManagerClient(fs.Client), "RenameTable", &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
SourceNamespace: []string{"ns"},
|
||||
SourceName: "t",
|
||||
@@ -353,8 +206,8 @@ func TestRenameTableDestNamespaceUnauthorized(t *testing.T) {
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeAccessDenied, s3Err.Type)
|
||||
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
assert.Nil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "dest"), "t2"), "destination must not be written")
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t"), "source must be untouched")
|
||||
assert.Nil(t, fs.Get(GetNamespacePath(renameTestBucket, "dest"), "t2"), "destination must not be written")
|
||||
}
|
||||
|
||||
func TestRenameTableInvalidName(t *testing.T) {
|
||||
@@ -378,7 +231,7 @@ func TestRenameTableInvalidName(t *testing.T) {
|
||||
func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, src)
|
||||
config := []byte(`{"icebergSnapshotManagement":{"status":"disabled"}}`)
|
||||
status := []byte(`{"icebergCompaction":{"status":"Successful"}}`)
|
||||
@@ -393,13 +246,13 @@ func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) {
|
||||
DestName: "t2",
|
||||
}))
|
||||
|
||||
dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t2")
|
||||
require.NotNil(t, dest)
|
||||
assert.Equal(t, config, dest.Extended[ExtendedKeyMaintenance], "the disable must move with the table")
|
||||
assert.Equal(t, status, dest.Extended[ExtendedKeyMaintenanceStatus])
|
||||
|
||||
// And must not linger on the old name, where a reused name would inherit it.
|
||||
moved := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
moved := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, moved)
|
||||
_, hasConfig := moved.Extended[ExtendedKeyMaintenance]
|
||||
assert.False(t, hasConfig, "source maintenance configuration must be cleared")
|
||||
@@ -413,7 +266,7 @@ func TestRenameTableCarriesMaintenanceConfiguration(t *testing.T) {
|
||||
func TestRenameTableRejectsConcurrentMaintenanceWrite(t *testing.T) {
|
||||
fs, _ := startRenameManager(t)
|
||||
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, src)
|
||||
copied := []byte(`{"icebergCompaction":{"status":"enabled"}}`)
|
||||
src.Extended[ExtendedKeyMaintenance] = copied
|
||||
@@ -425,7 +278,7 @@ func TestRenameTableRejectsConcurrentMaintenanceWrite(t *testing.T) {
|
||||
src.Extended[ExtendedKeyMaintenance] = landedLate
|
||||
|
||||
h := NewS3TablesHandler()
|
||||
err := NewManagerClient(fs.client).WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
err := NewManagerClient(fs.Client).WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
return h.removeExtendedAttributesIf(context.Background(),
|
||||
client, GetTablePath(renameTestBucket, "ns", "t"), expected, renamedTableAttributes...)
|
||||
})
|
||||
|
||||
@@ -45,8 +45,8 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
|
||||
// Validate format
|
||||
if req.Format != "ICEBERG" {
|
||||
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, "only ICEBERG format is supported")
|
||||
if req.Format != FormatIceberg && req.Format != FormatLance {
|
||||
h.writeError(w, http.StatusBadRequest, ErrCodeInvalidRequest, fmt.Sprintf("unsupported format %q", req.Format))
|
||||
return fmt.Errorf("invalid format")
|
||||
}
|
||||
|
||||
@@ -65,18 +65,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
|
||||
// Check if namespace exists
|
||||
namespacePath := GetNamespacePath(bucketName, namespaceName)
|
||||
var namespaceMetadata namespaceMetadata
|
||||
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(data, &namespaceMetadata); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal namespace metadata: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName)
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName))
|
||||
@@ -135,6 +124,15 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
return err
|
||||
}
|
||||
|
||||
// A bucket declares the format it holds, and a table of another format would
|
||||
// be invisible to the catalog serving it. A bucket made before the
|
||||
// declaration existed has none, and keeps taking anything.
|
||||
if bucketMetadata.Format != "" && bucketMetadata.Format != req.Format {
|
||||
message := fmt.Sprintf("table bucket %s holds %s tables", bucketName, bucketMetadata.Format)
|
||||
h.writeError(w, http.StatusConflict, ErrCodeConflict, message)
|
||||
return fmt.Errorf("%s", message)
|
||||
}
|
||||
|
||||
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
||||
identityActions := getIdentityActions(r)
|
||||
nsAllowed := CheckPermissionWithContext("CreateTable", accountID, namespaceMetadata.OwnerAccountID, namespacePolicy, bucketARN, &PolicyContext{
|
||||
@@ -174,7 +172,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) == EntryTypeView {
|
||||
if EntryType(entry.Extended) == EntryTypeView {
|
||||
existingIsView = true
|
||||
return nil
|
||||
}
|
||||
@@ -193,6 +191,14 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
|
||||
h.writeError(w, http.StatusConflict, ErrCodeTableAlreadyExists, fmt.Sprintf("a view named %s already exists", tableName))
|
||||
return fmt.Errorf("view name conflict: %s", tableName)
|
||||
}
|
||||
// Creating a table that already exists is idempotent, but only for the
|
||||
// same format. Handing a Lance client an Iceberg table's location, or the
|
||||
// reverse, has it write one format's files into the other's directory.
|
||||
if existingMetadata.Format != "" && existingMetadata.Format != req.Format {
|
||||
h.writeError(w, http.StatusConflict, ErrCodeTableAlreadyExists,
|
||||
fmt.Sprintf("a %s table named %s already exists", existingMetadata.Format, tableName))
|
||||
return fmt.Errorf("format conflict: %s", tableName)
|
||||
}
|
||||
tableARN := h.generateTableARN(existingMetadata.OwnerAccountID, bucketName, namespaceName+"/"+tableName)
|
||||
h.writeJSON(w, http.StatusOK, &CreateTableResponse{
|
||||
TableARN: tableARN,
|
||||
@@ -357,17 +363,7 @@ func (h *S3TablesHandler) handleRegisterTable(w http.ResponseWriter, r *http.Req
|
||||
|
||||
// Namespace must exist.
|
||||
namespacePath := GetNamespacePath(bucketName, namespaceName)
|
||||
var namespaceMetadata namespaceMetadata
|
||||
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := json.Unmarshal(data, &namespaceMetadata); err != nil {
|
||||
return fmt.Errorf("failed to unmarshal namespace metadata: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName)
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName))
|
||||
@@ -468,7 +464,7 @@ func (h *S3TablesHandler) handleRegisterTable(w http.ResponseWriter, r *http.Req
|
||||
metadata := &tableMetadataInternal{
|
||||
Name: tableName,
|
||||
Namespace: namespaceName,
|
||||
Format: "ICEBERG",
|
||||
Format: FormatIceberg,
|
||||
CreatedAt: now,
|
||||
ModifiedAt: now,
|
||||
OwnerAccountID: namespaceMetadata.OwnerAccountID,
|
||||
@@ -551,7 +547,7 @@ func (h *S3TablesHandler) handleGetTable(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) == EntryTypeView {
|
||||
if EntryType(entry.Extended) == EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -923,7 +919,7 @@ func (h *S3TablesHandler) listTablesWithClient(r *http.Request, client filer_pb.
|
||||
}
|
||||
|
||||
// Views share the table layout; exclude them from table listings.
|
||||
if entryType(entry.Entry.Extended) == EntryTypeView {
|
||||
if EntryType(entry.Entry.Extended) == EntryTypeView {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -945,11 +941,13 @@ func (h *S3TablesHandler) listTablesWithClient(r *http.Request, client filer_pb.
|
||||
tableARN := h.generateTableARN(metadata.OwnerAccountID, bucketName, namespaceName+"/"+entry.Entry.Name)
|
||||
|
||||
tables = append(tables, TableSummary{
|
||||
Name: entry.Entry.Name,
|
||||
TableARN: tableARN,
|
||||
Namespace: expandNamespace(namespaceName),
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
ModifiedAt: metadata.ModifiedAt,
|
||||
Name: entry.Entry.Name,
|
||||
TableARN: tableARN,
|
||||
Namespace: expandNamespace(namespaceName),
|
||||
Format: metadata.Format,
|
||||
CreatedAt: metadata.CreatedAt,
|
||||
ModifiedAt: metadata.ModifiedAt,
|
||||
MetadataLocation: metadata.MetadataLocation,
|
||||
})
|
||||
|
||||
if len(tables) >= maxTables {
|
||||
@@ -1435,7 +1433,7 @@ func (h *S3TablesHandler) renameCatalogEntry(w http.ResponseWriter, r *http.Requ
|
||||
|
||||
// Tables and views share the namespace directory, so a rename must not pick
|
||||
// up the other kind under the same name.
|
||||
if entryType(srcExtended) != kind.entryType {
|
||||
if EntryType(srcExtended) != kind.entryType {
|
||||
h.writeError(w, http.StatusNotFound, kind.notFoundCode, fmt.Sprintf("%s %s not found", kind.noun, srcName))
|
||||
return fmt.Errorf("%s %s not found", kind.noun, srcName)
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ func TestUpdateTableRejectsLostUpdate(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
winnerLocation := "s3://" + renameTestBucket + "/ns/t/metadata/v4-winner.metadata.json"
|
||||
fs.beforeUpdate = func() {
|
||||
fs.BeforeUpdate = func() {
|
||||
winner, err := json.Marshal(tableMetadataInternal{
|
||||
Name: "t",
|
||||
Namespace: "ns",
|
||||
@@ -37,13 +37,13 @@ func TestUpdateTableRejectsLostUpdate(t *testing.T) {
|
||||
VersionToken: generateVersionToken(),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, entry)
|
||||
entry.Extended[ExtendedKeyMetadata] = winner
|
||||
}
|
||||
|
||||
loserLocation := "s3://" + renameTestBucket + "/ns/t/metadata/v4-loser.metadata.json"
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable",
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable",
|
||||
updateTableRequest(t, loserLocation, 4), nil, "")
|
||||
|
||||
require.Error(t, err)
|
||||
@@ -64,14 +64,14 @@ func TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
policy := []byte(`{"Version":"2012-10-17","Statement":[]}`)
|
||||
fs.beforeUpdate = func() {
|
||||
entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
fs.BeforeUpdate = func() {
|
||||
entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, entry)
|
||||
entry.Extended[ExtendedKeyPolicy] = policy
|
||||
}
|
||||
|
||||
location := "s3://" + renameTestBucket + "/ns/t/metadata/v4.metadata.json"
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable",
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable",
|
||||
updateTableRequest(t, location, 4), nil, "")
|
||||
|
||||
require.Error(t, err)
|
||||
@@ -79,7 +79,7 @@ func TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges(t *testing.T) {
|
||||
require.ErrorAs(t, err, &s3Err)
|
||||
assert.Equal(t, ErrCodeConflict, s3Err.Type)
|
||||
|
||||
entry := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
entry := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "t")
|
||||
require.NotNil(t, entry)
|
||||
assert.Equal(t, policy, entry.Extended[ExtendedKeyPolicy], "the concurrent policy write must survive")
|
||||
}
|
||||
@@ -88,7 +88,7 @@ func TestUpdateTableAppliesWithoutContention(t *testing.T) {
|
||||
fs, m := startRenameManager(t)
|
||||
|
||||
location := "s3://" + renameTestBucket + "/ns/t/metadata/v4.metadata.json"
|
||||
require.NoError(t, m.Execute(context.Background(), NewManagerClient(fs.client), "UpdateTable",
|
||||
require.NoError(t, m.Execute(context.Background(), NewManagerClient(fs.Client), "UpdateTable",
|
||||
updateTableRequest(t, location, 4), nil, ""))
|
||||
|
||||
got, err := runGetTable(t, m, fs, "ns", "t")
|
||||
|
||||
@@ -52,14 +52,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
|
||||
// Check if namespace exists
|
||||
namespacePath := GetNamespacePath(bucketName, namespaceName)
|
||||
var namespaceMetadata namespaceMetadata
|
||||
err = filerClient.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
|
||||
data, err := h.getExtendedAttribute(r.Context(), client, namespacePath, ExtendedKeyMetadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return json.Unmarshal(data, &namespaceMetadata)
|
||||
})
|
||||
namespaceMetadata, err := h.loadNamespaceMetadata(r.Context(), filerClient, bucketName, namespaceName)
|
||||
if err != nil {
|
||||
if errors.Is(err, filer_pb.ErrNotFound) {
|
||||
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, fmt.Sprintf("namespace %s not found", namespaceName))
|
||||
@@ -77,6 +70,14 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
return err
|
||||
}
|
||||
|
||||
// A view is Iceberg metadata, so it belongs only in a bucket that holds
|
||||
// Iceberg tables.
|
||||
if bucketMetadata.Format != "" && bucketMetadata.Format != FormatIceberg {
|
||||
message := fmt.Sprintf("table bucket %s holds %s tables and cannot hold views", bucketName, bucketMetadata.Format)
|
||||
h.writeError(w, http.StatusConflict, ErrCodeConflict, message)
|
||||
return fmt.Errorf("%s", message)
|
||||
}
|
||||
|
||||
bucketARN := h.generateTableBucketARN(bucketMetadata.OwnerAccountID, bucketName)
|
||||
if !h.authorizeViewOp(r, "CreateView", accountID, namespaceMetadata.OwnerAccountID, bucketMetadata.OwnerAccountID, namespacePolicy, bucketPolicy, bucketARN, bucketName, namespaceName, viewName, bucketTags) {
|
||||
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to create view in this namespace")
|
||||
@@ -93,7 +94,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
existingIsTable = true
|
||||
return nil
|
||||
}
|
||||
@@ -125,7 +126,7 @@ func (h *S3TablesHandler) handleCreateView(w http.ResponseWriter, r *http.Reques
|
||||
metadata := &tableMetadataInternal{
|
||||
Name: viewName,
|
||||
Namespace: namespaceName,
|
||||
Format: "ICEBERG",
|
||||
Format: FormatIceberg,
|
||||
CreatedAt: now,
|
||||
ModifiedAt: now,
|
||||
OwnerAccountID: namespaceMetadata.OwnerAccountID,
|
||||
@@ -185,7 +186,7 @@ func (h *S3TablesHandler) handleGetView(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -354,7 +355,7 @@ func (h *S3TablesHandler) listViewsInNamespace(r *http.Request, client filer_pb.
|
||||
continue
|
||||
}
|
||||
// Only include view entries; skip tables and untagged entries.
|
||||
if entryType(entry.Entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Entry.Extended) != EntryTypeView {
|
||||
continue
|
||||
}
|
||||
data, ok := entry.Entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -411,7 +412,7 @@ func (h *S3TablesHandler) handleUpdateView(w http.ResponseWriter, r *http.Reques
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
@@ -516,7 +517,7 @@ func (h *S3TablesHandler) handleDeleteView(w http.ResponseWriter, r *http.Reques
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entryType(entry.Extended) != EntryTypeView {
|
||||
if EntryType(entry.Extended) != EntryTypeView {
|
||||
return filer_pb.ErrNotFound
|
||||
}
|
||||
data, ok := entry.Extended[ExtendedKeyMetadata]
|
||||
|
||||
@@ -6,12 +6,14 @@ import (
|
||||
"path"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// seedView adds a view alongside the table the rename harness already creates.
|
||||
func seedView(t *testing.T, fs *memFilerServer, name string) {
|
||||
func seedView(t *testing.T, fs *s3tablestest.MemFiler, name string) {
|
||||
t.Helper()
|
||||
|
||||
viewMeta, err := json.Marshal(tableMetadataInternal{
|
||||
@@ -24,19 +26,19 @@ func seedView(t *testing.T, fs *memFilerServer, name string) {
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
fs.putEntry(GetNamespacePath(renameTestBucket, "ns"), name, map[string][]byte{
|
||||
fs.Put(GetNamespacePath(renameTestBucket, "ns"), name, map[string][]byte{
|
||||
ExtendedKeyMetadata: viewMeta,
|
||||
ExtendedKeyMetadataVersion: []byte("1"),
|
||||
ExtendedKeyEntryType: []byte(EntryTypeView),
|
||||
})
|
||||
viewPath := GetTablePath(renameTestBucket, "ns", name)
|
||||
fs.putEntry(viewPath, "metadata", nil)
|
||||
fs.putEntry(path.Join(viewPath, "metadata"), "v1.metadata.json", nil)
|
||||
fs.Put(viewPath, "metadata", nil)
|
||||
fs.Put(path.Join(viewPath, "metadata"), "v1.metadata.json", nil)
|
||||
}
|
||||
|
||||
func runRenameView(t *testing.T, m *Manager, fs *memFilerServer, sourceName, destName string) error {
|
||||
func runRenameView(t *testing.T, m *Manager, fs *s3tablestest.MemFiler, sourceName, destName string) error {
|
||||
t.Helper()
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.client), "RenameView", &RenameTableRequest{
|
||||
return m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameView", &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
SourceNamespace: []string{"ns"},
|
||||
SourceName: sourceName,
|
||||
@@ -51,9 +53,9 @@ func TestRenameViewMovesCatalogPointer(t *testing.T) {
|
||||
|
||||
require.NoError(t, runRenameView(t, m, fs, "v", "v2"))
|
||||
|
||||
dest := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v2")
|
||||
dest := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v2")
|
||||
require.NotNil(t, dest)
|
||||
assert.Equal(t, EntryTypeView, entryType(dest.Extended), "destination must stay a view")
|
||||
assert.Equal(t, EntryTypeView, EntryType(dest.Extended), "destination must stay a view")
|
||||
|
||||
var moved tableMetadataInternal
|
||||
require.NoError(t, json.Unmarshal(dest.Extended[ExtendedKeyMetadata], &moved))
|
||||
@@ -61,11 +63,11 @@ func TestRenameViewMovesCatalogPointer(t *testing.T) {
|
||||
assert.Equal(t, "s3://"+renameTestBucket+"/ns/v/metadata/v1.metadata.json", moved.MetadataLocation,
|
||||
"rename is catalog-only, the metadata stays where it was written")
|
||||
|
||||
src := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
src := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
require.NotNil(t, src)
|
||||
_, stillListed := src.Extended[ExtendedKeyMetadata]
|
||||
assert.False(t, stillListed, "source name must stop resolving")
|
||||
assert.NotNil(t, fs.getEntry(path.Join(GetTablePath(renameTestBucket, "ns", "v"), "metadata"), "v1.metadata.json"),
|
||||
assert.NotNil(t, fs.Get(path.Join(GetTablePath(renameTestBucket, "ns", "v"), "metadata"), "v1.metadata.json"),
|
||||
"the view's metadata file must survive")
|
||||
}
|
||||
|
||||
@@ -110,18 +112,18 @@ func TestRenameViewAuthorizesAgainstTheViewARN(t *testing.T) {
|
||||
viewARN := "arn:aws:s3tables:" + DefaultRegion + ":" + DefaultAccountID + ":bucket/" + renameTestBucket + "/view/ns/v"
|
||||
viewPolicy := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"` + principal +
|
||||
`","Action":"s3tables:RenameView","Resource":"` + viewARN + `"}]}`
|
||||
view := fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
view := fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v")
|
||||
require.NotNil(t, view)
|
||||
view.Extended[ExtendedKeyPolicy] = []byte(viewPolicy)
|
||||
|
||||
// Landing in the namespace needs create permission there as well.
|
||||
namespacePolicy := `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"` + principal +
|
||||
`","Action":"s3tables:CreateView","Resource":"` + mustBucketARN(t) + `"}]}`
|
||||
namespace := fs.getEntry(GetTableBucketPath(renameTestBucket), "ns")
|
||||
namespace := fs.Get(GetTableBucketPath(renameTestBucket), "ns")
|
||||
require.NotNil(t, namespace)
|
||||
namespace.Extended[ExtendedKeyPolicy] = []byte(namespacePolicy)
|
||||
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.client), "RenameView", &RenameTableRequest{
|
||||
err := m.Execute(context.Background(), NewManagerClient(fs.Client), "RenameView", &RenameTableRequest{
|
||||
TableBucketARN: mustBucketARN(t),
|
||||
SourceNamespace: []string{"ns"},
|
||||
SourceName: "v",
|
||||
@@ -130,5 +132,5 @@ func TestRenameViewAuthorizesAgainstTheViewARN(t *testing.T) {
|
||||
}, nil, principal)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.NotNil(t, fs.getEntry(GetNamespacePath(renameTestBucket, "ns"), "v2"))
|
||||
assert.NotNil(t, fs.Get(GetNamespacePath(renameTestBucket, "ns"), "v2"))
|
||||
}
|
||||
|
||||
@@ -16,8 +16,8 @@ func TestEntryType(t *testing.T) {
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if got := entryType(c.extended); got != c.want {
|
||||
t.Fatalf("entryType() = %q, want %q", got, c.want)
|
||||
if got := EntryType(c.extended); got != c.want {
|
||||
t.Fatalf("EntryType() = %q, want %q", got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -20,12 +20,29 @@ import (
|
||||
const uuidPattern = `[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}`
|
||||
|
||||
var (
|
||||
// Allowed directories in an Iceberg table
|
||||
icebergAllowedDirs = map[string]bool{
|
||||
// Allowed directories in a table. The set is the union of what every
|
||||
// supported format writes, because this runs on the S3 door where the
|
||||
// table's format is not in hand: metadata/ and data/ for Iceberg,
|
||||
// _versions/ and _indices/ for Lance, which also puts its fragments in
|
||||
// data/.
|
||||
tableAllowedDirs = map[string]bool{
|
||||
"metadata": true,
|
||||
"data": true,
|
||||
}
|
||||
|
||||
// A format's own bookkeeping lives in underscore-prefixed directories that
|
||||
// the catalog does not interpret: Lance writes _versions, _transactions,
|
||||
// _indices and _deletions, and enumerating them here would mean guessing at
|
||||
// the next one. Iceberg writes none, so admitting them costs it nothing.
|
||||
formatOwnedDirPattern = regexp.MustCompile(`^_[a-z][a-z0-9_]*$`)
|
||||
|
||||
// Marker files a Lance table keeps at its root to record that it is
|
||||
// reserved but not yet written, or deregistered but still on storage.
|
||||
lanceMarkerFiles = map[string]bool{
|
||||
".lance-reserved": true,
|
||||
".lance-deregistered": true,
|
||||
}
|
||||
|
||||
// Patterns for valid metadata files.
|
||||
//
|
||||
// Note: Iceberg engines (Flink/Spark/Trino, plus different Iceberg versions)
|
||||
@@ -58,6 +75,7 @@ var (
|
||||
regexp.MustCompile(`^[^/]+\.parquet$`), // Parquet files
|
||||
regexp.MustCompile(`^[^/]+\.orc$`), // ORC files
|
||||
regexp.MustCompile(`^[^/]+\.avro$`), // Avro files
|
||||
regexp.MustCompile(`^[^/]+\.lance$`), // Lance fragments
|
||||
}
|
||||
|
||||
// Data file partition path pattern (e.g., year=2024/month=01/)
|
||||
@@ -91,11 +109,16 @@ func (v *IcebergLayoutValidator) ValidateFilePath(relativePath string) error {
|
||||
|
||||
topDir := parts[0]
|
||||
|
||||
// A Lance table's marker files sit at its root rather than in a directory.
|
||||
if len(parts) == 1 && lanceMarkerFiles[topDir] {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check if top-level directory is allowed
|
||||
if !icebergAllowedDirs[topDir] {
|
||||
if !tableAllowedDirs[topDir] && !formatOwnedDirPattern.MatchString(topDir) {
|
||||
return &IcebergLayoutError{
|
||||
Code: ErrCodeInvalidIcebergLayout,
|
||||
Message: "files must be placed in 'metadata/' or 'data/' directories",
|
||||
Message: "files must be placed in 'metadata/', 'data/' or a format-owned '_' directory",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -119,6 +142,28 @@ func (v *IcebergLayoutValidator) ValidateFilePath(relativePath string) error {
|
||||
return v.validateDataFile(remainingPath)
|
||||
}
|
||||
|
||||
// Everything else is a format-owned directory: the only guard is that the
|
||||
// path stays inside the table.
|
||||
return validateFileSegments(remainingPath)
|
||||
}
|
||||
|
||||
// IsTableMarkerFile reports whether a name is one of the marker files a table
|
||||
// keeps at its root.
|
||||
func IsTableMarkerFile(name string) bool {
|
||||
return lanceMarkerFiles[name]
|
||||
}
|
||||
|
||||
// validateFileSegments rejects traversal and empty segments in a path whose
|
||||
// contents the catalog does not otherwise interpret.
|
||||
func validateFileSegments(path string) error {
|
||||
for _, segment := range strings.Split(path, "/") {
|
||||
if segment == "" || segment == "." || segment == ".." {
|
||||
return &IcebergLayoutError{
|
||||
Code: ErrCodeInvalidIcebergLayout,
|
||||
Message: "invalid path segment in: " + path,
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -100,3 +100,37 @@ func TestIcebergLayoutValidator_AcceptsRealWorldDataFiles(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A Lance dataset writes fragments into data/ and keeps its own bookkeeping in
|
||||
// underscore-prefixed directories. The validator runs on the S3 door, where the
|
||||
// table's format is not in hand, so it has to admit both formats' layouts.
|
||||
func TestValidateFilePathAcceptsLanceLayout(t *testing.T) {
|
||||
v := NewIcebergLayoutValidator()
|
||||
allowed := []string{
|
||||
"data/01111110101110001101011164cadc43919eace9c608107bd9.lance",
|
||||
"_versions/1.manifest",
|
||||
"_versions/9223372036854775806.manifest",
|
||||
"_transactions/0-ddb27ab7-2e5c-42c7-b4bf-265d8a3ff636.txn",
|
||||
"_indices/85814508-ed9a-41f2-b939-2050bb7a0ed5-fts/index.idx",
|
||||
"_deletions/_deletions-1.arrow",
|
||||
".lance-reserved",
|
||||
".lance-deregistered",
|
||||
}
|
||||
for _, path := range allowed {
|
||||
if err := v.ValidateFilePath(path); err != nil {
|
||||
t.Errorf("ValidateFilePath(%q) = %v, want nil", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
rejected := []string{
|
||||
"_versions/../../escape",
|
||||
"_versions//empty",
|
||||
"notadir/file.lance",
|
||||
"random.txt",
|
||||
}
|
||||
for _, path := range rejected {
|
||||
if err := v.ValidateFilePath(path); err == nil {
|
||||
t.Errorf("ValidateFilePath(%q) = nil, want an error", path)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
// Package s3tablestest provides an in-memory filer for driving S3 Tables and
|
||||
// Lance namespace operations end-to-end without a live cluster.
|
||||
package s3tablestest
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"net"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
// MemFiler is an in-memory filer used to drive Manager operations
|
||||
// end-to-end without a live cluster.
|
||||
type MemFiler struct {
|
||||
filer_pb.UnimplementedSeaweedFilerServer
|
||||
// mu guards entries. The real filer serves concurrent RPCs, and a test that
|
||||
// races writers against each other - the point of an exclusive create - hits
|
||||
// this map from several goroutines at once.
|
||||
mu sync.RWMutex
|
||||
entries map[string]map[string]*filer_pb.Entry // dir -> name -> entry
|
||||
Client filer_pb.SeaweedFilerClient
|
||||
// BeforeUpdate runs once, at the start of the next UpdateEntry, so a test
|
||||
// can land a competing write in a handler's read-to-write window.
|
||||
BeforeUpdate func()
|
||||
}
|
||||
|
||||
func newMemFiler() *MemFiler {
|
||||
return &MemFiler{entries: make(map[string]map[string]*filer_pb.Entry)}
|
||||
}
|
||||
|
||||
func (f *MemFiler) Get(dir, name string) *filer_pb.Entry {
|
||||
f.mu.RLock()
|
||||
defer f.mu.RUnlock()
|
||||
if d, ok := f.entries[dir]; ok {
|
||||
return d[name]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) Put(dir, name string, extended map[string][]byte) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if _, ok := f.entries[dir]; !ok {
|
||||
f.entries[dir] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[dir][name] = &filer_pb.Entry{Name: name, IsDirectory: true, Extended: extended}
|
||||
}
|
||||
|
||||
// PutFile adds a file entry with an explicit modification time, which callers
|
||||
// that age entries out (orphan cleanup, expiry) need in order to see them.
|
||||
func (f *MemFiler) PutFile(dir, name string, mtime time.Time) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if _, ok := f.entries[dir]; !ok {
|
||||
f.entries[dir] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[dir][name] = &filer_pb.Entry{
|
||||
Name: name,
|
||||
Attributes: &filer_pb.FuseAttributes{Mtime: mtime.Unix(), Crtime: mtime.Unix()},
|
||||
}
|
||||
}
|
||||
|
||||
func (f *MemFiler) LookupDirectoryEntry(_ context.Context, req *filer_pb.LookupDirectoryEntryRequest) (*filer_pb.LookupDirectoryEntryResponse, error) {
|
||||
if e := f.Get(req.Directory, req.Name); e != nil {
|
||||
return &filer_pb.LookupDirectoryEntryResponse{Entry: e}, nil
|
||||
}
|
||||
// Carry the sentinel text so filer_pb.LookupEntry maps it to ErrNotFound.
|
||||
return nil, status.Errorf(codes.NotFound, "%s: %s/%s", filer_pb.ErrNotFound.Error(), req.Directory, req.Name)
|
||||
}
|
||||
|
||||
// ListEntries honours prefix, start-from and limit the way the real filer does.
|
||||
// A harness that ignores them makes a paginating caller re-read the first page
|
||||
// forever, which looks like duplicated entries rather than a broken listing.
|
||||
func (f *MemFiler) ListEntries(req *filer_pb.ListEntriesRequest, stream grpc.ServerStreamingServer[filer_pb.ListEntriesResponse]) error {
|
||||
f.mu.RLock()
|
||||
d, ok := f.entries[req.Directory]
|
||||
names := make([]string, 0, len(d))
|
||||
snapshot := make(map[string]*filer_pb.Entry, len(d))
|
||||
for name, entry := range d {
|
||||
names = append(names, name)
|
||||
snapshot[name] = entry
|
||||
}
|
||||
f.mu.RUnlock()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
|
||||
sent := uint32(0)
|
||||
for _, name := range names {
|
||||
if req.Prefix != "" && !strings.HasPrefix(name, req.Prefix) {
|
||||
continue
|
||||
}
|
||||
if req.StartFromFileName != "" {
|
||||
if name < req.StartFromFileName {
|
||||
continue
|
||||
}
|
||||
if name == req.StartFromFileName && !req.InclusiveStartFrom {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if err := stream.Send(&filer_pb.ListEntriesResponse{Entry: snapshot[name]}); err != nil {
|
||||
return err
|
||||
}
|
||||
sent++
|
||||
if req.Limit > 0 && sent >= req.Limit {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) CreateEntry(_ context.Context, req *filer_pb.CreateEntryRequest) (*filer_pb.CreateEntryResponse, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
// O_EXCL is the filer's put-if-not-exists. Ignoring it here would let a test
|
||||
// that races two writers see both of them win.
|
||||
if _, exists := f.entries[req.Directory][req.Entry.Name]; exists && req.OExcl {
|
||||
return &filer_pb.CreateEntryResponse{ErrorCode: filer_pb.FilerError_ENTRY_ALREADY_EXISTS}, nil
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.CreateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) UpdateEntry(_ context.Context, req *filer_pb.UpdateEntryRequest) (*filer_pb.UpdateEntryResponse, error) {
|
||||
// The hook runs before the lock is taken: its whole purpose is to land a
|
||||
// competing write in the read-to-write window, and that write needs the lock.
|
||||
if hook := f.BeforeUpdate; hook != nil {
|
||||
f.BeforeUpdate = nil
|
||||
hook()
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
// The real filer validates ExpectedExtended under the per-path lock; without
|
||||
// it here a lost update would look like a success.
|
||||
for key, expected := range req.ExpectedExtended {
|
||||
var actual []byte
|
||||
if d, ok := f.entries[req.Directory]; ok {
|
||||
if existing := d[req.Entry.Name]; existing != nil {
|
||||
actual = existing.Extended[key]
|
||||
}
|
||||
}
|
||||
if !bytes.Equal(actual, expected) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "extended attribute %q changed", key)
|
||||
}
|
||||
}
|
||||
if _, ok := f.entries[req.Directory]; !ok {
|
||||
f.entries[req.Directory] = make(map[string]*filer_pb.Entry)
|
||||
}
|
||||
f.entries[req.Directory][req.Entry.Name] = req.Entry
|
||||
return &filer_pb.UpdateEntryResponse{}, nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) DeleteEntry(_ context.Context, req *filer_pb.DeleteEntryRequest) (*filer_pb.DeleteEntryResponse, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if d, ok := f.entries[req.Directory]; ok {
|
||||
delete(d, req.Name)
|
||||
}
|
||||
// Honor recursive data deletion so a regression that wipes the table directory
|
||||
// also drops its metadata/ and data/ children (the data-loss this guards against).
|
||||
if req.IsRecursive && req.IsDeleteData {
|
||||
child := path.Join(req.Directory, req.Name)
|
||||
for dir := range f.entries {
|
||||
if dir == child || strings.HasPrefix(dir, child+"/") {
|
||||
delete(f.entries, dir)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &filer_pb.DeleteEntryResponse{}, nil
|
||||
}
|
||||
|
||||
// GetFilerConfiguration answers with the defaults, so operations that resolve
|
||||
// the buckets directory before touching an entry work against this filer.
|
||||
func (f *MemFiler) GetFilerConfiguration(_ context.Context, _ *filer_pb.GetFilerConfigurationRequest) (*filer_pb.GetFilerConfigurationResponse, error) {
|
||||
return &filer_pb.GetFilerConfigurationResponse{DirBuckets: s3_constants.DefaultBucketsPath}, nil
|
||||
}
|
||||
|
||||
func (f *MemFiler) Ping(_ context.Context, _ *filer_pb.PingRequest) (*filer_pb.PingResponse, error) {
|
||||
now := time.Now().UnixNano()
|
||||
return &filer_pb.PingResponse{StartTimeNs: now, RemoteTimeNs: now, StopTimeNs: now}, nil
|
||||
}
|
||||
|
||||
func Start(t *testing.T) *MemFiler {
|
||||
t.Helper()
|
||||
fs := newMemFiler()
|
||||
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("start filer: %v", err)
|
||||
}
|
||||
|
||||
server := grpc.NewServer()
|
||||
filer_pb.RegisterSeaweedFilerServer(server, fs)
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(server.GracefulStop)
|
||||
|
||||
conn, err := grpc.NewClient(listener.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||
if err != nil {
|
||||
t.Fatalf("start filer: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
|
||||
fs.Client = filer_pb.NewSeaweedFilerClient(conn)
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
pingCtx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
_, err := fs.Client.Ping(pingCtx, &filer_pb.PingRequest{})
|
||||
cancel()
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("filer not ready: %v", err)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return fs
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package s3tables
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -12,11 +13,17 @@ type TableBucket struct {
|
||||
Name string `json:"name"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type CreateTableBucketRequest struct {
|
||||
Name string `json:"name"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
// Format is the table format this bucket holds. A bucket is a catalog and a
|
||||
// catalog serves one protocol, so declaring it here is what lets a caller be
|
||||
// told where to connect. Empty means ICEBERG, which is what AWS S3 Tables
|
||||
// serves and therefore what an SDK that has never heard of this field means.
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type CreateTableBucketResponse struct {
|
||||
@@ -32,6 +39,9 @@ type GetTableBucketResponse struct {
|
||||
Name string `json:"name"`
|
||||
OwnerAccountID string `json:"ownerAccountId"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
// Format is empty for a bucket created before formats were declared. Such a
|
||||
// bucket accepts any format, which is what it did when it was made.
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type ListTableBucketsRequest struct {
|
||||
@@ -44,6 +54,7 @@ type TableBucketSummary struct {
|
||||
ARN string `json:"arn"`
|
||||
Name string `json:"name"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type ListTableBucketsResponse struct {
|
||||
@@ -235,9 +246,12 @@ type ListTablesRequest struct {
|
||||
}
|
||||
|
||||
type TableSummary struct {
|
||||
Name string `json:"name"`
|
||||
TableARN string `json:"tableARN"`
|
||||
Namespace []string `json:"namespace"`
|
||||
Name string `json:"name"`
|
||||
TableARN string `json:"tableARN"`
|
||||
Namespace []string `json:"namespace"`
|
||||
// Format lets a caller tell an Iceberg table from a catalog-only one without
|
||||
// a GetTable per row. AWS omits it; listing a mixed catalog needs it.
|
||||
Format string `json:"format,omitempty"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
ModifiedAt time.Time `json:"modifiedAt"`
|
||||
MetadataLocation string `json:"metadataLocation,omitempty"`
|
||||
@@ -562,6 +576,36 @@ func (e *S3TablesError) Error() string {
|
||||
return e.Message
|
||||
}
|
||||
|
||||
// Table formats a catalog entry may declare.
|
||||
//
|
||||
// ICEBERG tables carry metadata the catalog maintains and the maintenance
|
||||
// worker rewrites. LANCE is catalog-only: the entry records a name and the
|
||||
// dataset root in MetadataLocation, and the Lance client owns every byte under
|
||||
// it. Nothing in this package interprets a catalog-only table's files.
|
||||
const (
|
||||
FormatIceberg = "ICEBERG"
|
||||
FormatLance = "LANCE"
|
||||
)
|
||||
|
||||
// IsCatalogOnlyFormat reports whether the catalog only records where a table of
|
||||
// this format lives, without understanding its files.
|
||||
func IsCatalogOnlyFormat(format string) bool {
|
||||
return format == FormatLance
|
||||
}
|
||||
|
||||
// NormalizeFormat folds a caller's spelling onto the canonical one and reports
|
||||
// whether it names a format this catalog serves.
|
||||
func NormalizeFormat(format string) (string, bool) {
|
||||
switch strings.ToUpper(strings.TrimSpace(format)) {
|
||||
case FormatIceberg:
|
||||
return FormatIceberg, true
|
||||
case FormatLance:
|
||||
return FormatLance, true
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// Error codes
|
||||
const (
|
||||
ErrCodeBucketAlreadyExists = "BucketAlreadyExists"
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user