144 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandGitHub	8c7d714d5e	Lance catalog, and a Rust plugin worker to maintain it (#10841) ... * iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm	2026-08-19 22:59:56 -07:00
Chris LuandGitHub	ec37ef5aaa	iceberg: add view rename, scan-report and snapshots=refs to the catalog (#10776) ... * iceberg: add view rename, scan-report and snapshots=refs to the catalog Three gaps against the REST spec that clients hit in normal use: Views had no rename, though tables did and views are stored the same way, so the move is the same catalog-only pointer move. Tables and views share a namespace directory, so both renames now refuse the other kind instead of moving it. Engines POST a scan or commit report after planning; a 404 there turns into an error line per query. Accept the report and discard it - the catalog keeps no metrics store. LoadTable ignored ?snapshots=refs and always returned the whole snapshot history, which is what clients use the parameter to avoid on long-lived tables. * iceberg: authorize view rename against the view ARN, tighten the metrics endpoint Review follow-ups: The shared rename checked the source against a table ARN whatever the kind, so a policy scoped to a view's own ARN never matched and one written for a table ARN was evaluated for a view. The entry kind now carries the ARN builder. The metrics endpoint truncated a report at 1 MiB and then failed to parse it, answering 400 for a query that had actually succeeded. Read one byte past the limit to tell "fits" from "cut short", and discard an oversized report instead of rejecting it. Empty bodies and reports without a report-type are now rejected, which the REST schema requires. ?snapshots= is defined for LoadTable, so it no longer filters what CreateTable echoes back.	2026-08-16 12:56:45 -07:00
Chris LuandGitHub	d044839ab2	iceberg: make a table commit a compare-and-swap (#10775) ... * iceberg: make a table commit a compare-and-swap The catalog validated the caller's version token, ran its authorization checks, and only then wrote the new metadata xattr. Two engines committing against the same base both passed that check and both wrote, so the second silently dropped the first one's snapshot. Both also derive the same v{N}.metadata.json name and the file write overwrote, leaving the surviving pointer aimed at the loser's metadata - and the loser's conflict cleanup then deleted the winner's file. Write the metadata file with an exclusive create and update the xattr conditionally on the bytes the handler read, the way the maintenance worker already commits. A writer that lost the race re-reads and retries, and reports 409 CommitFailedException once out of attempts. * iceberg: stage a commit under a unique name when the versioned one is taken Two follow-ups from review of the commit compare-and-swap: Refusing to overwrite v{N}.metadata.json also refused to get past a file left behind by a commit that died between staging and updating the pointer. Every later commit derived the same name, saw the collision, and reported a conflict, so the table stayed uncommittable until an orphan sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's file is overwritten and the catalog pointer still decides who won, which is how the maintenance worker has always staged its own metadata. metadataVersionFromLocation learned to read the version back out of that name. The conditional update guarded only the metadata attribute while the write replaced the whole entry, so a policy or tag written in the same window was silently reverted. Guard every catalog attribute, which turns that into a conflict the caller retries on fresh state. * iceberg: give saveMetadataFile the exclusive flag instead of a second name saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName were three new names around one existing helper. The flag now rides on saveMetadataFile and saveMetadataBlob, and the unique-name construction sits where it is used. * iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively #10773 landed mutateEntryExtended, which already writes an entry back under a whole-entry precondition and retries. Drop the helper this branch added and route the table commit through it: the check that the metadata is still the one this request read now lives in the mutation, where it sees current state. The policy the request was authorized against is asserted too, so an administrator restricting it mid-commit sends the caller back through authorization instead of having a stale decision applied. Bucket and namespace policies live on other entries and a single-entry precondition cannot cover them. Multi-table transactions stage their metadata exclusively for the same reason single-table commits do, and carry the name they landed on into the pointer flip.	2026-08-16 12:55:42 -07:00
Chris LuandGitHub	eef6f3d1e6	s3tables: add the maintenance configuration APIs (#10773) ... * s3tables: add the maintenance configuration APIs Stores the configuration verbatim as the wire shape under a new s3tables.maintenance extended attribute, so Get hands back what Put took and no translation layer can drift from the AWS model. Nothing reads the configuration yet. Put merges a single type into the stored map so configuring compaction does not drop snapshot management, and asserts the attribute's prior value so two concurrent Puts cannot silently clobber each other. * iceberg: apply the maintenance configuration in the worker The worker now reads the per-table and per-bucket maintenance configuration written by the control plane, so the wildcard plugin config is a default rather than the only setting a table can have. Table properties still win by default, since a table declaring its own layout is what every engine honours and the compactor has to agree with whoever writes the files. Clearing table_properties_override makes the maintenance configuration authoritative instead. Status is not part of that contest: a disabled type drops its operations and no property can re-enable them, so the operator's kill switch always holds. Manifest and delete-file rewrites have no AWS equivalent and ride with compaction. Detection reads both attributes from entries it already lists. * s3tables: report maintenance job status The worker records the outcome of each run in its own extended attribute, separate from the configuration so operator and worker writes do not contend, and GetTableMaintenanceJobStatus reads it back. Only the types a run touched are written, so a partial run cannot erase what an earlier one recorded. The reader fills in the rest: Disabled when the configuration switched a type off, Not_Yet_Run otherwise. Status is advisory, so a lost race is logged rather than failing a job whose work already committed. * s3tables: route the maintenance APIs over REST The five actions were only reachable by X-Amz-Target dispatch, which the AWS CLI and SDK do not use for this service. They address the operations by path, so the APIs were unreachable from any official client. * s3tables: fix the table bucket ARN field name GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the wire field is tableBucketARN, as every other response in this package already spells it. Official SDK deserializers ignore the unknown key, so the required field came back unset. * s3tables: carry the compaction strategy through to the worker IcebergCompactionSettings modelled only targetFileSizeMB, so a request naming a strategy was accepted and then dropped on the way to storage. The worker now maps binpack and sort onto its own rewrite strategy and lets auto defer to the worker configuration. z-order is rejected rather than accepted and quietly binpacked. * s3tables: report bucket-level maintenance status GetTableMaintenanceJobStatus read only the table's configuration, so unreferenced file removal — which is configured on the bucket — reported Not_Yet_Run or a stale success after an operator disabled it. The merge helper now lives in this package and the worker shares it. * iceberg: delete orphans only after the non-current window AWS marks a file non-current once it has been unreferenced for unreferencedDays, then deletes it a further nonCurrentDays later. The cutoff was taken from unreferencedDays alone, so a 3/10 configuration hard-deleted on day three and threw away the ten day recovery window. remove_orphans deletes in one step rather than marking, so the cutoff is now the sum of the two. * s3tables: assert every attribute when rewriting an entry UpdateEntry writes the whole entry back from the snapshot the caller read, and its precondition only covers the keys the caller names. Both maintenance writers named one key, so a job status write could revert a maintenance configuration an operator had just disabled, turning an advisory write into a silent re-enable. Both now assert the entry's full attribute set, including the target key when absent so a concurrent create also fails the precondition. * s3tables: assert absent attributes when rewriting an entry The precondition covered the attributes present when the writer read the entry, so an attribute created between that read and the write was absent from it. A first-time PutTableMaintenanceConfiguration disabling a type therefore lands, passes the per-key checks, and is then deleted by the stale whole-entry write. Every attribute this package stores is now asserted, absent ones included. The metadata commit and planning index writers rewrite the same entries and had the same exposure, so both use the shared snapshot too. * iceberg: implement the auto compaction strategy auto was accepted, stored and read back, but left the worker on its own default, so a sorted table configured as auto was compacted with binpack. AWS defines auto as sorting tables that declare a sort order and bin-packing the rest. That needs the table metadata, so the choice is made where the rewrite plan is resolved: an unsorted table falls back to binpack rather than failing the way an explicit sort request does. * s3tables: validate the maintenance setting ranges PUT accepted zero, negative and oversized values for every numeric setting. The worker then ignores a non-positive value and saturates an oversized one, so the configuration read back was not the one that ran. AWS bounds all five to 1..2147483647, which is now enforced. The fields are pointers so an explicit zero is distinguishable from an omitted one and can be rejected rather than silently ignored. * s3tables: give every entry writer the same compare-and-swap updateExtendedAttribute asserted the entry's attributes, but the helpers behind the metadata, policy and tag handlers still wrote the whole entry unconditionally. Any of them could land on a stale snapshot and delete a maintenance configuration an operator had just written. They all share one read-modify-write loop now, so the precondition and the bounded retry apply wherever an entry is rewritten. * s3tables: move the maintenance configuration with a renamed table RenameTable carried the metadata, version, policy and tags to the new name but left the maintenance configuration and job status behind. A table with snapshot management disabled came back enabled under its new name, and the stale configuration stayed on the old name where a table created there would inherit it. The decoupled-delete cleanup left the same two attributes behind. * s3tables: accept every AWS partition in ARNs The route regexes and the ARN patterns both hardcoded arn:aws, so valid aws-cn and aws-us-gov ARNs never reached a handler. The router now shares the partition-tolerant prefix with the parser, and a generated ARN uses the partition its region belongs to so it parses back. * s3tables: generate ARNs in the region's partition The handler's own ARN generators still formatted arn:aws directly rather than going through the partition-aware builder, so a China or GovCloud deployment routed the request but then returned a commercial ARN and matched IAM policies against it. The round-trip test missed this because parsing accepts any partition, so it now asserts the prefix the region implies. * s3tables: complete the ARN partition table aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-* and eusc-* regions fell through to the commercial partition. * s3tables: do not let a rename swallow a concurrent maintenance write Rename copied the source attributes early and cleared the source at the end, so a Put landing in between missed the copy to the destination and was then deleted by the cleanup. It succeeded and vanished. The cleanup now clears the source only while it still holds exactly what was copied, and returns a conflict otherwise. Put checks the catalog identity inside the same conditional mutation, so it also cannot write to a name that a rename or delete has already soft-deleted.	2026-08-16 10:36:59 -07:00
Chris LuandGitHub	c191b2fe01	iceberg: let clients select their table bucket as the catalog warehouse (#10549) ... * iceberg: accept bare bucket names and ARNs as the catalog warehouse Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table bucket name or as the s3tables bucket ARN -- the two forms users reach for first, the latter being what AWS S3 Tables itself takes -- was silently dropped, so every call landed on the default "warehouse" bucket and failed with "table bucket warehouse not found". * iceberg: report a missing table bucket as 404, not 500 Pointing a client at a table bucket that does not exist -- which every client with no warehouse set does, since the default bucket "warehouse" rarely exists -- returned InternalServerError with a message naming a bucket the client never asked for. Answer 404 and say how to select one. * admin: show the warehouse in the PyIceberg example The example connected without one, so it always resolved to the default table bucket and every client that copied it failed on the first call. * test: pin bearer auth against a table bucket that exists The subtest called the catalog with no warehouse and accepted 500 as proof that auth had passed, since the default bucket does not exist. A missing table bucket now answers 404, which the test read as an auth failure. Give it a real table bucket so only 200 passes. * test: assert the missing-bucket guidance reaches the client The status and error type were checked but not the message, which is the part of the mapping that tells a user how to select a table bucket. * test: encode the warehouse query value The ARN case pasted raw colons and slashes into the query string. Go's parser tolerates them, so the test passed without modelling how a client actually sends the request.	2026-08-03 13:25:37 -07:00
Chris LuandGitHub	4b0d09683a	iceberg: read manifest lists that omit the Avro format version (#10475) ... * s3tables: read Iceberg manifest lists that omit the Avro format version The Iceberg spec pins the Avro header metadata of manifest files but says nothing about manifest lists, so writers disagree. Java and PyIceberg record "format-version"; DuckDB writes no header metadata at all. iceberg-go reads a missing entry as v1, so every v2 manifest listed in a DuckDB-written list is rejected with manifest file's 'format-version' metadata indicates version 2, but entry from manifest list indicates version 1 and, because v1 has no "content" field, delete manifests silently decode as data manifests. ReadManifestList derives the version from the record schema the writer embedded - v2 added "content" and the sequence numbers, v3 added "first_row_id" - and splices it into the header before handing the bytes to iceberg-go. Lists that already carry the entry, and input that is not a parseable Avro container, go through untouched. * iceberg: parse DuckDB-written manifest lists in maintenance and data preview Every manifest list read - the four maintenance operations and the admin table data preview - went straight to iceberg-go, so tables written by DuckDB failed detection and all of compact, remove_orphans, rewrite_manifests and expire_snapshots before they touched anything. Route them through s3tables.ReadManifestList, which recovers the format version the writer left out of the Avro header. This also restores the manifest content type on those tables: with the list read as v1 every delete manifest looked like a data manifest, which hid deletes from the compaction guard and made the preview report a table with position deletes as having none.	2026-07-28 16:42:17 -07:00
Chris LuandGitHub	b4b0346f95	iceberg maintenance: resolve table files from the recorded location (#10418) ... The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table.	2026-07-24 02:40:14 -07:00
Chris LuandGitHub	c15989387b	s3tables: allow hyphens in namespace and table names (#10093) ... * s3tables: allow hyphens in namespace and table names Iceberg REST clients routinely use hyphenated namespace/table names, but the S3 Tables charset (a-z, 0-9, _) rejected them with 400. Accept '-' as an interior character (names must still start, and namespaces end, with a letter or digit), making the catalog conformant for those clients. A permissive superset of the AWS S3 Tables charset. * s3tables: allow hyphens in table ARN parsing too The ARN regexes still excluded '-', so parseTableFromARN rejected ARNs with hyphenated namespace/table names and existing reject-the-hyphen tests broke. Widen the ARN patterns to match the validator, retarget those tests at a still-invalid leading-hyphen name, and cover ARN parsing with hyphens.	2026-06-24 16:24:45 -07:00
Chris LuandGitHub	1c5f8244a4	s3tables: fix create-after-rename overwriting the renamed table (#10091) ... * s3tables: purge decoupled table data without deleting the reused name path A renamed or created-over-leftover table keeps its data at a location that differs from its catalog name path. Drop now purges that data location and clears the marker, instead of recursively deleting the name path, which may still hold another table's data. * iceberg: route a table created over a leftover to a unique location When the default location is occupied by a leftover directory (data kept when another table was renamed to this name), create the new table at a unique location so it cannot overwrite that table's metadata. Common case is unchanged. * iceberg: fail table create when the leftover-path check errors A transient filer lookup error fell through as "not occupied", routing the new table back to the default path and risking the very overwrite this check guards against. Propagate the error and return 500 instead. * s3tables: assert all catalog xattrs cleared on decoupled drop Seed the full marker set so the test catches a regression that leaves the policy, tags, version, or entry-type attribute on the reused name path. * s3tables: refuse to drop a table whose data path is an ancestor Corrupt metadata can resolve the data path to the bucket or namespace root, which the bucket-scope check still admits; a recursive purge there would wipe sibling tables. Reject an ancestor data path before deleting.	2026-06-24 14:37:04 -07:00
Chris LuandGitHub	c95401b11a	iceberg: support table rename (#10068) ... * s3tables: add RenameTable operation * iceberg: support table rename * iceberg: test table rename * s3tables: keep table data in place on rename rename is catalog-only: drop the source's catalog xattrs in place instead of recursively deleting its directory, which wiped the metadata.json and data files the renamed destination still points at. treat a missing table-metadata xattr as NoSuchTable in GetTable so the soft-deleted source name stops resolving. * s3tables: test rename preserves data make the in-memory filer honor recursive data deletion and seed the source table's metadata/ and data/ children, then assert a rename leaves them intact, the source name resolves to NoSuchTable, and the destination resolves to the preserved location. * iceberg: map rename errors through wrapped manager error * s3tables: authorize rename destination namespace rename moved a table into the destination namespace after only checking the source, letting a source-authorized caller place tables in namespaces they don't control. require CreateTable on the destination namespace and bucket before writing. * s3tables: purge renamed table data on drop * s3tables: test table data dir derivation	2026-06-23 20:18:11 -07:00
Chris LuandGitHub	0403e47ef6	iceberg: support views (#10069) ... * s3tables: tag table entries and exclude views from table listings * s3tables: add view CRUD operations * iceberg: support view create, load, exists, drop, and list * iceberg: support view update * iceberg: test view error classification and metadata round-trip * iceberg: pre-check existence and write view metadata only after create * iceberg: map view namespace-not-found to 404 * iceberg: test view create namespace-404 and duplicate no-clobber * s3tables: tag view metadata and entry type atomically CreateView wrote ExtendedKeyMetadata and ExtendedKeyEntryType in two UpdateEntry calls, so a partial failure could leave a view directory untagged. Add setExtendedAttributes to set both in one UpdateEntry. * iceberg: roll back view registration when metadata write fails The metadata file is written after the catalog registers the view. If that write fails, drop the just-created view so it doesn't linger pointing at a missing metadata.json. Reuse the DeleteView path via a shared dropView helper.	2026-06-23 15:22:31 -07:00
Chris LuandGitHub	628ce57625	iceberg: support table register (#10067) ... * s3tables: add RegisterTable op * iceberg: support table register * iceberg: test register table * iceberg: parse engine-written metadata version from location * iceberg: test metadata version parsing for both filename forms * iceberg: map register errors through wrapped manager error * iceberg: validate register metadata-location bucket and reject traversal * iceberg: log register metadata load failure	2026-06-23 14:07:13 -07:00
Chris LuandGitHub	0ded0984a4	iceberg: support namespace property updates (#10052) ... * iceberg: support namespace property updates Add POST /v1/namespaces/{namespace}/properties to the REST catalog. It applies the request's removals and updates and returns the removed/updated/ missing summary the spec defines. A new UpdateNamespace op on the S3 Tables manager rewrites the stored namespace properties; AWS S3 Tables namespaces have no properties, so this is the SeaweedFS-side backing for the catalog. * iceberg: dedup namespace property removals A key repeated in removals was deleted on its first occurrence, then reported as missing on the next — landing in both removed and missing. Skip keys already processed. * iceberg: map namespace-update backend errors to REST statuses UpdateNamespaceProperties returned 500 for every manager failure, masking the namespace being dropped between read and write, or a denied caller. Inspect the typed S3TablesError and answer 404/403 accordingly, 500 only for the rest. Also replaces the GetNamespace not-found string match. * iceberg: test the namespace-properties conflict path Cover the 422 returned when a key appears in both removals and updates. The check runs before any backend call, so it needs no filer.	2026-06-23 00:41:47 -07:00
Chris LuandGitHub	b13463880c	s3tables: scope management authorization to the caller's identity (#9961) ... * s3tables: resolve account-less identities to a distinct principal Static identities with no account block default to the shared admin account, so getAccountID returned "admin" for every such user and the permission checks treated them all as the admin principal. Only keep the admin account when the identity actually carries an admin action; otherwise fall back to the unique identity name. * s3tables: limit the open-by-default fallback to anonymous access The legacy permission path allowed any request that no policy explicitly denied whenever default-allow was on, which is the zero-config default. That let an authenticated identity without table permissions reach table resources owned by others. Restrict the fallback to requests with no identity or the anonymous identity; authenticated callers must pass an explicit action or policy check. Zero-config and anonymous access are unchanged. * s3tables: drop the no-op ListTableBuckets account gate The top-level check passed the principal as its own owner, so it always allowed. Per-bucket filtering in the loop is the real authority; remove the dead gate and the now-unused locals. * s3tables: derive the Iceberg catalog's default-allow from auth state The Iceberg catalog reuses the S3 Tables Manager, which hardcoded default-allow on. Authenticated callers were enforced only because the identity struct happens to propagate into the handler; if it were ever dropped, a secured catalog would fall open. Mirror the S3 port and set the Manager's default-allow from the authenticator, so an authenticated caller is enforced regardless. Shell and admin keep their own trusted Manager. Regression test covers the struct, name-only, and admin paths. * s3tables: drop redundant ACTION_ADMIN string conversion ACTION_ADMIN is an untyped string constant, so the conversion is a no-op. * s3tables: enforce name-only authenticated callers, add trusted bypass defaultAllowFor treated a request with no identity object as anonymous, but the Manager path forwards only the identity name (not the struct). A name-only authenticated caller could therefore be misclassified as anonymous and allowed under the open default. Treat a server-set identity name as authenticated too, and add an explicit trusted flag for the local shell/admin tooling that legitimately bypasses authorization. * s3tables: trim verbose comments	2026-06-14 13:55:36 -07:00
3 people	69c84801e4	fix(s3tables/iceberg): make metadata spec-compliant and accept real-world manifest names (#9703) ... * fix(s3tables/iceberg): make metadata spec-compliant and accept real-world manifest names Two related issues prevent SeaweedFS S3 Tables from interoperating with strict Iceberg clients (Java/Spark/Flink/Trino): 1. iceberg-go v0.5.0 serializes empty TableMetadata state by dropping keys via `omitempty` on optional pointer/slice fields. The Iceberg table spec, however, requires `current-snapshot-id`, `snapshots`, `snapshot-log`, `metadata-log`, and `refs` to be present even when empty (`current-snapshot-id` must be -1 for a table with no snapshots). Java's TableMetadataParser uses JsonUtil.getLong on `current-snapshot-id` and throws "Cannot parse missing long current-snapshot-id" against responses produced by this server. 2. The Iceberg layout validator only accepts manifest filenames that match Iceberg's internal naming (`{uuid}-m{n}.avro`, `snap-{n}-{n}-{uuid}.avro`). Real writers — notably Flink's sink — emit manifests like `{flink-job-id}-{checkpoint}-{operator-id}-{n}.avro`, which the validator rejects with 403, breaking INSERT commits. Fixes: * Add ensureMetadataSpecCompliance helper that backfills the five spec-required empty-state fields when iceberg-go omits them or emits explicit JSON null. Apply it on every code path that writes v*.metadata.json to S3 or returns metadata to clients (handlers_table create-table, handlers_commit, commit_helpers create-on-commit, plus MarshalJSON on LoadTableResult and CommitTableResponse). Real values from non-empty tables are never overwritten. * Add catch-all regex entries to metadataFilePatterns accepting any *.avro / *.metadata.json filename composed of [A-Za-z0-9._-]. The Iceberg spec does not mandate filename format; the strict patterns remain for documentation. Metadata-directory subdirectory rejection and the data-file path validation are unchanged. No upstream dependencies are forked: iceberg-go stays at v0.5.0 and go.mod is untouched. The compliance layer can be removed once upstream emits spec-compliant output. Tests (all pass under `go test -race`): - metadata_compliance_test.go: 5 cases covering missing fields, preserved real values, explicit null, invalid JSON, empty input. - iceberg_layout_test.go: 3 groups (16 subtests) covering real-world manifest names from Flink/Spark/Iceberg, security boundary (subdirectories, bad extensions), and data-file regression. * fix(s3tables/iceberg): preserve metadata key order and keep config field stable Two small follow-ups on the spec-compliance fix: * ensureMetadataSpecCompliance now splices missing keys in at the byte level just before the closing brace, so iceberg-go's struct-declared key order survives the backfill. The previous unmarshal/remarshal through map[string]json.RawMessage silently alphabetized every key in the document, which is spec-legal but breaks byte-equality fixtures and any downstream hashing of the persisted metadata. The slower remarshal path is kept for the rare explicit-null replacement case. * LoadTableResult.MarshalJSON now serializes Config without omitempty, matching the struct field tag. The custom marshaler had silently flipped the tag to ,omitempty, which made the "config" key disappear from the response whenever s3Endpoint was unset (since buildFileIOConfig returned an empty but non-nil Properties map). Tests: - PreservesOriginalKeyOrder pins the byte-level output against iceberg-go's emitted shape; would have caught the alphabetization regression. - EmptyObjectBackfilled covers the {} -> sentinels-only case (no leading comma). - AllPresentReturnsSameBytes confirms the no-op path returns input bytes unchanged, with whitespace intact. - iceberg_layout_test pins the catch-all $ anchor: metadata/file.avro.txt must still be rejected. * fix(s3tables/iceberg): guard ensureMetadataSpecCompliance against top-level null json.Unmarshal of a JSON `null` literal succeeds but leaves the map nil. The current byte-append path no-ops gracefully on this input, but the slow remarshal path would panic with "assignment to entry in nil map" if the input ever combined `null` with the explicit-null detection. Add an explicit nil-map short-circuit so the safety property is obvious from the source, and a test that pins the contract. * test(s3tables/iceberg): assert full byte equality in AllPresentReturnsSameBytes The prefix check only caught a missing "{\n " opener, so the test would have passed even if the function silently reordered keys or collapsed whitespace later in the document. Switch to a full string comparison so any future regression in the no-op path is loud. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-05-27 13:05:41 -07:00
Chris LuandGitHub	cce98fcecf	fix(s3): strip client-supplied X-SeaweedFS-Principal/Session-Token in AuthSignatureOnly (#9120) ... * fix(s3): strip client-supplied X-SeaweedFS-Principal/Session-Token in AuthSignatureOnly AuthSignatureOnly is the only auth gate in front of S3Tables routes (incl. CreateTableBucket) and UnifiedPostHandler, but unlike authenticateRequestInternal it did not clear the internal IAM trust headers before running signature verification. S3Tables authorizeIAMAction reads X-SeaweedFS-Principal directly from the request and prefers it over the authenticated identity's PrincipalArn, so a signed low-privilege caller could append that header after signing (unsigned header, SigV4 still verifies) and have IAM policy evaluated against a spoofed principal, bypassing authorization. Clear both X-SeaweedFS-Principal and X-SeaweedFS-Session-Token at the top of AuthSignatureOnly, mirroring the existing guard in authenticateRequestInternal. Add a regression test covering the header-injection path. * refactor(s3): route AuthSignatureOnly through authenticateRequestInternal Addresses review feedback: both entry points were independently maintaining the internal-IAM-header stripping and the auth-type dispatch switch. Collapse AuthSignatureOnly into a thin wrapper around authenticateRequestInternal so the security-critical header scrub and the signature-verify switch live in one place. Post-auth behavior unique to AuthSignatureOnly (AmzAccountId header) stays inline. No functional change beyond two harmless telemetry tweaks that now match authenticateRequestInternal: the per-branch glog verbosity shifts from V(3) to V(4), and the anonymous-found path now sets AmzAuthType. * refactor(s3): centralize X-SeaweedFS-Principal/Session-Token header names Introduce SeaweedFSPrincipalHeader and SeaweedFSSessionTokenHeader in weed/s3api/s3_constants so the trust-header literals are defined once and referenced consistently by the auth scrub, JWT auth path, bucket policy principal resolution, IAM authorization, and S3Tables IAM evaluation. Replace every remaining usage in weed/s3api and weed/s3api/s3tables. This removes the drift risk the reviewer called out: adding another call site with a typo can no longer silently bypass the scrub. Pure rename, no behavior change. No-op integration-test helper in test/s3/iam/s3_iam_framework.go left untouched (separate module, and the server now strips the client-supplied value regardless).	2026-04-17 12:23:21 -07:00
Chris LuandGitHub	995dfc4d5d	chore: remove ~50k lines of unreachable dead code (#8913) ... * chore: remove unreachable dead code across the codebase Remove ~50,000 lines of unreachable code identified by static analysis. Major removals: - weed/filer/redis_lua: entire unused Redis Lua filer store implementation - weed/wdclient/net2, resource_pool: unused connection/resource pool packages - weed/plugin/worker/lifecycle: unused lifecycle plugin worker - weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy, multipart IAM, key rotation, and various SSE helper functions - weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions - weed/mq/offset: unused SQL storage and migration code - weed/worker: unused registry, task, and monitoring functions - weed/query: unused SQL engine, parquet scanner, and type functions - weed/shell: unused EC proportional rebalance functions - weed/storage/erasure_coding/distribution: unused distribution analysis functions - Individual unreachable functions removed from 150+ files across admin, credential, filer, iam, kms, mount, mq, operation, pb, s3api, server, shell, storage, topology, and util packages * fix(s3): reset shared memory store in IAM test to prevent flaky failure TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because the MemoryStore credential backend is a singleton registered via init(). Earlier tests that create anonymous identities pollute the shared store, causing LookupAnonymous() to unexpectedly return true. Fix by calling Reset() on the memory store before the test runs. * style: run gofmt on changed files * fix: restore KMS functions used by integration tests * fix(plugin): prevent panic on send to closed worker session channel The Plugin.sendToWorker method could panic with "send on closed channel" when a worker disconnected while a message was being sent. The race was between streamSession.close() closing the outgoing channel and sendToWorker writing to it concurrently. Add a done channel to streamSession that is closed before the outgoing channel, and check it in sendToWorker's select to safely detect closed sessions without panicking.	2026-04-03 16:04:27 -07:00
Chris LuandGitHub	acea36a181	filer: add conditional update preconditions (#8647) ... * filer: add conditional update preconditions * iceberg: tighten metadata CAS preconditions	2026-03-16 12:33:32 -07:00
Chris LuandGitHub	4f647e1036	Worker set its working directory (#8461) ... * set working directory * consolidate to worker directory * working directory * correct directory name * refactoring to use wildcard matcher * simplify * cleaning ec working directory * fix reference * clean * adjust test	2026-02-27 12:22:21 -08:00
Chris LuandGitHub	8eba7ba5b2	feat: drop table location mapping support (#8458) ... * feat: drop table location mapping support Disable external metadata locations for S3 Tables and remove the table location mapping index entirely. Table metadata must live under the table bucket paths, so lookups no longer use mapping directories. Changes: - Remove mapping lookup and cache from bucket path resolution - Reject metadataLocation in CreateTable and UpdateTable - Remove mapping helpers and tests * compile * refactor * fix: accept metadataLocation in S3 Tables API requests We removed the external table location mapping feature, but still need to accept and store metadataLocation values from clients like Trino. The mapping feature was an internal implementation detail that mapped external buckets to internal table paths. The metadataLocation field itself is part of the S3 Tables API and should be preserved. * fmt * fix: handle MetadataLocation in UpdateTable requests Mirror handleCreateTable behavior by updating metadata.MetadataLocation when req.MetadataLocation is provided in UpdateTable requests. This ensures table metadata location can be updated, not just set during creation.	2026-02-26 16:36:24 -08:00
Chris LuandGitHub	641351da78	fix: table location mappings to /etc/s3tables (#8457) ... * fix: move table location mappings to /etc/s3tables to avoid bucket name validation Fixes #8362 - table location mappings were stored under /buckets/.table-location-mappings which fails bucket name validation because it starts with a dot. Moving them to /etc/s3tables resolves the migration error for upgrades. Changes: - Table location mappings now stored under /etc/s3tables - Ensure parent /etc directory exists before creating /etc/s3tables - Normal writes go to new location only (no legacy compatibility) - Removed bucket name validation exception for old location * refactor: simplify lookupTableLocationMapping by removing redundant mappingPath parameter The mappingPath function parameter was redundant as the path can be derived from mappingDir and bucket using path.Join. This simplifies the code and reduces the risk of path mismatches between parameters.	2026-02-26 15:35:13 -08:00
3 people	0fac6e39ea	weed/s3api/s3tables: fix dropped errors (#8456) ... * weed/s3api/s3tables: fix dropped errors * enhance errors * fail fast when listing tables --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-02-26 11:12:10 -08:00
Chris Lu	8e8edd7706	not empty only if there are actual files in the bucket	2026-02-23 00:12:04 -08:00
Chris LuandGitHub	36c469e34e	Enforce IAM for S3 Tables bucket creation (#8388) ... * Enforce IAM for s3tables bucket creation * Prefer IAM path when policies exist * Ensure IAM enforcement honors default allow * address comments * Reused the precomputed principal when setting tableBucketMetadata.OwnerAccountID, avoiding the redundant getAccountID call. * get identity * fix * dedup * fix * comments * fix tests * update iam config * go fmt * fix ports * fix flags * mini clean shutdown * Revert "update iam config" This reverts commit ca48fdbb0a. Revert "mini clean shutdown" This reverts commit 9e17f6baff. Revert "fix flags" This reverts commit e9e7b29d2f. Revert "go fmt" This reverts commit bd3241960b. * test/s3tables: share single weed mini per test package via TestMain Previously each top-level test function in the catalog and s3tables package started and stopped its own weed mini instance. This caused failures when a prior instance wasn't cleanly stopped before the next one started (port conflicts, leaked global state). Changes: - catalog/iceberg_catalog_test.go: introduce TestMain that starts one shared TestEnvironment (external weed binary) before all tests and tears it down after. All individual test functions now use sharedEnv. Added randomSuffix() for unique resource names across tests. - catalog/pyiceberg_test.go: updated to use sharedEnv instead of per-test environments. - catalog/pyiceberg_test_helpers.go -> pyiceberg_test_helpers_test.go: renamed to a _test.go file so it can access TestEnvironment which is defined in a test file. - table-buckets/setup.go: add package-level sharedCluster variable. - table-buckets/s3tables_integration_test.go: introduce TestMain that starts one shared TestCluster before all tests. TestS3TablesIntegration now uses sharedCluster. Extract startMiniClusterInDir (no *testing.T) for TestMain use. TestS3TablesCreateBucketIAMPolicy keeps its own cluster (different IAM config). Remove miniClusterMutex (no longer needed). Fix Stop() to not panic when t is nil." * delete * parse * default allow should work with anonymous * fix port * iceberg route The failures are from Iceberg REST using the default bucket warehouse when no prefix is provided. Your tests create random buckets, so /v1/namespaces was looking in warehouse and failing. I updated the tests to use the prefixed Iceberg routes (/v1/{bucket}/...) via a small helper. * test(s3tables): fix port conflicts and IAM ARN matching in integration tests - Pass -master.dir explicitly to prevent filer store directory collision between shared cluster and per-test clusters running in the same process - Pass -volume.port.public and -volume.publicUrl to prevent the global publicPort flag (mutated from 0 → concrete port by first cluster) from being reused by a second cluster, causing 'address already in use' - Remove the flag-reset loop in Stop() that reset global flag values while other goroutines were reading them (race → panic) - Fix IAM policy Resource ARN in TestS3TablesCreateBucketIAMPolicy to use wildcards (arn:aws:s3tables:*:*:bucket/<name>) because the handler generates ARNs with its own DefaultRegion (us-east-1) and principal name ('admin'), not the test constants testRegion/testAccountID	2026-02-19 22:52:05 -08:00
Chris LuandGitHub	d1fecdface	Fix IAM defaults and S3Tables IAM regression (#8374) ... * Fix IAM defaults and s3tables identities * Refine S3Tables identity tests * Clarify identity tests	2026-02-18 18:20:03 -08:00
Chris Lu	eda4a000cc	Revert "Fix IAM defaults and s3tables identities" ... This reverts commit bf71fe0039.	2026-02-18 16:23:13 -08:00
Chris Lu	bf71fe0039	Fix IAM defaults and s3tables identities	2026-02-18 16:21:48 -08:00
Chris Lu	35ad7d08a5	remove debug	2026-02-16 14:03:02 -08:00
Chris LuandGitHub	0d8588e3ae	S3: Implement IAM defaults and STS signing key fallback (#8348) ... * S3: Implement IAM defaults and STS signing key fallback logic * S3: Refactor startup order to init SSE-S3 key manager before IAM * S3: Derive STS signing key from KEK using HKDF for security isolation * S3: Document STS signing key fallback in security.toml * fix(s3api): refine anonymous access logic and secure-by-default behavior - Initialize anonymous identity by default in `NewIdentityAccessManagement` to prevent nil pointer exceptions. - Ensure `ReplaceS3ApiConfiguration` preserves the anonymous identity if not present in the new configuration. - Update `NewIdentityAccessManagement` signature to accept `filerClient`. - In legacy mode (no policy engine), anonymous defaults to Deny (no actions), preserving secure-by-default behavior. - Use specific `LookupAnonymous` method instead of generic map lookup. - Update tests to accommodate signature changes and verify improved anonymous handling. * feat(s3api): make IAM configuration optional - Start S3 API server without a configuration file if `EnableIam` option is set. - Default to `Allow` effect for policy engine when no configuration is provided (Zero-Config mode). - Handle empty configuration path gracefully in `loadIAMManagerFromConfig`. - Add integration test `iam_optional_test.go` to verify empty config behavior. * fix(iamapi): fix signature mismatch in NewIdentityAccessManagementWithStore * fix(iamapi): properly initialize FilerClient instead of passing nil * fix(iamapi): properly initialize filer client for IAM management - Instead of passing `nil`, construct a `wdclient.FilerClient` using the provided `Filers` addresses. - Ensure `NewIdentityAccessManagementWithStore` receives a valid `filerClient` to avoid potential nil pointer dereferences or limited functionality. * clean: remove dead code in s3api_server.go * refactor(s3api): improve IAM initialization, safety and anonymous access security * fix(s3api): ensure IAM config loads from filer after client init * fix(s3): resolve test failures in integration, CORS, and tagging tests - Fix CORS tests by providing explicit anonymous permissions config - Fix S3 integration tests by setting admin credentials in init - Align tagging test credentials in CI with IAM defaults - Added goroutine to retry IAM config load in iamapi server * fix(s3): allow anonymous access to health targets and S3 Tables when identities are present * fix(ci): use /healthz for Caddy health check in awscli tests * iam, s3api: expose DefaultAllow from IAM and Policy Engine This allows checking the global "Open by Default" configuration from other components like S3 Tables. * s3api/s3tables: support DefaultAllow in permission logic and handler Updated CheckPermissionWithContext to respect the DefaultAllow flag in PolicyContext. This enables "Open by Default" behavior for unauthenticated access in zero-config environments. Added a targeted unit test to verify the logic. * s3api/s3tables: propagate DefaultAllow through handlers Propagated the DefaultAllow flag to individual handlers for namespaces, buckets, tables, policies, and tagging. This ensures consistent "Open by Default" behavior across all S3 Tables API endpoints. * s3api: wire up DefaultAllow for S3 Tables API initialization Updated registerS3TablesRoutes to query the global IAM configuration and set the DefaultAllow flag on the S3 Tables API server. This completes the end-to-end propagation required for anonymous access in zero-config environments. Added a SetDefaultAllow method to S3TablesApiServer to facilitate this. * s3api: fix tests by adding DefaultAllow to mock IAM integrations The IAMIntegration interface was updated to include DefaultAllow(), breaking several mock implementations in tests. This commit fixes the build errors by adding the missing method to the mocks. * env * ensure ports * env * env * fix default allow * add one more test using non-anonymous user * debug * add more debug * less logs	2026-02-16 13:59:13 -08:00
4 people	0385acba02	s3tables: fix shared table-location bucket mapping collisions (#8286) ... * s3tables: prevent shared table-location bucket mapping overwrite * Update weed/s3api/bucket_paths.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2026-02-10 11:28:29 -08:00
Chris LuandGitHub	5ae3be44d1	iceberg: persist namespace properties for create/get (#8276) ... * iceberg: persist namespace properties via s3tables metadata * iceberg: simplify namespace properties normalization * s3tables: broaden namespace properties round-trip test * adjust logs * adjust logs	2026-02-09 22:20:45 -08:00
Chris Lu	be26ce74ce	s3tables: support multi-level namespace normalization	2026-02-09 19:42:31 -08:00
Chris LuandGitHub	5a0204310c	Add Iceberg admin UI (#8246) ... * Add Iceberg table details view * Enhance Iceberg catalog browsing UI * Fix Iceberg UI security and logic issues - Fix selectSchema() and partitionFieldsFromFullMetadata() to always search for matching IDs instead of checking != 0 - Fix snapshotsFromFullMetadata() to defensive-copy before sorting to prevent mutating caller's slice - Fix XSS vulnerabilities in s3tables.js: replace innerHTML with textContent/createElement for user-controlled data - Fix deleteIcebergTable() to redirect to namespace tables list on details page instead of reloading - Fix data-bs-target in iceberg_namespaces.templ: remove templ.SafeURL for CSS selector - Add catalogName to delete modal data attributes for proper redirect - Remove unused hidden inputs from create table form (icebergTableBucketArn, icebergTableNamespace) * Regenerate templ files for Iceberg UI updates * Support complex Iceberg type objects in schema Change Type field from string to json.RawMessage in both IcebergSchemaFieldInfo and internal icebergSchemaField to properly handle Iceberg spec's complex type objects (e.g. {"type": "struct", "fields": [...]}). Currently test data only shows primitive string types, but this change makes the implementation defensively robust for future complex types by preserving the exact JSON representation. Add typeToString() helper and update schema extraction functions to marshal string types as JSON. Update template to convert json.RawMessage to string for display. * Regenerate templ files for Type field changes * templ * Fix additional Iceberg UI issues from code review - Fix lazy-load flag that was set before async operation completed, preventing retries on error; now sets loaded flag only after successful load and throws error to caller for proper error handling and UI updates - Add zero-time guards for CreatedAt and ModifiedAt fields in table details to avoid displaying Go zero-time values; render dash when time is zero - Add URL path escaping for all catalog/namespace/table names in URLs to prevent malformed URLs when names contain special characters like /, ?, or # - Remove redundant innerHTML clear in loadIcebergNamespaceTables that cleared twice before appending the table list - Fix selectSnapshotForMetrics to remove != 0 guard for consistency with selectSchema fix; now always searches for CurrentSnapshotID without zero-value gate - Enhance typeToString() helper to display '(complex)' for non-primitive JSON types * Regenerate templ files for Phase 3 updates * Fix template generation to use correct file paths Run templ generate from repo root instead of weed/admin directory to ensure generated _templ.go files have correct absolute paths in error messages (e.g., 'weed/admin/view/app/iceberg_table_details.templ' instead of 'app/iceberg_table_details.templ'). This ensures both 'make admin-generate' at repo root and 'make generate' in weed/admin directory produce identical output with consistent file path references. * Regenerate template files with correct path references * Validate S3 Tables names in UI - Add client-side validation for table bucket and namespace names to surface errors for invalid characters (dots/underscores) before submission - Use HTML validity messages with reportValidity for immediate feedback - Update namespace helper text to reflect actual constraints (single-level, lowercase letters, numbers, and underscores) * Regenerate templ files for namespace helper text * Fix Iceberg catalog REST link and actions * Disallow S3 object access on table buckets * Validate Iceberg layout for table bucket objects * Fix REST API link to /v1/config * merge iceberg page with table bucket page * Allowed Trino/Iceberg stats files in metadata validation * fixes - Backend/data handling: - Normalized Iceberg type display and fallback handling in weed/admin/dash/s3tables_management.go. - Fixed snapshot fallback pointer semantics in weed/admin/dash/s3tables_management.go. - Added CSRF token generation/propagation/validation for namespace create/delete in: - weed/admin/dash/csrf.go - weed/admin/dash/auth_middleware.go - weed/admin/dash/middleware.go - weed/admin/dash/s3tables_management.go - weed/admin/view/layout/layout.templ - weed/admin/static/js/s3tables.js - UI/template fixes: - Zero-time guards for CreatedAt fields in: - weed/admin/view/app/iceberg_namespaces.templ - weed/admin/view/app/iceberg_tables.templ - Fixed invalid templ-in-script interpolation and host/port rendering in: - weed/admin/view/app/iceberg_catalog.templ - weed/admin/view/app/s3tables_buckets.templ - Added data-catalog-name consistency on Iceberg delete action in weed/admin/view/app/iceberg_tables.templ. - Updated retry wording in weed/admin/static/js/s3tables.js. - Regenerated all affected _templ.go files. - S3 API/comment follow-ups: - Reused cached table-bucket validator in weed/s3api/bucket_paths.go. - Added validation-failure debug logging in weed/s3api/s3api_object_handlers_tagging.go. - Added multipart path-validation design comment in weed/s3api/s3api_object_handlers_multipart.go. - Build tooling: - Fixed templ generate working directory issues in weed/admin/Makefile (watch + pattern rule). * populate data * test/s3tables: harden populate service checks * admin: skip table buckets in object-store bucket list * admin sidebar: move object store to top-level links * admin iceberg catalog: guard zero times and escape links * admin forms: add csrf/error handling and client-side name validation * admin s3tables: fix namespace delete modal redeclaration * admin: replace native confirm dialogs with modal helpers * admin modal-alerts: remove noisy confirm usage console log * reduce logs * test/s3tables: use partitioned tables in trino and spark populate * admin file browser: normalize filer ServerAddress for HTTP parsing	2026-02-08 20:06:32 -08:00
Chris LuandGitHub	403592bb9f	Add Spark Iceberg catalog integration tests and CI support (#8242) ... * Add Spark Iceberg catalog integration tests and CI support Implement comprehensive integration tests for Spark with SeaweedFS Iceberg REST catalog: - Basic CRUD operations (Create, Read, Update, Delete) on Iceberg tables - Namespace (database) management - Data insertion, querying, and deletion - Time travel capabilities via snapshot versioning - Compatible with SeaweedFS S3 and Iceberg REST endpoints Tests mirror the structure of existing Trino integration tests but use Spark's Python SQL API and PySpark for testing. Add GitHub Actions CI job for spark-iceberg-catalog-tests in s3-tables-tests.yml to automatically run Spark integration tests on pull requests. * fmt * Fix Spark integration tests - code review feedback * go mod tidy * Add go mod tidy step to integration test jobs Add 'go mod tidy' step before test runs for all integration test jobs: - s3-tables-tests - iceberg-catalog-tests - trino-iceberg-catalog-tests - spark-iceberg-catalog-tests This ensures dependencies are clean before running tests. * Fix remaining Spark operations test issues Address final code review comments: Setup & Initialization: - Add waitForSparkReady() helper function that polls Spark readiness with backoff instead of hardcoded 10-second sleep - Extract setupSparkTestEnv() helper to reduce boilerplate duplication between TestSparkCatalogBasicOperations and TestSparkTimeTravel - Both tests now use helpers for consistent, reliable setup Assertions & Validation: - Make setup-critical operations (namespace, table creation, initial insert) use t.Fatalf instead of t.Errorf to fail fast - Validate setupSQL output in TestSparkTimeTravel and fail if not 'Setup complete' - Add validation after second INSERT in TestSparkTimeTravel: verify row count increased to 2 before time travel test - Add context to error messages with namespace and tableName params Code Quality: - Remove code duplication between test functions - All critical paths now properly validated - Consistent error handling throughout * Fix go vet errors in S3 Tables tests Fixes: 1. setup_test.go (Spark): - Add missing import: github.com/testcontainers/testcontainers-go/wait - Use wait.ForLog instead of undefined testcontainers.NewLogStrategy - Remove unused strings import 2. trino_catalog_test.go: - Use net.JoinHostPort instead of fmt.Sprintf for address formatting - Properly handles IPv6 addresses by wrapping them in brackets * Use weed mini for simpler SeaweedFS startup Replace complex multi-process startup (master, volume, filer, s3) with single 'weed mini' command that starts all services together. Benefits: - Simpler, more reliable startup - Single weed mini process vs 4 separate processes - Automatic coordination between components - Better port management with no manual coordination Changes: - Remove separate master, volume, filer process startup - Use weed mini with -master.port, -filer.port, -s3.port flags - Keep Iceberg REST as separate service (still needed) - Increase timeout to 15s for port readiness (weed mini startup) - Remove volumePort and filerProcess fields from TestEnvironment - Simplify cleanup to only handle two processes (mini, iceberg rest) * Clean up dead code and temp directory leaks Fixes: 1. Remove dead s3Process field and cleanup: - weed mini bundles S3 gateway, no separate process needed - Removed s3Process field from TestEnvironment - Removed unnecessary s3Process cleanup code 2. Fix temp config directory leak: - Add sparkConfigDir field to TestEnvironment - Store returned configDir in writeSparkConfig - Clean up sparkConfigDir in Cleanup() with os.RemoveAll - Prevents accumulation of temp directories in test runs 3. Simplify Cleanup: - Now handles only necessary processes (weed mini, iceberg rest) - Removes both seaweedfsDataDir and sparkConfigDir - Cleaner shutdown sequence * Use weed mini's built-in Iceberg REST and fix python binary Changes: - Add -s3.port.iceberg flag to weed mini for built-in Iceberg REST Catalog - Remove separate 'weed server' process for Iceberg REST - Remove icebergRestProcess field from TestEnvironment - Simplify Cleanup() to only manage weed mini + Spark - Add port readiness check for iceberg REST from weed mini - Set Spark container Cmd to '/bin/sh -c sleep 3600' to keep it running - Change python to python3 in container.Exec calls This simplifies to truly one all-in-one weed mini process (master, filer, s3, iceberg-rest) plus just the Spark container. * go fmt * clean up * bind on a non-loopback IP for container access, aligned Iceberg metadata saves/locations with table locations, and reworked Spark time travel to use TIMESTAMP AS OF with safe timestamp extraction. * shared mini start * Fixed internal directory creation under /buckets so .objects paths can auto-create without failing bucket-name validation, which restores table bucket object writes * fix path Updated table bucket objects to write under `/buckets/<bucket>` and saved Iceberg metadata there, adjusting Spark time-travel timestamp to committed_at +1s. Rebuilt the weed binary (`go install ./weed`) and confirmed passing tests for Spark and Trino with focused test commands. * Updated table bucket creation to stop creating /buckets/.objects and switched Trino REST warehouse to s3://<bucket> to match Iceberg layout. * Stabilize S3Tables integration tests * Fix timestamp extraction and remove dead code in bucketDir * Use table bucket as warehouse in s3tables tests * Update trino_blog_operations_test.go * adds the CASCADE option to handle any remaining table metadata/files in the schema directory * skip namespace not empty	2026-02-08 10:03:53 -08:00
3 people	e6ee293c17	Add table operations test (#8241) ... * Add Trino blog operations test * Update test/s3tables/catalog_trino/trino_blog_operations_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: add table bucket path helpers and filer operations - Add table object root and table location mapping directories - Implement ensureDirectory, upsertFile, deleteEntryIfExists helpers - Support table location bucket mapping for S3 access * feat: manage table bucket object roots on creation/deletion - Create .objects directory for table buckets on creation - Clean up table object bucket paths on deletion - Enable S3 operations on table bucket object roots * feat: add table location mapping for Iceberg REST - Track table location bucket mappings when tables are created/updated/deleted - Enable location-based routing for S3 operations on table data * feat: route S3 operations to table bucket object roots - Route table-s3 bucket names to mapped table paths - Route table buckets to object root directories - Support table location bucket mapping lookup * feat: emit table-s3 locations from Iceberg REST - Generate unique table-s3 bucket names with UUID suffix - Store table metadata under table bucket paths - Return table-s3 locations for Trino compatibility * fix: handle missing directories in S3 list operations - Propagate ErrNotFound from ListEntries for non-existent directories - Treat missing directories as empty results for list operations - Fixes Trino non-empty location checks on table creation * test: improve Trino CSV parsing for single-value results - Sanitize Trino output to skip jline warnings - Handle single-value CSV results without header rows - Strip quotes from numeric values in tests * refactor: use bucket path helpers throughout S3 API - Replace direct bucket path operations with helper functions - Leverage centralized table bucket routing logic - Improve maintainability with consistent path resolution * fix: add table bucket cache and improve filer error handling - Cache table bucket lookups to reduce filer overhead on repeated checks - Use filer_pb.CreateEntry and filer_pb.UpdateEntry helpers to check resp.Error - Fix delete order in handler_bucket_get_list_delete: delete table object before directory - Make location mapping errors best-effort: log and continue, don't fail API - Update table location mappings to delete stale prior bucket mappings on update - Add 1-second sleep before timestamp time travel query to ensure timestamps are in past - Fix CSV parsing: examine all lines, not skip first; handle single-value rows * fix: properly handle stale metadata location mapping cleanup - Capture oldMetadataLocation before mutation in handleUpdateTable - Update updateTableLocationMapping to accept both old and new locations - Use passed-in oldMetadataLocation to detect location changes - Delete stale mapping only when location actually changes - Pass empty string for oldLocation in handleCreateTable (new tables have no prior mapping) - Improve logging to show old -> new location transitions * refactor: cleanup imports and cache design - Remove unused 'sync' import from bucket_paths.go - Use filer_pb.UpdateEntry helper in setExtendedAttribute and deleteExtendedAttribute for consistent error handling - Add dedicated tableBucketCache map[string]bool to BucketRegistry instead of mixing concerns with metadataCache - Improve cache separation: table buckets cache is now separate from bucket metadata cache * fix: improve cache invalidation and add transient error handling Cache invalidation (critical fix): - Add tableLocationCache to BucketRegistry for location mapping lookups - Clear tableBucketCache and tableLocationCache in RemoveBucketMetadata - Prevents stale cache entries when buckets are deleted/recreated Transient error handling: - Only cache table bucket lookups when conclusive (found or ErrNotFound) - Skip caching on transient errors (network, permission, etc) - Prevents marking real table buckets as non-table due to transient failures Performance optimization: - Cache tableLocationDir results to avoid repeated filer RPCs on hot paths - tableLocationDir now checks cache before making expensive filer lookups - Cache stores empty string for 'not found' to avoid redundant lookups Code clarity: - Add comment to deleteDirectory explaining DeleteEntry response lacks Error field * go fmt * fix: mirror transient error handling in tableLocationDir and optimize bucketDir Transient error handling: - tableLocationDir now only caches definitive results - Mirrors isTableBucket behavior to prevent treating transient errors as permanent misses - Improves reliability on flaky systems or during recovery Performance optimization: - bucketDir avoids redundant isTableBucket call via bucketRoot - Directly use s3a.option.BucketsPath for regular buckets - Saves one cache lookup for every non-table bucket operation * fix: revert bucketDir optimization to preserve bucketRoot logic The optimization to directly use BucketsPath bypassed bucketRoot's logic and caused issues with S3 list operations on delimiter+prefix cases. Revert to using path.Join(s3a.bucketRoot(bucket), bucket) which properly handles all bucket types and ensures consistent path resolution across the codebase. The slight performance cost of an extra cache lookup is worth the correctness and consistency benefits. * feat: move table buckets under /buckets Add a table-bucket marker attribute, reuse bucket metadata cache for table bucket detection, and update list/validation/UI/test paths to treat table buckets as /buckets entries. * Fix S3 Tables code review issues - handler_bucket_create.go: Fix bucket existence check to properly validate entryResp.Entry before setting s3BucketExists flag (nil Entry should not indicate existing bucket) - bucket_paths.go: Add clarifying comment to bucketRoot() explaining unified buckets root path for all bucket types - file_browser_data.go: Optimize by extracting table bucket check early to avoid redundant WithFilerClient call * Fix list prefix delimiter handling * Handle list errors conservatively * Fix Trino FOR TIMESTAMP query - use past timestamp Iceberg requires the timestamp to be strictly in the past. Use current_timestamp - interval '1' second instead of current_timestamp. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2026-02-07 13:27:47 -08:00
Chris LuandGitHub	a3b83f8808	test: add Trino Iceberg catalog integration test (#8228) ... * test: add Trino Iceberg catalog integration test - Create test/s3/catalog_trino/trino_catalog_test.go with TestTrinoIcebergCatalog - Tests integration between Trino SQL engine and SeaweedFS Iceberg REST catalog - Starts weed mini with all services and Trino in Docker container - Validates Iceberg catalog schema creation and listing operations - Uses native S3 filesystem support in Trino with path-style access - Add workflow job to s3-tables-tests.yml for CI execution * fix: preserve AWS environment credentials when replacing S3 configuration When S3 configuration is loaded from filer/db, it replaces the identities list and inadvertently removes AWS_ACCESS_KEY_ID credentials that were added from environment variables. This caused auth to remain disabled even though valid credentials were present. Fix by preserving environment-based identities when replacing the configuration and re-adding them after the replacement. This ensures environment credentials persist across configuration reloads and properly enable authentication. * fix: use correct ServerAddress format with gRPC port encoding The admin server couldn't connect to master because the master address was missing the gRPC port information. Use pb.NewServerAddress() which properly encodes both HTTP and gRPC ports in the address string. Changes: - weed/command/mini.go: Use pb.NewServerAddress for master address in admin - test/s3/policy/policy_test.go: Store and use gRPC ports for master/filer addresses This fix applies to: 1. Admin server connection to master (mini.go) 2. Test shell commands that need master/filer addresses (policy_test.go) * move * move * fix: always include gRPC port in server address encoding The NewServerAddress() function was omitting the gRPC port from the address string when it matched the port+10000 convention. However, gRPC port allocation doesn't always follow this convention - when the calculated port is busy, an alternative port is allocated. This caused a bug where: 1. Master's gRPC port was allocated as 50661 (sequential, not port+10000) 2. Address was encoded as '192.168.1.66:50660' (gRPC port omitted) 3. Admin client called ToGrpcAddress() which assumed port+10000 offset 4. Admin tried to connect to 60660 but master was on 50661 → connection failed Fix: Always include explicit gRPC port in address format (host:httpPort.grpcPort) unless gRPC port is 0. This makes addresses unambiguous and works regardless of the port allocation strategy used. Impacts: All server-to-server gRPC connections now use properly formatted addresses. * test: fix Iceberg REST API readiness check The Iceberg REST API endpoints require authentication. When checked without credentials, the API returns 403 Forbidden (not 401 Unauthorized). The readiness check now accepts both auth error codes (401/403) as indicators that the service is up and ready, it just needs credentials. This fixes the 'Iceberg REST API did not become ready' test failure. * Fix AWS SigV4 signature verification for base64-encoded payload hashes AWS SigV4 canonical requests must use hex-encoded SHA256 hashes, but the X-Amz-Content-Sha256 header may be transmitted as base64. Changes: - Added normalizePayloadHash() function to convert base64 to hex - Call normalizePayloadHash() in extractV4AuthInfoFromHeader() - Added encoding/base64 import Fixes 403 Forbidden errors on POST requests to Iceberg REST API when clients send base64-encoded content hashes in the header. Impacted services: Iceberg REST API, S3Tables * Fix AWS SigV4 signature verification for base64-encoded payload hashes AWS SigV4 canonical requests must use hex-encoded SHA256 hashes, but the X-Amz-Content-Sha256 header may be transmitted as base64. Changes: - Added normalizePayloadHash() function to convert base64 to hex - Call normalizePayloadHash() in extractV4AuthInfoFromHeader() - Added encoding/base64 import - Removed unused fmt import Fixes 403 Forbidden errors on POST requests to Iceberg REST API when clients send base64-encoded content hashes in the header. Impacted services: Iceberg REST API, S3Tables * pass sigv4 * s3api: fix identity preservation and logging levels - Ensure environment-based identities are preserved during config replacement - Update accessKeyIdent and nameToIdentity maps correctly - Downgrade informational logs to V(2) to reduce noise * test: fix trino integration test and s3 policy test - Pin Trino image version to 479 - Fix port binding to 0.0.0.0 for Docker connectivity - Fix S3 policy test hang by correctly assigning MiniClusterCtx - Improve port finding robustness in policy tests * ci: pre-pull trino image to avoid timeouts - Pull trinodb/trino:479 after Docker setup - Ensure image is ready before integration tests start * iceberg: remove unused checkAuth and improve logging - Remove unused checkAuth method - Downgrade informational logs to V(2) - Ensure loggingMiddleware uses a status writer for accurate reported codes - Narrow catch-all route to avoid interfering with other subsystems * iceberg: fix build failure by removing unused s3api import * Update iceberg.go * use warehouse * Update trino_catalog_test.go	2026-02-06 13:12:25 -08:00
Chris LuandGitHub	b244bb58aa	s3tables: redesign Iceberg REST Catalog using iceberg-go and automate integration tests (#8197) ... * full integration with iceberg-go * Table Commit Operations (handleUpdateTable) * s3tables: fix Iceberg v2 compliance and namespace properties This commit ensures SeaweedFS Iceberg REST Catalog is compliant with Iceberg Format Version 2 by: - Using iceberg-go's table.NewMetadataWithUUID for strict v2 compliance. - Explicitly initializing namespace properties to empty maps. - Removing omitempty from required Iceberg response fields. - Fixing CommitTableRequest unmarshaling using table.Requirements and table.Updates. * s3tables: automate Iceberg integration tests - Added Makefile for local test execution and cluster management. - Added docker-compose for PyIceberg compatibility kit. - Added Go integration test harness for PyIceberg. - Updated GitHub CI to run Iceberg catalog tests automatically. * s3tables: update PyIceberg test suite for compatibility - Updated test_rest_catalog.py to use latest PyIceberg transaction APIs. - Updated Dockerfile to include pyarrow and pandas dependencies. - Improved namespace and table handling in integration tests. * s3tables: address review feedback on Iceberg Catalog - Implemented robust metadata version parsing and incrementing. - Ensured table metadata changes are persisted during commit (handleUpdateTable). - Standardized namespace property initialization for consistency. - Fixed unused variable and incorrect struct field build errors. * s3tables: finalize Iceberg REST Catalog and optimize tests - Implemented robust metadata versioning and persistence. - Standardized namespace property initialization. - Optimized integration tests using pre-built Docker image. - Added strict property persistence validation to test suite. - Fixed build errors from previous partial updates. * Address PR review: fix Table UUID stability, implement S3Tables UpdateTable, and support full metadata persistence individually * fix: Iceberg catalog stable UUIDs, metadata persistence, and file writing - Ensure table UUIDs are stable (do not regenerate on load). - Persist full table metadata (Iceberg JSON) in s3tables extended attributes. - Add `MetadataVersion` to explicitly track version numbers, replacing regex parsing. - Implement `saveMetadataFile` to persist metadata JSON files to the Filer on commit. - Update `CreateTable` and `UpdateTable` handlers to use the new logic. * test: bind weed mini to 0.0.0.0 in integration tests to fix Docker connectivity * Iceberg: fix metadata handling in REST catalog - Add nil guard in createTable - Fix updateTable to correctly load existing metadata from storage - Ensure full metadata persistence on updates - Populate loadTable result with parsed metadata * S3Tables: add auth checks and fix response fields in UpdateTable - Add CheckPermissionWithContext to UpdateTable handler - Include TableARN and MetadataLocation in UpdateTable response - Use ErrCodeConflict (409) for version token mismatches * Tests: improve Iceberg catalog test infrastructure and cleanup - Makefile: use PID file for precise process killing - test_rest_catalog.py: remove unused variables and fix f-strings * Iceberg: fix variable shadowing in UpdateTable - Rename inner loop variable `req` to `requirement` to avoid shadowing outer request variable * S3Tables: simplify MetadataVersion initialization - Use `max(req.MetadataVersion, 1)` instead of anonymous function * Tests: remove unicode characters from S3 tables integration test logs - Remove unicode checkmarks from test output for cleaner logs * Iceberg: improve metadata persistence robustness - Fix MetadataLocation in LoadTableResult to fallback to generated location - Improve saveMetadataFile to ensure directory hierarchy existence and robust error handling	2026-02-03 15:30:04 -08:00
Chris LuandGitHub	2bb21ea276	feat: Add Iceberg REST Catalog server and admin UI (#8175) ... * feat: Add Iceberg REST Catalog server Implement Iceberg REST Catalog API on a separate port (default 8181) that exposes S3 Tables metadata through the Apache Iceberg REST protocol. - Add new weed/s3api/iceberg package with REST handlers - Implement /v1/config endpoint returning catalog configuration - Implement namespace endpoints (list/create/get/head/delete) - Implement table endpoints (list/create/load/head/delete/update) - Add -port.iceberg flag to S3 standalone server (s3.go) - Add -s3.port.iceberg flag to combined server mode (server.go) - Add -s3.port.iceberg flag to mini cluster mode (mini.go) - Support prefix-based routing for multiple catalogs The Iceberg REST server reuses S3 Tables metadata storage under /table-buckets and enables DuckDB, Spark, and other Iceberg clients to connect to SeaweedFS as a catalog. * feat: Add Iceberg Catalog pages to admin UI Add admin UI pages to browse Iceberg catalogs, namespaces, and tables. - Add Iceberg Catalog menu item under Object Store navigation - Create iceberg_catalog.templ showing catalog overview with REST info - Create iceberg_namespaces.templ listing namespaces in a catalog - Create iceberg_tables.templ listing tables in a namespace - Add handlers and routes in admin_handlers.go - Add Iceberg data provider methods in s3tables_management.go - Add Iceberg data types in types.go The Iceberg Catalog pages provide visibility into the same S3 Tables data through an Iceberg-centric lens, including REST endpoint examples for DuckDB and PyIceberg. * test: Add Iceberg catalog integration tests and reorg s3tables tests - Reorganize existing s3tables tests to test/s3tables/table-buckets/ - Add new test/s3tables/catalog/ for Iceberg REST catalog tests - Add TestIcebergConfig to verify /v1/config endpoint - Add TestIcebergNamespaces to verify namespace listing - Add TestDuckDBIntegration for DuckDB connectivity (requires Docker) - Update CI workflow to use new test paths * fix: Generate proper random UUIDs for Iceberg tables Address code review feedback: - Replace placeholder UUID with crypto/rand-based UUID v4 generation - Add detailed TODO comments for handleUpdateTable stub explaining the required atomic metadata swap implementation * fix: Serve Iceberg on localhost listener when binding to different interface Address code review feedback: properly serve the localhost listener when the Iceberg server is bound to a non-localhost interface. * ci: Add Iceberg catalog integration tests to CI Add new job to run Iceberg catalog tests in CI, along with: - Iceberg package build verification - Iceberg unit tests - Iceberg go vet checks - Iceberg format checks * fix: Address code review feedback for Iceberg implementation - fix: Replace hardcoded account ID with s3_constants.AccountAdminId in buildTableBucketARN() - fix: Improve UUID generation error handling with deterministic fallback (timestamp + PID + counter) - fix: Update handleUpdateTable to return HTTP 501 Not Implemented instead of fake success - fix: Better error handling in handleNamespaceExists to distinguish 404 from 500 errors - fix: Use relative URL in template instead of hardcoded localhost:8181 - fix: Add HTTP timeout to test's waitForService function to avoid hangs - fix: Use dynamic ephemeral ports in integration tests to avoid flaky parallel failures - fix: Add Iceberg port to final port configuration logging in mini.go * fix: Address critical issues in Iceberg implementation - fix: Cache table UUIDs to ensure persistence across LoadTable calls The UUID now remains stable for the lifetime of the server session. TODO: For production, UUIDs should be persisted in S3 Tables metadata. - fix: Remove redundant URL-encoded namespace parsing mux router already decodes %1F to \x1F before passing to handlers. Redundant ReplaceAll call could cause bugs with literal %1F in namespace. * fix: Improve test robustness and reduce code duplication - fix: Make DuckDB test more robust by failing on unexpected errors Instead of silently logging errors, now explicitly check for expected conditions (extension not available) and skip the test appropriately. - fix: Extract username helper method to reduce duplication Created getUsername() helper in AdminHandlers to avoid duplicating the username retrieval logic across Iceberg page handlers. * fix: Add mutex protection to table UUID cache Protects concurrent access to the tableUUIDs map with sync.RWMutex. Uses read-lock for fast path when UUID already cached, and write-lock for generating new UUIDs. Includes double-check pattern to handle race condition between read-unlock and write-lock. * style: fix go fmt errors * feat(iceberg): persist table UUID in S3 Tables metadata * feat(admin): configure Iceberg port in Admin UI and commands * refactor: address review comments (flags, tests, handlers) - command/mini: fix tracking of explicit s3.port.iceberg flag - command/admin: add explicit -iceberg.port flag - admin/handlers: reuse getUsername helper - tests: use 127.0.0.1 for ephemeral ports and os.Stat for file size check * test: check error from FileStat in verify_gc_empty_test	2026-02-02 23:12:13 -08:00
3 people	621834d96a	s3tables: add Iceberg file layout validation for table buckets (#8176) ... * s3tables: add Iceberg file layout validation for table buckets This PR adds file layout validation for table buckets to enforce Apache Iceberg table structure. Files uploaded to table buckets must conform to the expected Iceberg layout: - metadata/ directory: contains metadata files (*.json, *.avro) - v*.metadata.json (table metadata) - snap-*.avro (snapshot manifests) - *-m*.avro (manifest files) - version-hint.text - data/ directory: contains data files (*.parquet, *.orc, *.avro) - Supports partition paths (e.g., year=2024/month=01/) - Supports bucket subdirectories The validator exports functions for use by the S3 API: - IsTableBucketPath: checks if a path is under /table-buckets/ - GetTableInfoFromPath: extracts bucket/namespace/table from path - ValidateTableBucketUpload: validates file layout for table bucket uploads - ValidateTableBucketUploadWithClient: validates with filer client access Invalid uploads receive InvalidIcebergLayout error response. * Address review comments: regex performance, error handling, stricter patterns * Fix validateMetadataFile and validateDataFile to handle subdirectories and directory creation * Fix error handling, metadata validation, reduce code duplication * Fix empty remainingPath handling for directory paths * Refactor: unify validateMetadataFile and validateDataFile * Refactor: extract UUID pattern constant * fix: allow Iceberg partition and directory paths without trailing slashes Modified validateFile to correctly handle directory paths that do not end with a trailing slash. This ensures that paths like 'data/year=2024' are validated as directories if they match partition or subdirectory patterns, rather than being incorrectly rejected as invalid files. Added comprehensive test cases for various directory and partition path combinations. * refactor: use standard path package and idiomatic returns Simplified directory and filename extraction in validateFile by using the standard path package (aliased as pathpkg). This improves readability and avoids manual string manipulation. Also updated GetTableInfoFromPath to use naked returns for named return values, aligning with Go conventions for short functions. * feat: enforce strict Iceberg top-level directories and metadata restrictions Implemented strict validation for Iceberg layout: - Bare top-level keys like 'metadata' and 'data' are now rejected; they must have a trailing slash or a subpath. - Subdirectories under 'metadata/' are now prohibited to enforce the flat structure required by Iceberg. - Updated the test suite with negative test cases and ensured proper formatting. * feat: allow table root directory markers in ValidateTableBucketUpload Modified ValidateTableBucketUpload to short-circuit and return nil when the relative path within a table is empty. This occurs when a trailing slash is used on the table directory (e.g., /table-buckets/mybucket/myns/mytable/). Added a test case 'table dir with slash' to verify this behavior. * test: add regression cases for metadata subdirs and table markers Enforced a strictly flat structure for the metadata directory by removing the "directory without trailing slash" fallback in validateFile for metadata. Added regression test cases: - metadata/nested (must fail) - /table-buckets/.../mytable/ (must pass) Verified all tests pass. * feat: reject double slashes in Iceberg table paths Modified validateDirectoryPath to return an error when encountering empty path segments, effectively rejecting double slashes like 'data//file.parquet'. Updated validateFile to use manual path splitting instead of the 'path' package for intermediate directories to ensure redundant slashes are not auto-cleaned before validation. Added regression tests for various double slash scenarios. * refactor: separate isMetadata logic in validateDirectoryPath Following reviewer feedback, refactored validateDirectoryPath to explicitly separate the handling of metadata and data paths. This improves readability and clarifies the function's intent while maintaining the strict validation rules and double-slash rejection previously implemented. * feat: validate bucket, namespace, and table path segments Updated ValidateTableBucketUpload to ensure that bucket, namespace, and table segments in the path are non-empty. This prevents invalid paths like '/table-buckets//myns/mytable/...' from being accepted during upload. Added regression tests for various empty segment scenarios. * Update weed/s3api/s3tables/iceberg_layout.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: block double-slash bypass in table relative paths Added a guard in ValidateTableBucketUpload to reject tableRelativePath if it starts with a '/' or contains '//'. This ensures that paths like '/table-buckets/b/ns/t//data/file.parquet' are properly rejected and cannot bypass the layout validation. Added regression tests to verify. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2026-02-02 10:05:04 -08:00
Chris LuandGitHub	79722bcf30	Add s3tables shell and admin UI (#8172) ... * Add shared s3tables manager * Add s3tables shell commands * Add s3tables admin API * Add s3tables admin UI * Fix admin s3tables namespace create * Rename table buckets menu * Centralize s3tables tag validation * Reuse s3tables manager in admin * Extract s3tables list limit * Add s3tables bucket ARN helper * Remove write middleware from s3tables APIs * Fix bucket link and policy hint * Fix table tag parsing and nav link * Disable namespace table link on invalid ARN * Improve s3tables error decode * Return flag parse errors for s3tables tag * Accept query params for namespace create * Bind namespace create form data * Read s3tables JS data from DOM * s3tables: allow empty region ARN * shell: pass s3tables account id * shell: require account for table buckets * shell: use bucket name for namespaces * shell: use bucket name for tables * shell: use bucket name for tags * admin: add table buckets links in file browser * s3api: reuse s3tables tag validation * admin: harden s3tables UI handlers * fix admin list table buckets * allow admin s3tables access * validate s3tables bucket tags * log s3tables bucket metadata errors * rollback table bucket on owner failure * show s3tables bucket owner * add s3tables iam conditions * Add s3tables user permissions UI * Authorize s3tables using identity actions * Add s3tables permissions to user modal * Disambiguate bucket scope in user permissions * Block table bucket names that match S3 buckets * Pretty-print IAM identity JSON * Include tags in s3tables permission context * admin: refactor S3 Tables inline JavaScript into a separate file * s3tables: extend IAM policy condition operators support * shell: use LookupEntry wrapper for s3tables bucket conflict check * admin: handle buildBucketPermissions validation in create/update flows	2026-01-30 22:57:05 -08:00
3 people	f1e27b8f30	s3: change s3 tables to use RESTful API (#8169) ... * s3: refactor s3 tables to use RESTful API * test/s3tables: guard empty namespaces * s3api: document tag parsing and validate get-table * s3api: limit S3Tables REST body size * Update weed/s3api/s3api_tables.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3tables/handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * s3api: accept encoded table bucket ARNs * s3api: validate namespaces and close body * s3api: match encoded table bucket ARNs * s3api: scope table bucket ARN routes * s3api: dedupe table bucket request builders * test/s3tables: allow list tables without namespace * s3api: validate table params and tag ARN * s3api: tighten tag handling and get-table params * s3api: loosen tag ARN route matching * Fix S3 Tables REST routing and tests * Adjust S3 Tables request parsing * Gate S3 Tables target routing * Avoid double decoding namespaces --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2026-01-30 10:37:34 -08:00
Chris Lu	88c27615c4	/table-buckets	2026-01-29 20:03:17 -08:00
Chris Lu	d399113e0c	test: fix duplicate subtest names in permissions_test.go ... Rename duplicate 'combined * and ?' test cases to include singular/plural suffix for clarity and to support targeted test runs.	2026-01-28 19:42:19 -08:00
Chris Lu	a4217dff5f	s3tables: enhance DeleteTable authorization with policy checking ... Fetch and evaluate table policies in DeleteTable handler to support policy-based delegation. Aligns authorization behavior with GetTable and ListTables handlers instead of only checking ownership.	2026-01-28 19:42:12 -08:00
Chris Lu	745a7e40a6	s3tables: improve bucket policy error handling in DeleteTableBucket ... Explicitly handle ErrAttributeNotFound vs other errors when fetching bucket policy. Return errors for non-expected failures to prevent masking filer issues and ensure correct authorization decisions.	2026-01-28 19:42:08 -08:00
Chris Lu	d5ce6a4cda	s3tables: refactor bucket name validation into single function ... Combine length, character, and reserved pattern validation into validateBucketName() which returns descriptive error messages. Keep isValidBucketName() for backward compatibility. This simplifies handler validation and provides better error reporting.	2026-01-28 19:42:01 -08:00
Chris Lu	549b65785d	refactor	2026-01-28 18:55:16 -08:00
Chris Lu	590e7efbef	s3tables: Separate table name pattern constant for clarity ... Define a separate tableNamePatternStr constant for the table name component in the ARN regex, even though it currently has the same value as tableNamespacePatternStr. This improves code clarity and maintainability, making it easier to modify if the naming rules for tables and namespaces diverge in the future.	2026-01-28 18:40:02 -08:00
Chris Lu	78c00e313a	go fmt	2026-01-28 18:34:32 -08:00
Chris Lu	f5d26b803b	s3tables: Fix ListTables authorization and policy parsing ... Make ListTables authorization consistent with GetTable/CreateTable: 1. ListTables authorization now evaluates policies instead of owner-only checks: - For namespace listing: checks namespace policy AND bucket policy - For bucket-wide listing: checks bucket policy - Uses CanListTables permission framework 2. Remove owner-only filter in listTablesWithClient that prevented policy-based sharing of tables. Authorization is now enforced at the handler level, so all tables in the namespace/bucket are returned to authorized callers (who have access either via ownership or policy). 3. Add flexible PolicyDocument.UnmarshalJSON to support both single-object and array forms of Statement field: - Handles: {"Statement": {...}} - Handles: {"Statement": [{...}, {...}]} - Improves AWS IAM compatibility This ensures cross-account table listing works when delegated via bucket/namespace policies, consistent with the authorization model for other operations.	2026-01-28 18:27:37 -08:00