* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
The worker assumed every file of a table sits under its catalog path, so
loadFileByIcebergPath stripped the scheme off a recorded location and joined
the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog
placed elsewhere in the bucket — which is what a client gets whenever the
catalog path is already occupied — then resolves to a doubled path:
lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro
so the very first manifest list read fails and the job fails again on every
scan interval, indefinitely.
Resolve absolute references (s3:// URIs and /buckets paths) from the bucket
root and keep relative ones under the table's own directory; the
bucket-relative form is now the canonical key everywhere references are
compared. That directory comes from the metadata location the catalog stores,
so reads, writes and deletes all land where the table's other files are
instead of splitting it across two trees. References outside the table's
bucket are rejected rather than silently misresolved.
Rewritten position-delete files now name their data file by absolute URI,
the way the table itself names it, instead of a path relative to the table.
* s3tables: add RenameTable operation
* iceberg: support table rename
* iceberg: test table rename
* s3tables: keep table data in place on rename
rename is catalog-only: drop the source's catalog xattrs in place instead of recursively deleting its directory, which wiped the metadata.json and data files the renamed destination still points at. treat a missing table-metadata xattr as NoSuchTable in GetTable so the soft-deleted source name stops resolving.
* s3tables: test rename preserves data
make the in-memory filer honor recursive data deletion and seed the source table's metadata/ and data/ children, then assert a rename leaves them intact, the source name resolves to NoSuchTable, and the destination resolves to the preserved location.
* iceberg: map rename errors through wrapped manager error
* s3tables: authorize rename destination namespace
rename moved a table into the destination namespace after only checking the source, letting a source-authorized caller place tables in namespaces they don't control. require CreateTable on the destination namespace and bucket before writing.
* s3tables: purge renamed table data on drop
* s3tables: test table data dir derivation
Chris LuandGitHub