mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 20:26:45 +00:00
Chris LuandGitHub
d044839ab2
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
38 lines
1.3 KiB
Go
38 lines
1.3 KiB
Go
package s3tables
|
|
|
|
import (
|
|
"path"
|
|
"testing"
|
|
)
|
|
|
|
func TestTableDataDirFromMetadataLocation(t *testing.T) {
|
|
cases := []struct {
|
|
loc string
|
|
want string
|
|
}{
|
|
{"s3://warehouse/sales/orders/metadata/v1.metadata.json", path.Join(TablesPath, "warehouse/sales/orders")},
|
|
{"s3://warehouse/sales/orders/metadata/00003-9f1c.metadata.json", path.Join(TablesPath, "warehouse/sales/orders")},
|
|
{"s3://warehouse/ns/tbl", path.Join(TablesPath, "warehouse/ns/tbl")},
|
|
{"", ""},
|
|
}
|
|
for _, c := range cases {
|
|
if got := TableDataDirFromMetadataLocation(c.loc); got != c.want {
|
|
t.Errorf("TableDataDirFromMetadataLocation(%q) = %q, want %q", c.loc, got, c.want)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestMetadataVersionFromLocationUniqueSuffix(t *testing.T) {
|
|
cases := map[string]int{
|
|
"s3://bkt/ns/t/metadata/v4.metadata.json": 4,
|
|
"s3://bkt/ns/t/metadata/v4-0a1b2c3d-4e5f-6789-abcd-ef0123456789.metadata.json": 4,
|
|
"s3://bkt/ns/t/metadata/00007-0a1b2c3d.metadata.json": 7,
|
|
"s3://bkt/ns/t/metadata/whatever.metadata.json": 1,
|
|
}
|
|
for location, want := range cases {
|
|
if got := metadataVersionFromLocation(location); got != want {
|
|
t.Errorf("metadataVersionFromLocation(%q) = %d, want %d", location, got, want)
|
|
}
|
|
}
|
|
}
|