mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 04:06:44 +00:00
0dd33bd7ec012096dffcdfe66b2c5eb2cb069edb
456
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
3a61debaa5 |
filer: rebuild peer metadata subscriptions after a master reconnect (#10648)
* filer: keep the existing peer subscription on a repeated add A cluster node add for a peer that is already followed restarted the subscription, dropping the metadata events between the two runs. * master: tell a connecting client the current cluster membership Cluster node updates are only broadcast to the clients connected at that moment. A filer that lost its master stream while a peer came back never learned about the peer, and stopped replicating its metadata for good. * test: a filer joining the master learns about the filers already there * test: a filer resubscribes to a peer that registered while it was disconnected Runs the reported sequence against real processes: filer2 leaves, filer1 is paused and its master stream is broken, filer2 registers again, and filer1 has to replicate from it after reconnecting. |
||
|
Chris LuandGitHub
|
37f3dff677 |
volume: validate the file extension in CopyFile and ReceiveFile (#10644)
* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
|
||
|
Chris LuandGitHub
|
213eb4c23a |
s3tables: add ClickHouse iceberg catalog integration test (#10637)
* s3tables: add ClickHouse iceberg catalog integration test * ci: run the ClickHouse iceberg catalog test * s3tables: bound setup HTTP calls in the ClickHouse test * s3tables: pin the ClickHouse writer image dependencies |
||
|
Chris LuandGitHub
|
4527947afc |
mount: absorb the WinFsp metadata cache window in the concurrent-reader test (#10636)
WriteFile's own existence probe runs while the file does not exist, and WinFsp may serve that answer from its metadata cache for up to the mount's FileInfoTimeout. A reader racing into that window failed its open with not-found, which is the cache being a cache, not a defect in concurrent reading. Establish visibility once before racing the readers, so the test exercises what it is named for. |
||
|
Chris LuandGitHub
|
a5e8254ffd |
s3: give a versioned metadata-only copy its own chunks (#10594)
* s3: give a versioned metadata-only copy its own chunks A self-copy that only rewrites metadata clones the source entry, chunk fids and all, and writes the clone back. With no versioning that is exactly right: the clone replaces the entry it came from, so one entry owns the needles the whole time. Under versioning the clone lands in a new .versions/ file and the source stays live, and nothing refcounts a plain shared chunk list -- deleting either version (a NoncurrentVersionExpiration rule, say) frees needles the other still points at, and the next vacuum makes that permanent. rclone hits this on every upload, since it stamps mtime with exactly this copy. Take the metadata-only path only where the write replaces the entry it read: the bare key of a bucket without versioning. Versioned, suspended, and versionId-pinned copies fall through to the regular copy path, which gives the destination its own chunks. * s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks A same-object copy that changes the KMS key id hands the source chunks straight back, on the assumption that the copy overwrites the entry they came from. A versioned bucket writes a new version beside the source instead, so the two end up sharing needles that nothing refcounts, and deleting either one frees the other's data. Reuse the chunks only when the destination really is the source entry; otherwise fall through to the reencrypt path, which also gives the new version the key it asked for rather than leaving it on the old one. * s3: make one predicate decide whether a copy replaces its source The metadata-only branch and the key-rotation strategy both answer the same question -- does this copy write back to the entry it read -- so let them share one predicate instead of pairing a same-destination check with it separately at each site. * test(s3): fail the copy regression tests when the vacuum does not run The helper swallowed a failed or non-200 request to the master, so a vacuum that never ran turned both chunk-ownership assertions into no-ops: the tombstoned needles were still readable and the surviving version looked fine either way. Require the endpoint, the request, and a 200. * ci(s3): run every versioning test in the regression gate The gate named the tests it wanted, so a new regression test sat there uncovered until someone remembered this file -- it fooled me into thinking two tests added in this PR never ran anywhere, when the comprehensive job had them all along. Invert it: run everything, and name a test only to keep it out. The delete job beside this one already works that way, and the suite costs about two minutes. Only the pagination stress tests are excluded; they build 1500+ versions, skip themselves without ENABLE_STRESS_TESTS, and have their own make target. Go's regexp has no negation, so the pattern is still assembled from a listing, the way the volume-server integration workflow does it. Note the trailing $$: make eats a lone trailing $ and takes the anchor with it. |
||
|
Chris LuandGitHub
|
c2b47967bd |
s3: retire the suspended null marker only once the PUT has committed (#10589)
The suspended PUT dropped the null delete marker before writing, so a failed write left the .versions pointer naming a marker that was gone. The read path heals a dangling pointer by promoting the newest survivor, so a key the caller had deleted came back serving an older version, and the heal persisted that pointer. Move the retire into afterCreate via the shared finalize, which also brings the ownership check the copy and multipart paths already have. |
||
|
Chris LuandGitHub
|
f09bc14165 |
s3: report the effective ownership when a bucket has none stored (#10591)
* s3: report the effective ownership when a bucket has none stored GetBucketOwnershipControls read Seaweed-X-Amz-Ownership straight out of the bucket entry, so a bucket that never had one written reported an empty ObjectOwnership. The object write path defaults the same missing attribute to BucketOwnerEnforced, so the API contradicted the behavior it describes. Resolve the stored value through one helper both readers share, and let PutBucketOwnershipControls persist unconditionally so setting the default value still gives DeleteBucketOwnershipControls something to remove. * test: cover the bucket ownership controls round trip Pins the behaviors the ownership default fix depends on: a bucket that never had ownership controls written reports BucketOwnerEnforced, and putting that same value on such a bucket still persists it, so the delete that follows has something to remove. The put-then-delete case gets its own bucket -- run after an ObjectWriter put, it would pass against an implementation that skips only the initial write. The acl workflow already runs this package against a live weed mini, so it needs no wiring. |
||
|
Chris LuandGitHub
|
5269d93fa8 |
s3: let a suspended-versioning multipart completion replace the null delete marker (#10585)
* s3: let a suspended-versioning multipart completion replace the null delete marker In a versioning-suspended bucket a DELETE writes a null delete marker into the key's .versions directory. CompleteMultipartUpload then writes the new null version at the regular path but left that marker in place, so the completion returned 200 and the object listed while HEAD and GET kept resolving the marker and answered NoSuchKey. PutObject already handles this; do the same on the multipart path. * s3: order the suspended-versioning null cleanup behind the multipart write Removing the null delete marker before writing left a failed completion having already published the key's newest real version: the marker was gone, the pointer still named it, so reads rescanned .versions and promoted the older version. Do both fixups only once the write commits, pointer first so reads never see a pointer aimed at a marker that is no longer there, and fail the completion when the pointer cannot be cleared instead of returning 200 for an object HEAD and GET still miss - a non-ErrNone finalize keeps the upload directory, so the caller's retry replays it. Also cover a pre-suspension real version in the regression test. * s3: skip the suspended null cleanup when a concurrent write won the key The completion's .versions fixups are unconditional rewrites of shared state and the routed path runs off the object write lock, so a DELETE landing between the multipart write and the cleanup had its own null delete marker erased - leaving a successfully deleted key readable as an older retained version. Re-read the object first and leave the cleanup alone unless it is still the one we wrote. This narrows the window rather than closing it; a compare-and-set pointer flip is the real answer and wants its own change. * s3: re-read the completed object from the filer that took the write The guard compared the object against our upload id through the routed read, which skips an owner it recently found unreachable and falls back local-first. A write that just landed on the owner could then read as superseded on another filer, skipping the cleanup and leaving the key unreadable - the bug this set out to fix. Read back from the filer the write went to instead. * s3: trim the suspended-completion comments to the non-obvious why * s3: lift the suspended null-write finalize into a named helper The pointer-then-marker ordering is policy shared by every suspended null write, not something the multipart path should be stating on its own; putSuspendedVersioningObject and the copy path each restate it today. Give it a home next to the versioned finalize helpers, and reuse the canonical key normalizer and the existing test helpers rather than open-coding both. * s3: retire the null delete marker on a suspended-versioning copy The suspended CopyObject branch cleared the .versions latest pointer but left the null delete marker a preceding DELETE wrote. While the regular-path object owns the null slot that marker is shadowed, so it reads and lists correctly - but it resurfaces as a phantom delete for a key nobody deleted once that null version goes away. Route the branch through the shared finalize. * s3: keep the suspended null cleanup from erasing a concurrent delete Retiring the marker on the copy path reopened the race the multipart path had already closed: a DELETE landing between the write and the cleanup lost its own marker, so a rescan promoted an older version under a deleted key. Move the ownership check into the shared finalize, keyed on the attribute that identifies the caller's write, so both paths get it. |
||
|
Chris LuandGitHub
|
7063b3e14c |
s3 lifecycle: bound the daily-replay pass so a quiet cluster stops wedging the job (#10578)
* s3 lifecycle: bound the daily-replay subscription at the pass boundary A pass opens one meta-log subscription and 16 shard drains, then waits on all of them. Nothing told the subscription where the pass ends, so the only exit was the fan-out spotting an event past runNow — i.e. some unrelated write landing under /buckets after the pass started. On a cluster that goes quiet the reader parks in Recv, every shard drain starves on an empty channel, and Run never returns. The job sits at stage "starting" with the executor slot held and no log line, so expiry stops cluster-wide until someone restarts the worker. The pass covers (globalStartTsNs, runNow], so say that: UntilNs on the subscribe request makes the filer end the stream once it has shipped that range. The reader then closes the event channel on the way out, which is what unblocks the fan-out and the drains when the stream finishes on its own rather than by cancellation. Same fix retires the other silent hang: a reader that failed early (subscribe error, stream error) also left every drain waiting forever. * s3 lifecycle: keep a halted shard from starving the shared fan-out A drain that halts mid-stream (BLOCKED / RETRY_LATER / an RPC error on dispatch) returns while the fan-out is still routing that shard's events. After 256 of them the per-shard buffer is full and the fan-out blocks on the send, so no other shard sees another event. Run's WaitGroup never drains, and the teardown that would cancel the reader sits behind that wait — the pass wedges exactly like an idle subscription did, with one S3 hiccup as the trigger. Keep discarding the channel after runShard returns. The events are past this shard's saved cursor and get re-scanned next pass anyway. * s3 lifecycle: assert the starved shard actually made progress The fan-out test only checked that Run returned, which a version that quietly dropped the second shard's events would also satisfy. Assert the dispatch landed and the cursor moved. recordingClient gains a per-object outcome map: the two shards dispatch from separate goroutines, so pinning BLOCKED by call index was a race waiting to pick the wrong shard. * s3 lifecycle: fail the pass when the shared subscription dies Closing the event channel on reader exit is what unblocks the shard drains, but it also means a subscribe that never opened, or a stream that broke mid-pass, now ends every drain cleanly. Run logged that at V(2) and returned the shard result — so a filer failure produced a green lifecycle job that had processed nothing. Surface it as the pass error. Cursors still hold what was processed and tomorrow resumes there; what changes is that the job stops claiming success. Cancellation has to stay a non-error — the shell driver's -runtime cap is a truncated pass, not a failed one — and a canceled gRPC stream arrives as a status code, not a wrapped context.Canceled, so isCanceled checks both forms the way the rest of the tree does. * s3 lifecycle: decide reader cancellation by intent, not status code A stream we cancel and a stream the filer cancels both arrive as codes.Canceled, so classifying the reader's exit by its error let a truncated pass report success whenever the failure happened to carry a cancellation status. Intent is knowable exactly, so read that instead: the pass stops on purpose only when the caller's context ended (the shell driver's -runtime cap) or the fan-out hit the pass boundary itself. Everything else is a broken subscription and fails the pass. TestRun_ServerSideCancelFailsThePass and TestRun_CappedPassIsNotAFailure are the same codes.Canceled from the reader with opposite verdicts — the pair only passes because the decision no longer looks at the error. * s3 lifecycle: time out a subscription that stops delivering UntilNs ends a healthy stream and gRPC keepalive catches a dead connection, but neither reaches a filer that keeps answering pings while its handler has stopped producing. The pass would wait on that forever, since s3_lifecycle is the one job type with no execution timeout. Bound the wait for each response at 20 minutes, and opt into the filer's idle heartbeats so a caught-up stream proves liveness instead of looking stalled. The default sits above the filer's 15-minute metadata-gap recovery budget, so a subscriber legitimately parked on a gap is never mistaken for a stalled one. Recv is only interruptible by killing the RPC, so it moves to its own goroutine behind a per-response deadline. The timer covers only the wait on the filer — dispatch to Events happens outside it, so a slow consumer can't trip the watchdog. Approach and the 20-minute figure are from #10577 by way of comparing the two fixes; the wiring differs because the reader here ends the pass by closing its event channel rather than cancelling the fan-out. * s3 lifecycle: trim the comments added by this branch Keep the non-obvious why, drop the prose restating what the code says. * s3 lifecycle: snapshot reader intent where the reader stops Sampling ctx.Err() during teardown reads it after the drains and cursor saves have run. A reader that failed while the deadline was still live, on a pass whose teardown then outlives that deadline, was classified as an intentional stop and reported success. Sampling earlier in Run is not the fix either: before the shard wait, a legitimately capped pass has not reached its deadline yet and would be misclassified the other way. Intent belongs where the reader actually stops, so the reader goroutine records it next to the error it returns. Reported by greptile on #10578. * s3 lifecycle: cover the worker-dispatched pass with nothing due The e2e suite drives the shell command in 14 of 15 files; the one test on the real admin->worker path backdates an object, so its own delete pushes a meta-log event past the pass boundary and ends the pass. The branch where a pass has nothing to dispatch was never exercised through the worker. Cover it, asserting the pass returns on its own: no admin cancellation, and the executor slot free for the next one. This is not a regression test for the wedge. A pass used to end when any write landed past its boundary, and on a shared test cluster something usually does — the whole suite passes on the unfixed build, verified. The deterministic guards stay the dailyrun unit tests; this one would catch a pass that hangs unconditionally. |
||
|
Chris LuandGitHub
|
7d6c83b126 |
s3: stop treating a directory marker as a versioned object (#10573)
* s3: delete a directory marker instead of versioning it The key "dir/" is stored as the filer directory itself, so a delete marker cannot stand in for it without hiding the children underneath, and its history has to sit inside the directory it describes, where listings keep meeting it. Delete it the way an unversioned bucket already does: remove the directory when nothing is left under it, demote it to a plain directory when children remain, and drop a history an older build recorded for it. * s3: stop resolving directory markers through a version history Nothing records one for them any more, so the lookups that read it are dead weight - and the one in the listing was a filer round trip per directory marker returned, which for a bucket that keeps a marker per directory is the whole listing cost. A listing reads what a directory stands for straight off the entry it already has; a unit test pins that N markers cost one ListEntries rather than N+1. The guard that keeps a history left inside a directory by an older build from surfacing as a key named after it stays. * s3: do not let deleting "dir/" destroy the object at "dir" Writing under an existing object turns that object's entry into a directory while it keeps its data, so the keys "m2" and "m2/" end up sharing one entry. Stripping the entry to delete "m2/" therefore wiped the object at "m2" - a different key, and in a versioned bucket one no delete marker records. Leave a directory holding uploaded data alone; "m2/" does not name it. * s3: make the directory-marker delete fail closed and take the write lock The guard that spares a promoted file only fired when the entry read succeeded, so a transient filer error fell through to the delete and could destroy the object at "dir" anyway. Fail the request instead, take the object write lock so the entry cannot change between the check and the delete, and report a stale history that cannot be removed rather than leaving it to keep naming the key in ListObjectVersions. * s3: check If-Match inside the directory-marker delete lock The lock belongs to the caller: taking it inside the delete nested it under the batch handler's own lock, and since every lock from a gateway shares one owner the inner release would have freed it while the outer caller still assumed it held it. Both callers now own the lock, the single-object path re-checks If-Match inside it the way the other delete paths do, and a batch delete of a trailing-slash key in an unversioned bucket goes through the same marker path instead of the raw delete. A history lookup that fails now fails the delete. |
||
|
Chris LuandGitHub
|
d448e9db7b |
iceberg: withhold the S3 endpoint from credential-vending clients (#10570)
* iceberg: withhold the S3 endpoint from credential-vending clients A client that sends X-Iceberg-Access-Delegation: vended-credentials builds its storage credential out of the LoadTable config and drops the one it was configured with. We vend no credentials, so the endpoint we advertised left DuckDB signing nothing: every metadata and data file came back 403, and its attempt to refresh the empty credential 404ed on stage-created tables. Answer those clients with no config at all so they keep their own credentials. Clients that do not ask for delegation still get the endpoint. * iceberg: mark load responses as varying on the delegation header The FileIO config in a table or view load response now depends on whether the client asked for vended credentials, so a cache between us and the client must key on that header rather than on the URL alone. * test: cover the DuckDB vended-credentials access pattern Runs weed mini with -s3.externalUrl, which is what makes the catalog advertise an endpoint at all, and checks both halves: a plain LoadTable still gets the endpoint, while one asking for vended credentials never gets an endpoint without the credentials to sign with. The DuckDB round trip creates a table from a query and reads it back, which is the flow that failed with 403 on every data file. |
||
|
Chris LuandGitHub
|
474a0713b0 |
s3: honor the version history of a directory marker (#10571)
* s3: stop listing a directory marker whose latest version is a delete marker A directory marker is stored as the filer directory itself, so deleting the key "dir/" writes its delete marker into dir/.versions while the directory keeps its mime and stays a key object. Every listing kept reporting the key. Consult that history before treating the entry as a key, and demote it in memory when it is delete-marked so live children still hold the prefix. Also skip the container's own .versions entry while listing inside it: the suffix match read it as the history of a nested object named "", which surfaces as a phantom dir/dir key as soon as a live directory version exists. * s3: a directory marker with version history is not also the latest null version The directory entry behind the key "dir/" is that key's null version, but list-object-versions reported it with IsLatest hardcoded true. After a delete the key came back twice, once as the delete marker and once as a null version, both claiming to be latest. Read the pointer under the directory instead. * s3: resolve directory markers through their version history on GET and HEAD GET and HEAD short-circuit any trailing-slash key straight to the filer directory, so a directory marker kept answering 200 after its delete marker was written. Resolve the key from dir/.versions first when the bucket is versioned: a delete-marked current version answers 404 with x-amz-delete-marker, a named delete-marker version answers 405, and a key with no history keeps today's directory-probe behavior untouched. * s3: re-creating a directory marker retires its delete marker PutObject on a trailing-slash key never looked at the bucket's versioning state, so re-creating a marker after a delete left the latest-version pointer on the delete marker and the key stayed invisible to every versioned read. Point the key back at the directory entry, which is its null version, and drop the null version .versions may still hold — the same two steps a suspended write already takes, now shared. * s3: fail a directory-marker request whose version history cannot be read Every lookup of dir/.versions treated any error as "no history", so a filer hiccup served the directory entry for a key whose current version may be a delete marker, reported a null version as latest over one, and let a PUT report success without retiring the delete marker it was meant to retire. Only a confirmed absence takes the no-history path now. * s3: cancel the directory-marker probe stream instead of abandoning it The probe answers off the first entry and returns, leaving the ListEntries stream open for the life of the parent context. Give it a context of its own. |
||
|
Chris LuandGitHub
|
d01ed36118 |
test: cover delete-on-close on the windows mount (#10561)
* test: cover delete-on-close on the windows mount Windows software creates temporaries with FILE_FLAG_DELETE_ON_CLOSE and never deletes them explicitly. The conformance suite showed a file outliving its last handle — an aborted test left its file behind and every later test hit a name collision — but nothing in this suite asks for the flag, because os offers no way to. Skips where the flag is unavailable rather than passing quietly. * test: fail delete-on-close on a real error instead of skipping Skipping on any error meant a refused flag looked the same as a platform that cannot ask for it, so the test could pass by never running. It now skips only on that one sentinel and reports everything else. Also stops printing a nil error when the file is still there after its last handle closed, and checks the closes it was discarding. |
||
|
Chris LuandGitHub
|
b1fecf3b44 |
mount: mark windows files archived and ignore a zero timestamp (#10559)
* mount: mark windows files archived and ignore a zero timestamp Windows synthesises NORMAL when a file reports no attributes at all, which is not the same as ARCHIVE and is what create_fileattr_test checks. Utimens also wrote a zero timestamp through. Windows sends zero for a field it is not setting, and storing it put 1970 in the atime overlay, which then overrode the entry's real time — so a file created a moment ago reported an access time of 1970 whenever the caller asked through an open handle. Reading the path instead went down a different route and looked right, which is why a probe of a fresh file showed nothing wrong. * mount: match the file type by its mask, and only treat the epoch as unset S_IFDIR is part of the multi-bit type field rather than a flag, so masking against it alone also matched a symlink, which shares the bit. A regular file is now identified by the type mask. Rejecting every timestamp at or below zero also rejected a date genuinely before 1970. Only the epoch itself is what Windows sends for a field it is not setting, so that is all that is refused. create_fileattr goes back on the known-failures list: the archive fix works and the test simply moves on to ask for READONLY too, which needs Chflags. Taking it off was premature. * mount: drop the time overlays when an inode is released atimeMap and dirMtimeMap are keyed by inode and were only ever trimmed by a random eviction at capacity. Inodes are derived from the path, so a delete and recreate hands the same number to a different file, which then reported the previous file's access time — a file created a moment ago answering with a time from long before it existed. Cleared when Forget actually releases the inode, not on every decrement: a partial forget still has users. Forget now reports that so callers holding state keyed by the inode know when to drop it. * ci: keep getfileinfo listed while its access time is unexplained Two causes have been fixed and neither closed it, so the honest state is listed-with-a-reason rather than removed in hope. * mount: drop timestamp overlays while the inode table is locked Forget released the inode under the table's lock but cleaned up the atime and dir-mtime overlays after returning from it. Inode numbers are derived from the path, so a lookup arriving in that window is handed the same number back and can store a time that the cleanup then deletes. Run the cleanup at the release point instead, as a callback under the lock. The directory-cache purge stays deferred until after the unlock, where it has to be. Claude-Session: https://claude.ai/code/session_01EgY2QA3iiPtiu6ww3P2EBn |
||
|
Chris LuandGitHub
|
edaee0e426 |
ci: run each conformance test on its own (#10564)
* ci: run each conformance test on its own Run as one batch with --no-abort, a test that fails part way leaves its files behind and the next one fails creating them, so the report showed a cascade of failures that were really one. The whole rdwr and flush group passes when run alone, and was only ever collateral. Each test now gets its own directory and its own invocation, which costs a process start per test and makes the list mean what it says. * ci: clear a case directory before reusing it -Force creates the directory but leaves anything already in it, so a leftover from an interrupted run would defeat the isolation this exists to provide. * ci: list the one real failure isolation exposed With each test on its own, 31 of 32 pass. The exception is rdwr_mmap_test, which compares mapped bytes against what was written and finds them different — a genuine data mismatch that only appeared once the test could run to completion instead of tripping over a previous one's leftovers. |
||
|
Chris LuandGitHub
|
5a5cd15054 |
mount: report . and .. from windows directories (#10556)
* mount: report . and .. from windows directories WinFsp strips the dot entries for the root itself and expects every other directory to report them, the way a real NTFS enumeration does: its dirctl test asserts a subdirectory's first two entries are "." and ".." and that a hundred files enumerate as 102 entries. Dropping them unconditionally is what fails querydir_test. The Go test that guarded the old behaviour went with it: os.File.Readdir filters dot entries itself, so it could never have observed either way. * mount: give the windows dot entries their directory type The readdir fills an attribute block only for real children, so "." and ".." arrived with a zeroed one and were reported with mode 0. Windows refuses to enumerate a directory whose first entry is not marked as a directory, which is the assertion querydir_test fails on with STATUS_OBJECT_NAME_NOT_FOUND. They now carry the type the readdir already knew. The explorer walk also names any unexpected entry rather than only counting, so a dot entry leaking through reads differently from a missing file. |
||
|
Chris LuandGitHub
|
89ce6e175d |
ci: run WinFsp's conformance suite against the windows mount (#10555)
* ci: run WinFsp's conformance suite against the windows mount
The FUSE mount is held to pjdfstest with an empty known-failures list;
the Windows mount had 24 hand-written tests. winfsp-tests is what WinFsp
uses to check a filesystem behaves like NTFS, and --fuse-external points
it at ours instead of the bundled memfs, so it is the same bar in the
same shape: anything failing that is not listed is a regression.
It reaches oplocks, security descriptors, POSIX unlink-and-rename and
directory-buffer resumption — the places a Windows filesystem actually
breaks, and none of which the current suite touches.
known_failures.txt starts with the four groups that cannot pass by
construction. The first run will show what else needs listing.
* ci: make the conformance runner fail loudly instead of running empty
The first run reported "0 excluded entries" and then died with
STATUS_DLL_NOT_FOUND, so it never tested anything while looking like a
normal failing run.
winfsp-tests links against winfsp-x64.dll, which the installer puts
somewhere the loader does not search, so the WinFsp bin directory goes on
PATH. A missing or empty known-failures list is now an error rather than
a silent run with nothing excluded, which would read as a clean sweep
with no known failures. ${env:ProgramFiles(x86)} needs the braces, and a
mount point without a trailing separator makes Join-Path build a path
relative to the drive's current directory rather than its root.
* ci: read winfsp-tests failures from its report, and list the real ones
The first run exited zero with 30 of 50 tests reporting KO, and the job
went green: --no-abort keeps the suite going past a failure and the exit
code stops reflecting them, so trusting it meant the check could not fail.
The report is now parsed for KO lines and each one named in the error.
known_failures.txt is populated from that run rather than guessed. The
groups are real gaps, not suite quirks: cached and overlapped IO fails as
a block, delete-while-open has no pending state, Windows file attributes
and creation time are not round-tripped, and directory enumeration does
not resume from a marker.
* ci: stop excluding the extended attribute tests
Forwarding landed, so the group runs instead of being taken on trust —
which is the only coverage it has had.
|
||
|
Chris LuandGitHub
|
e377149d39 |
mount: support mounting on Windows through WinFsp (#10536)
* mount: add the WinFsp filesystem adapter WinFsp speaks a path-based FUSE dialect; weed/mount implements the inode-based raw protocol the Linux kernel uses. This translates between them so Windows runs the same filesystem code as everywhere else rather than a second implementation: paths resolve to inodes one Lookup at a time, and the raw operations run unchanged underneath. Errno translation is spelled out rather than passed through. Go numbers Windows errnos as offsets from APPLICATION_ERROR, so the raw value would mean something unrelated by the time WinFsp read it. Hard links return ENOSYS since WinFsp has none, and byte-range locks stay with its kernel driver rather than the mount's lock table. Not reachable from the mount command yet. * mount: build the winfsp errno table with explicit precedence Platforms alias errnos differently: freebsd has no ENODATA and linux makes ENOATTR the same value as it. A map literal with colliding constant keys does not compile, so build the table and let the first entry win, keeping the general codes their own meaning. * mount: wire the winfsp adapter into the mount command RunMount was one function doing filer setup, mount-point preparation and serving. The setup is the same everywhere, so it moves to mount_common.go and each platform keeps only what differs. Windows differs mostly in the mount point: WinFsp wants a drive letter or a path that does not exist yet, so none of the unix preparation applies, and a bad one is worth rejecting up front because WinFsp reports failure as a bare false. Adds -windows.caseInsensitive for software that expects Windows naming rules. * ci: mount on windows and exercise it Builds weed.exe, installs WinFsp, starts a cluster, mounts S: and runs a test suite against it: round trips at several sizes, offset writes, rename, delete, nested directories, concurrent writers, and a directory wide enough to stand in for the case that prompted this. Nothing else here can run the Windows mount, so without this the adapter is only known to compile. * ci: build the windows mount without cgo The runner has MinGW, so cgo is on by default and cgofuse compiles its cgo variant, which needs WinFsp's headers. The nocgo variant loads the DLL at run time and is what the released weed.exe uses. * mount: make the winfsp path splitting portable and test it resolve and resolveParent had the splitting inline in a windows-tagged file, so the cases that matter most there — both separators, empty and dot components, the root having no parent to create in — could not be tested on any runner that builds this. * test: check the windows mount persists across a remount Reading a file back through the same live mount proves nothing about durability; the answer can come from the mount's own caches. Write the fixtures, confirm the filer serves them with the mount out of the path, then re-read after a teardown and remount. * test: cover the windows mount operations that had none Truncate, append, chtimes and the hard-link refusal were implemented but never exercised, and the errno table was only unit-tested for mapping, never end to end. Adds names that have to survive the UTF-16 boundary, rename over an existing target and across directories, and concurrent handles on one file rather than one file each. * ci: dial the filer over ipv4 and run the persistence phases localhost resolves to ::1 first on windows and the cluster binds ipv4 only, so the mount's grpc dial was refused while the http readiness probe passed by falling back to ipv4. * ci: pin the cluster to loopback and probe ports by connecting weed mini advertises the runner's LAN address and binds filer grpc there, so the mount's dial to 127.0.0.1:18888 was refused while http answered. The readiness probe also passed with nothing on 18888: Test-NetConnection reported success for a port that then refused a connection, so it now opens a socket instead. * ci: report listening ports before mounting The readiness probe connects to the filer grpc port and the mount is then refused on it, which cannot both be true; print the actual state. * ci: run the cluster, mount and tests in one step The runner tears down a step's process tree when its shell exits, so the cluster started in an earlier step was already gone: the readiness probe passed against a live filer, the step ended, and the mount then found nothing listening. A diagnostic step reported no weed.exe at all. Everything that needs those processes alive now shares a step. * mount: key windows file io on the handle, not the path Read and Write walked the path on every call to fill in a NodeId the raw filesystem never reads: both look the file up by handle. Under eight writers creating files in one directory the walk transiently missed and the write failed with ENOENT before reaching the filesystem at all. Same for flush, fsync and the release calls. O_EXCL now fails on an existing name instead of taking it over, and Symlink is refused: the entry is easy to create but WinFsp only follows it once the reparse point is wired up, so it read back as an empty file. * mount: translate cgofuse open flags for windows cgofuse reports MSVC's numbering and the raw filesystem tests Go's, so only the access mode and O_TRUNC lined up: O_EXCL arrived as O_APPEND and O_CREAT as nothing at all. Also report which handle a failed write was using, to tell a handle that was never issued from one released while still in use. * mount: report which step of a windows create failed A concurrent create fails with ENOENT and the path walk, the parent lookup and the create itself are indistinguishable from the caller. * ci: send weed logs to stderr on windows glog writes to its own files by default, so the mount's own error output never reached the redirected log. Its flags are global and have to come before the subcommand. * mount: resolve known paths from the inode table on windows Every create walked the parent chain with a filer lookup per component. With eight writers creating files in one directory that is hundreds of concurrent lookups of the same parent, and lookupEntry reports an authoritative ENOENT when the directory is cached, the entry is not in the cache and the inode table has no record — a window a concurrent refresh can open for a directory that plainly exists. A path the mount already tracks now resolves straight out of that table. * test: sync the windows persistence fixtures before closing The mount is killed rather than unmounted, so anything still queued for flush is legitimately lost and the test was measuring crash durability while calling it persistence. A 9MB file lost four chunks that way. * mount: keep the lookup refresh on the target path Resolving a tracked path straight from the inode table skipped Lookup, which is also what refreshes the entry: a truncate then read back the pre-truncate size. Only the parent chain takes the shortcut now, which is where the concurrent creates were racing anyway. * mount: log every windows resolve failure Open suppressed ENOENT and Getattr logged nothing, which hid the two callbacks that can report a missing file during a create. * mount: drop dot entries from windows directory listings readdir reports "." and ".." for the kernel, but Windows enumerates a directory without them and displays whatever it is handed, so a folder of 200 files listed 202. Go's ReadDir filters them, which is why only the PowerShell walk caught it. * mount: flush queued writes when windows mount is interrupted The signal handler exits the process the moment its hooks return, so the WaitForAsyncFlush after Serve never ran on ctrl-c and queued writes were dropped. * mount: let windows mount over an empty directory WinFsp turns a directory mount point into a reparse point, which NTFS allows on an empty directory and refuses on a populated one. The check rejected every existing directory, so the ordinary habit of creating the mount point first failed with a message saying it should not exist. CI now mounts over a pre-created directory and writes through it. * ci: run the windows mount check on any pull request It is the only thing that exercises the Windows mount, so restricting it to pull requests based on master skipped it for stacked ones. Replaces the branch name that was pushed to trigger it. * mount: do not log a missing windows entry as an error Windows probes for entries that do not exist as a matter of course, so ENOENT from getattr and open is an answer rather than a fault and would have filled the log. * mount: take the fast path for parent chains in every windows resolve Narrowing it to resolveParent left Getattr and Open re-walking the parent with a filer lookup per component, and those are what Windows calls before a create: eight writers in one directory still raced a meta cache refresh there. Only the final component needs the Lookup refresh. The pass that suggested otherwise came from a run five times slower than the failing ones, where the race had no room to appear. * mount: drop the windows path resolution shortcut Resolving from the inode table skipped the Lookup that refreshes an entry, and a truncate then read back its old size. Applying it only to the parent chain kept truncate correct but left concurrent creates failing, and applying it to the final component too inverted that. The two cannot both be satisfied this way, so this returns to looking up every component and leaves the concurrent create failure open. * mount: fall back to the open handle when a deferred entry is evicted A create that defers the filer write leaves the entry only in the local cache. Creating many files at once pushes the directory past the hot threshold and evicts it, taking that placeholder with it, so a lookup went to the filer, found nothing, and reported a file that plainly exists as missing. The handle still holding the unflushed entry is authoritative for it. Caught by concurrent creates over a Windows mount, which resolves a path on every call rather than relying on a kernel dentry cache. * mount: let cgofuse resolve to the version the module graph requires rclone already depends on cgofuse at a newer commit than the v1.6.0 pin, so readonly builds refused the go.mod until it matched what MVS picks. The interface and flag values the adapter uses are unchanged there. * mount: wait for a pending async flush before looking up on the filer Open, unlink and rename already wait, but a plain lookup went straight to the filer and read pre-close metadata: truncate a file, close it, and a path probe during the flush window reported the old size. The kernel attr cache hides this on linux; a front end that resolves paths on every operation hit it directly. * mount: reject a umask wider than the file mode it becomes ParseUint allowed 64 bits and the result is narrowed to os.FileMode, which is 32, so an out-of-range umask truncated silently instead of being reported as unparseable. * mount: address review findings on the windows mount WaitForAsyncFlush closed its channel unconditionally and shutdown reaches it from both the interrupt hook and the path that resumes after serving, so a ctrl-c could panic on a second close. The deferred-entry fallback read an open handle's entry without its lock, which is what the other two readers of that field take so FromPbEntry does not walk the chunk slice mid-append. The async-flush wait also sat ahead of the meta cache, making every stat of a recently closed file queue behind uploads; it belongs just before the filer is consulted. Windows entries were persisted as uid 0: the raw filesystem stores InHeader's owner and the adapter left it zero. They now carry the identity the mount was started with. The errno table used Linux numbering while cgofuse decodes MSVC's, so ENAMETOOLONG arrived as EDEADLK and five others were likewise wrong; a windows test pins each value to cgofuse's own constant. Also: break the filer handshake loop on success rather than always running ten rounds, accept a drive letter written S:\\, report a missing WinFsp instead of panicking, keep commas out of the volume label, and drop -windows.caseInsensitive, which told WinFsp the mount folds case while lookups stayed exact. * mount: return windows lookup references so the inode table stays bounded Every operation that hands back an EntryOut grants a reference the Linux kernel returns with FORGET. WinFsp has no FORGET, so the adapter took one per path component per call, plus one per child of every readdirplus, and never gave any back: inodeToPath grew for the life of the mount. Walking the 200k-file directory this exists for stranded 200k references. The adapter now plays the part the kernel plays. Each resolution releases what it took, and an open handle keeps the reference for its inode until Release, counted because the raw filesystem reuses one handle for repeated opens. Holding it is not optional: completeAsyncFlush skips the metadata flush when the saved path no longer maps to the inode, so releasing early would lose a close's metadata. Also stops persisting the display owner. -o uid=-1 makes WinFsp report the calling user whatever we say, but the value handed to the raw filesystem is written to the filer, and 4294967295 is what every other client would read. -windows.uid and -windows.gid set what is recorded. * mount: fix windows behaviours the reference implementations guard against WinFsp has no ro option — it discards the flag and leaves the volume writable — so -readOnly accepted writes and deletes. The refusal now happens in the operations themselves. Windows sends times around its own 1601 epoch, which arrive as a large negative second count; casting them through stored a year-1601 timestamp that every other client then read. Those are now left alone. rclone carries the same guard. Chown returned ENOSYS, and WinFsp passes a chown failure straight out of SetSecurity, so Explorer's Security tab and icacls failed for edits that were not about ownership. It now accepts and discards. Only create and mkdir presented a caller; the rest sent uid 0, which hasAccess treats as root, so deletes and renames skipped the permission check that creates got. Every operation presents the same identity now. A drive letter written S:\ reached WinFsp unnormalised, which recognises a drive only as exactly two characters and then failed as a directory path. A test also pins the open flag translation, since swapping O_EXCL and O_TRUNC would turn 'fail if it exists' into 'truncate it'. * mount: answer windows getattr and truncate from the open handle WinFsp keeps the path a handle was opened with and never updates it when the file is renamed, so resolving the path again fails on a handle that is still perfectly valid — the ordinary write-temp-then-rename save pattern. The handle already knows its inode, which also removes a full path walk from two operations WinFsp calls constantly. Readlink on the root now refuses. WinFsp probes there to decide whether the volume has symlinks and enables them unless it fails, and with them on it resolves a path a component at a time, each one reaching us as its own walk — all for a feature Symlink already refuses. * mount: require the windows mount directory not to exist WinFsp creates the directory itself with FILE_CREATE and removes it when the filesystem goes away, so an existing one — empty or not — fails with "mount point in use". Allowing an empty directory was wrong, and the CI check that appeared to prove otherwise was the vacuous one: listing a plain directory succeeds whether or not anything is mounted on it, so the step passed while the mount had failed and the writes went to local disk. That check now waits for the reparse point, which is what caught this. * mount: apply review comments on the windows mount -windows.uid and -windows.gid reached the adapter but not the filesystem parameters, which is what carries the owner written to the filer, so the flags changed nothing. Readdir re-resolved the path while Getattr and Truncate answer from the handle; a directory renamed during an enumeration then failed on the stale path WinFsp still holds. Utimens now honours UTIME_OMIT instead of writing whatever came with it. * mount: tag the unix-only lock tests away from windows The production lock files were tagged when the package was made to build on windows, but the tests that exercise them were not, so anything that compiles tests for windows still failed on syscall.F_WRLCK. * ci: vet the mount tests for each target too Only compiling the non-test build let an untagged test keep a per-OS syscall constant without anything noticing. |
||
|
Chris LuandGitHub
|
c191b2fe01 |
iceberg: let clients select their table bucket as the catalog warehouse (#10549)
* iceberg: accept bare bucket names and ARNs as the catalog warehouse Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table bucket name or as the s3tables bucket ARN -- the two forms users reach for first, the latter being what AWS S3 Tables itself takes -- was silently dropped, so every call landed on the default "warehouse" bucket and failed with "table bucket warehouse not found". * iceberg: report a missing table bucket as 404, not 500 Pointing a client at a table bucket that does not exist -- which every client with no warehouse set does, since the default bucket "warehouse" rarely exists -- returned InternalServerError with a message naming a bucket the client never asked for. Answer 404 and say how to select one. * admin: show the warehouse in the PyIceberg example The example connected without one, so it always resolved to the default table bucket and every client that copied it failed on the first call. * test: pin bearer auth against a table bucket that exists The subtest called the catalog with no warehouse and accepted 500 as proof that auth had passed, since the default bucket does not exist. A missing table bucket now answers 404, which the test read as an auth failure. Give it a real table bucket so only 200 passes. * test: assert the missing-bucket guidance reaches the client The status and error type were checked but not the message, which is the part of the mapping that tells a user how to select a table bucket. * test: encode the warehouse query value The ARN case pasted raw colons and slashes into the query string. Go's parser tolerates them, so the test passed without modelling how a client actually sends the request. |
||
|
Chris LuandGitHub
|
82c67b5896 |
test: cover listings spanning a run of retracted keys (#10517)
* test: cover listings spanning a run of retracted keys A listing drops entries whose current version is a delete marker. When a run of consecutive entries drops out, the page being filled can come back empty, and an empty page is easily mistaken for the end of the listing — everything after the run then never appears and the caller is told those objects do not exist. Backup repositories produce exactly this shape: a batch of keys under one prefix is retracted while writing continues under the next. Covers a retracted run before live keys and between live keys, walked with page sizes smaller than the run so at least one page is filled entirely from entries that get dropped, plus the version view of the same namespace where every version and every delete marker must still be reported. * test: sweep every page size in both walks and paginate the version listing |
||
|
Chris LuandGitHub
|
d8d29c4ede |
s3: carry storage class in the cached listing metadata (#10516)
A listing on a versioned bucket is served from metadata cached on the .versions directory entry so the whole listing is a single scan. The cache carried size, mtime, ETag, owner and the delete-marker flag but not the storage class, so newListEntry found none and fell back to STANDARD. The result was that HEAD and the listings disagreed about the same object: HEAD reported the class the object was stored with, while ListObjectsV2 and ListObjectVersions reported STANDARD for every object. Clients that filter or tier on storage class act on the listing. Caches the class alongside the other listing fields, clears it with them, and copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree. |
||
|
Chris LuandGitHub
|
910fa1ff37 |
test: compare ListObjects and ListObjectVersions over the same namespace (#10515)
* test: compare ListObjects and ListObjectVersions over the same namespace The two listings walk the same tree through separate code paths, so a client navigating by versioned listings can see a different namespace than one navigating by plain listings, and concludes keys are missing that are plainly there. Testing each path on its own never catches that; only comparing them does, and nothing compared them. Asserts both report identical current keys and identical common prefixes across a backup-shaped tree: nested prefixes, a prefix naming an object exactly, a key that is simultaneously an object and the parent of other keys, a partial key fragment, and a prefix matching nothing. The version view is reduced to what a plain listing reports — latest versions that are not delete markers — so the comparison is like for like. * test: guard against truncated pages and cover the delete-marker path |
||
|
Chris LuandGitHub
|
fce4da5c9c |
test: pin verb parity on lock-arbitration keys through acquire and release (#10514)
* test: pin verb parity on lock-arbitration keys through acquire and release Backup clients arbitrate repository ownership by writing and retracting small keys under a fixed prefix and re-probing them, each probe using a different verb. They trust those verbs to agree; a key reported present by one and absent by another makes the client either spin or declare the repository corrupt, and neither shows up as an error on the storage side because each individual answer is locally correct. The keys are written and immediately deleted by version id, which is the cycle that empties a version container, so parity is asserted on both sides of the delete and across repeated re-acquire cycles where residue accumulates. Reports which verbs disagreed rather than just failing. * test: run the reacquire cycle on every lock key, and drain probe bodies |
||
|
Chris LuandGitHub
|
33a974b4c5 |
test: pin that an unusable version id is refused, never resolved (#10513)
* test: pin that an unusable version id is refused, never resolved A version id containing a path separator, or "." / "..", can never name a stored version. Resolving one to the null or latest version instead would let a caller destroy a live version by asking for one that does not exist, on a bucket configured for immutability. The guard exists today and holds on every verb; it had no test. Pins two properties: such a request is refused with a client error rather than a 5xx (a 5xx invites endless retries of something that can never succeed), and the version that does exist survives every refused request. * test: require exactly 400 for an unusable version id |
||
|
Chris LuandGitHub
|
2961448a36 |
test: cover delete idempotency on versioned object-locked buckets (#10512)
* test: cover delete idempotency on versioned object-locked buckets Backup clients probe and retract lock keys continuously, so they routinely delete keys and versions that are already gone, and they batch those deletes alongside keys that do exist. S3 makes all of that succeed; returning an error turns ordinary lock arbitration into a job failure. The behaviour is correct today but had no coverage, and it runs through the object-lock retention check, which is the most likely place for a missing object to start being reported as an error. Covers: deleting a key that never existed, deleting a well-formed version id that names nothing (twice, and without disturbing the version that does exist), and a batch whose middle key is missing — every requested key must come back under its own name rather than silently taking another row's slot. * test: verify the deletes actually took effect, not just that they returned |
||
|
Chris LuandGitHub
|
63d5140485 |
s3: allow copying an object onto itself in a versioned bucket (#10497)
* s3: allow copying an object onto itself in a versioned bucket The copy writes a new version instead of overwriting in place, which is how an earlier version is restored. Buckets with versioning off or suspended keep rejecting a self-copy that changes nothing. * s3: cover the suspended-versioning self-copy rejection Suspended versioning overwrites the null version in place, so a self-copy that changes nothing stays rejected. Pin that alongside the never-versioned case. |
||
|
Chris LuandGitHub
|
e7a678fa72 |
s3: keep the list marker exclusive for versioned objects (#10496)
* s3: keep the list marker exclusive for versioned objects A versioned object lives in a "<key>.versions" directory, so the entry name never matched the marker and start-after/marker returned the marker key itself. * s3: match the list marker against the raw entry name too A backend that echoes the marker it was given returns the ".versions" directory name, which no longer matched once the comparison used the object name alone. Cover both, and unit test each half. |
||
|
Chris LuandGitHub
|
a4692005e9 |
ci: harden the fusermount3 repair (#10485)
* ci: move the fusermount3 repair into a composite action Three copies of the same block were already drifting apart, and the target comes from PATH: only ever add setuid root to a root-owned, non-symlink binary under the system bin paths, and say why otherwise. * test: say that the process exited in the wait errors "process exit status 1 before ... accepted connections" is missing its verb. Also mark the SIGTERM return discarded - it fails with os.ErrProcessDone exactly when the select below already handles it. * ci: prefer the distro fusermount3 over escalating a shadow copy The shadowing /usr/local/bin/fusermount3 is not root-owned either, so setting its setuid bit would have handed root to a binary the runner user owns - the repair now symlinks the distro one earlier in PATH and touches nothing, keeping the in-place chmod for a root-owned binary with no distro alternative. A setuid bit only grants root when root owns the file, so accept an existing one only then. * ci: run the FUSE workflows when the shared action changes Their paths filters listed each workflow file but not the composite action all three now call. |
||
|
Chris LuandGitHub
|
167c114dae |
ci: fix FUSE mounts against the new runner image (#10484)
* ci: restore the setuid bit on a shadowed fusermount3 Newer ubuntu-22.04 runner images carry a source-built fusermount3 in /usr/local/bin that shadows the distro one in PATH and is not setuid root. go-fuse looks the helper up through PATH, so every unprivileged mount fails with "mount failed: Operation not permitted". * test: fail a fuse test as soon as its mount process dies A mount that cannot mount at all exits within a second, but the harness still waited out the 30s readiness timeout and then reported "mount point not ready within timeout", leaving the real cause buried in the log tail. Watch the child processes and report their exit instead. * mount: report a failed mount without a goroutine dump A mount failure is an environment problem - no /dev/fuse, fusermount not setuid, stale mount point - and the all-goroutine stack dump Fatalf adds buries the one line that says so. |
||
|
Chris LuandGitHub
|
62c4333074 |
s3: list the buckets an attached IAM policy grants (#10458)
* s3: list the buckets an attached IAM policy grants ListBuckets served an identity authorized by an attached IAM policy only the buckets it had created itself. A user granted s3:ListBucket on a bucket someone else provisioned could GetObject and ListObjectsV2 against it, but the bucket never showed up in the listing any S3 client uses to build its bucket picker. The owner-index fast path is only valid for an identity whose grants name every bucket it can reach, and the routing check assumed a policy could never be enumerated. Read the names out of the policy instead: statements that allow s3:ListBucket on a concrete bucket ARN become candidates, and the per-bucket permission re-check still decides what is listed. A policy that can reach a bucket it does not name -- a wildcard resource, a policy variable, a NotResource, an STS session policy -- falls back to the full scan, which evaluates the policy per bucket. * s3: share the attached policy name lookup authorizeWithIAM and the ListBuckets enumeration both built an identity's policy names the same way, its own plus the ones from its enabled groups. Pull that into one helper so group eligibility is decided in a single place. * s3: read policy actions the way the IAM authorizer matches them The IAM authorizer matches action names case-insensitively, so a policy granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets classifier read those actions with the local case-sensitive matcher and found no grant, so once the owner index was ready the buckets that policy allows dropped out of the listing. Match the action the looser way in the classifier: case-insensitive, and true for any pattern holding a policy variable. Over-matching only costs a candidate the per-bucket permission check then rejects, while under-matching hides a bucket the caller can read. * s3: infer a multipart grant in any case The classifier matches action patterns case-insensitively but looked the requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the s3:PutObject inference that the authorizer makes. Key the set for lookup in lower case, matching how the IAM authorizer holds it. |
||
|
Chris LuandGitHub
|
c392f45705 |
s3: stop listing prefixes whose objects are all delete-marked (#10419)
Deleting the only object under a prefix in a versioned bucket writes a delete marker and keeps the version history, so the filer directory survives with nothing a current-version listing would return. A delimited ListObjects kept reporting that path in CommonPrefixes, because the prefixes come from the directory tree rather than from the keys, while a listing scoped inside the prefix correctly came back empty. Probe a directory before reporting it: one that holds entries but no key the listing returns is neither a CommonPrefix nor a path the trailing-slash probe answers for. Empty directories keep the meaning they have today, and the probe only runs for buckets with versioning configured, the only ones that can reach this state. |
||
  |
9ad19c0ca4 |
build(deps): bump io.netty:netty-codec-http from 4.2.15.Final to 4.2.16.Final in /test/java/spark (#10408)
build(deps): bump io.netty:netty-codec-http in /test/java/spark Bumps [io.netty:netty-codec-http](https://github.com/netty/netty) from 4.2.15.Final to 4.2.16.Final. - [Release notes](https://github.com/netty/netty/releases) - [Commits](https://github.com/netty/netty/compare/netty-4.2.15.Final...netty-4.2.16.Final) --- updated-dependencies: - dependency-name: io.netty:netty-codec-http dependency-version: 4.2.16.Final dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
2e9b944e5c |
test(s3): aim collection force-delete at the master the suite actually runs (#10404)
The copying and tagging tests force-drop each bucket's collection at the master so volume slots are freed deterministically between tests. But the copy-tests CI job runs its master on 9336 and the tagging Makefile on 9338, while the tests default to 9333 — the cleanup dialed a dead port and quietly no-oped. Each test bucket then grows 7 volumes against -volume.max=100, and whenever async deletion lagged the data node ran out of slots and PutObject 500ed with "No writable volumes and no free volumes left". Set MASTER_ENDPOINT where the master port is non-default: the copy-tests workflow step, and the copying/tagging Makefiles (derived from MASTER_PORT). |
||
 
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
ce4b7f43bd |
build(deps): bump com.fasterxml.jackson.core:jackson-databind from 2.22.0 to 2.22.1 in /test/java/spark (#10378)
build(deps): bump com.fasterxml.jackson.core:jackson-databind Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson) from 2.22.0 to 2.22.1. - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.22.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
267f595660 |
batch delete: align the shard test and Rust server with continue-past-mismatch (#10349)
Commit
|
||
|
Chris LuandGitHub
|
68e6fc5f7c |
fuse test: keep derived filer gRPC port below the ephemeral floor (#10334)
The FUSE harness picks the filer HTTP port and lets "weed mount" derive the filer gRPC port as HTTP+10000. freePort kept the HTTP port under the Linux ephemeral floor (32768) but not the derived gRPC port, which ranged up to 42000. When the gRPC port landed above the floor an outbound connection could transiently hold it, so mini relocated its filer gRPC port while mount kept dialing HTTP+10000, timing the mount out. Cap the HTTP port at 22000 so the gRPC sibling stays at or below 32000, and verify both ports are bindable before returning. |
||
|
Chris LuandGitHub
|
29981f8d24 |
sftp: match permission paths on path-component boundaries (#10311)
* sftp: match permission paths on path-component boundaries Permission checks compared paths with raw string prefixes, so a permission entry for /tenants/alice also matched sibling paths such as /tenants/alice-archive, letting a scoped user read and overwrite another tenant's files. The home directory containment check had the same flaw. Route both through pathWithin, which cleans the paths and requires exact equality or a separator-delimited descendant. * sftp: clean permission path into a local when ranking matches |
||
|
|
c000addfc9 |
build(deps): bump golang.org/x/crypto from 0.51.0 to 0.52.0 in /test/kafka/kafka-client-loadtest (#10265)
build(deps): bump golang.org/x/crypto Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.51.0 to 0.52.0. - [Commits](https://github.com/golang/crypto/compare/v0.51.0...v0.52.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.52.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
1262147d9f |
build(deps): bump golang.org/x/crypto from 0.45.0 to 0.52.0 in /test/sftp (#10264)
build(deps): bump golang.org/x/crypto in /test/sftp Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.45.0 to 0.52.0. - [Commits](https://github.com/golang/crypto/compare/v0.45.0...v0.52.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.52.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
60e7b30009 |
admin: browse Iceberg table data (#10227)
* admin: move volume-server read JWT helper into dash The Iceberg data preview page needs the same per-fileId read token the file browser uses when streaming chunks from volume servers. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: add Iceberg table data preview page The admin UI browses the Iceberg catalog down to table details but not the data itself. Add a Browse Data page per table that walks the selected snapshot's manifests and shows sample rows from its Parquet data files, plus the data file list with per-file preview, a snapshot switcher, and a row limit selector. Rows are read through a ranged ReaderAt over stream-content so only the Parquet footer and needed pages are fetched, with the volume read JWT applied when configured. Iceberg locations resolve into /buckets with traversal guards, and the file parameter must match a manifest-listed data file. Snapshots with delete files get a warning that raw rows are shown. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: integration test for Iceberg catalog and data preview pages Starts a weed mini cluster with the admin UI, creates a table bucket, namespace, and tables via the S3 Tables manager, uploads real Parquet files via S3, writes manifests and snapshots with iceberg-go, and asserts on the rendered pages: catalog browsing, table details, current and historical snapshot previews, per-file preview, row limits, unknown snapshot and file errors, and a metadata-less table. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: write Iceberg preview chunk reads straight into the caller slice ReadAt wrapped the caller's buffer in a bytes.Buffer, which would silently allocate a fresh backing array and drop bytes if it ever grew. Copy directly into the destination slice and reject negative offsets so the ReaderAt contract holds. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: link to snapshot history when the preview switcher truncates The snapshot switcher caps at 25 entries; add a trailing item pointing at the table details page so older snapshots stay reachable. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * test: hoist mini cluster context assignment out of the goroutine Set MiniClusterCtx before launching the cluster goroutine and clear it in stop(), so the assignment is not buried in the command loop. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur |
||
|
Chris LuandGitHub
|
17af32f3ff |
s3: paginate ListBuckets and serve it from a bucket owner index (#10214)
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix ListBuckets buffered every bucket entry into one slice and one XML body, which falls over with very large bucket counts. Page through the filer listing instead, cap each response at 10000 buckets like AWS, and honor max-buckets, prefix, and an opaque keyset continuation-token. * s3: maintain a bucket owner index under /buckets/.system/owners Map each bucket owner to its buckets as zero-length entries at /buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the bucket's creation time. The bucket handlers write the index synchronously, the /buckets metadata subscription reconciles changes made elsewhere (weed shell, other gateways, direct filer operations), and a startup backfill indexes pre-existing buckets before writing a ready marker. Owner names are path-escaped so no identity name can escape the index directory. * s3: serve ListBuckets from the bucket owner index Once the owner index is ready, non-admin identities list their owned buckets straight from it, merged with any buckets their legacy actions name explicitly, so ListBuckets costs O(own buckets) instead of a scan of the global /buckets directory. Admins, identities with a bare List grant or wildcard action patterns, and policy-authorized identities whose grants cannot be enumerated keep the paged scan; policy-routed identities get their owned buckets, matching AWS ListBuckets returning only the caller's buckets. * s3: keep dot-prefixed names under /buckets out of bucket surfaces Dot-prefixed entries (.system) can never be valid bucket names, so refuse to resolve them as buckets and skip them in the shell bucket listing, matching what ListBuckets and the admin UI already do. * test: cover ListBuckets pagination and the owner index end to end * s3: fail closed on a nil identity when routing ListBuckets * s3: decide the IAM authorization mechanism in one place VerifyActionPermission and the ListBuckets owner-index routing each re-derived the session-token / attached-policy / legacy-actions split; extract the decision so the two cannot drift. * s3: heal the owner index on concurrent bucket recreation too The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like the up-front existence check, so give it the same index repair. * s3: drop owner-index records for buckets deleted during backfill A bucket removed between the backfill reading its page and writing the index record became a permanent phantom in its owner's listing: the delete's own cleanup ran before the record existed. After indexing each page, re-list the same name range and remove records whose bucket is gone; deletes landing after the re-list find the record and remove it themselves. * s3: add ContinuationToken and Prefix to the ListBuckets schema Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so a regeneration does not drop the pagination fields. |
||
|
Chris LuandGitHub
|
3089480c30 |
fix(fuse-tests): pass glog flags before the mount subcommand (#10215)
glog flags (-v, -logtostderr) are registered on weed's global flagset, so passing them after the subcommand name kills the process at flag parsing: flag provided but not defined: -logtostderr. The old stat-based mount readiness probe masked this — TestWriteBufferCap silently ran against the bare local directory and passed. The device-ID readiness check now surfaces the dead mount as a not-ready timeout. Move glog flags into MountGlobalOptions, emitted before the subcommand, and do the same for the EnableDebug verbosity flag on mini and mount. |
||
|
|
267ff3b187 |
build(deps): bump golang.org/x/net from 0.47.0 to 0.55.0 in /test/kafka/kafka-client-loadtest (#10210)
build(deps): bump golang.org/x/net in /test/kafka/kafka-client-loadtest Bumps [golang.org/x/net](https://github.com/golang/net) from 0.47.0 to 0.55.0. - [Commits](https://github.com/golang/net/compare/v0.47.0...v0.55.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.55.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
6206f60032 |
fix(master): let the growth initiator wait instead of shedding itself (#10202)
* fix(master): let the growth initiator wait for the growth it triggered The growth-in-flight shed also fired on the request that initiated the growth: it sets the pending flag right before the shed check, so a cold-start assign enqueued growth and immediately failed itself with "volume growth in progress". With no concurrent assigns around to pick up the freshly grown volume, a single writer against an empty cluster never completes a write despite ample free space. Claim the pending flag with a compare-and-swap so exactly one request becomes the initiator, triggering growth at most once, and let it wait for that growth to land. Everyone else still sheds retryably instead of pinning a goroutine: followers behind an in-flight growth, an initiator whose growth concluded without yielding a writable volume, and an initiator whose growth outlives the 10s wait budget, which previously surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a retry would have succeeded moments later. * fix(master): stop assign waits when the request is cancelled The assign retry loops slept through client cancellation, keeping a goroutine spinning for the rest of the 10s budget after the caller had gone; StreamAssign also ran assigns on a background context detached from the stream. Wait on the request context and pass the stream context through. * topology: drop the unconditional grow-request setter Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap now; keeping the raw Store(true) around invites the check-then-set race back. * test: cover cold-start first write with a real cluster Boot a fresh master plus three empty volume servers and require the very first assign - HTTP and gRPC, each on a cold volume layout, no client retries - to complete a write. The assign that triggers volume growth must wait for it rather than answering "volume growth in progress"; unit tests stub the topology, so only a real cluster exercises the assign-grow-wait path end to end. |
||
|
Chris LuandGitHub
|
c5240944e4 |
test: keep mini-allocated ports below the ephemeral floor (#10209)
* test: keep mini-allocated ports below the ephemeral floor Allocated ports could land in 32768-55000, so a transient outbound dial during mini startup (volume->master gRPC, etc.) could grab an allocated port as its source port before the filer bound it, failing with "bind: address already in use". Cap the range so port+GrpcPortOffset stays under 32768. * test: derive mini port cap from the ephemeral floor constant Name the 32768 floor once and compute miniPortMax as floor-GrpcPortOffset so the cap tracks the offset; reuse the constant in the regression test. |
||
|
Chris LuandGitHub
|
2540141ee7 |
fix(fuse-tests): don't declare the FUSE mount ready before it is mounted (#10208)
waitForMount probed the mount point with stat+ReadDir, which a bare local directory also passes, so Setup could return before the weed mount process finished mounting. Tests then wrote to the local disk underneath the mount point; when the mount activated it shadowed those files, producing the intermittent TestConcurrentFileOperations/ConcurrentReadWrite ENOENT with an empty ReadDir. Require the mount point's device ID to differ from its parent's before reporting ready. |
||
|
Chris LuandGitHub
|
2c2df751f5 |
Perf CI: benchmark the Rust volume server and report memory usage (#10111)
* ci: add per-process memory sampler for perf jobs Samples VmRSS once a second into a CSV and records peak VmHWM per process on stop. Linux only; reads /proc/<pid>/status. * ci: run perf benchmarks on the Rust volume server and report memory Matrix the throughput and S3 jobs over go/rust volume servers, using a standalone master (plus filer for S3) and swapping only the volume binary so the two are directly comparable. Sample peak RSS in every job and surface it per impl in the run summary. * ci: harden mem sampler arg handling and peak fallback Guard against missing args under set -u, and fall back to the max RSS sampled when a process exits before VmHWM can be read. |
||
|
Chris LuandGitHub
|
a88acaf061 |
Add performance CI (profiling, throughput, S3 read/write) (#10105)
* test: add self-contained S3 read/write load tool Concurrent PUT/GET against the S3 gateway, reporting requests/sec, transfer rate, and latency percentiles. Built on the aws-sdk-go-v2 client the S3 tests already use, so no extra benchmark binary is needed. * ci: add performance workflow Three parallel jobs: cpu/heap pprof of the server under write load, native throughput via weed benchmark plus the Go micro-benchmarks, and an S3 read/write benchmark against the gateway. Runs on push to master and manual dispatch with tunable duration, object count, size, and concurrency. |
||
|
Chris LuandGitHub
|
3b9e196e5f |
sts: enforce session-policy explicit deny during role chaining (#10103)
* sts: enforce session-policy explicit deny during role chaining A chained AssumeRole caller authenticates with an STS session token whose inline session policy can explicitly deny sts:AssumeRole. The deny check only evaluated the caller's named policies, so such a session could still chain into any role its trust policy admits. Validate the session token in the deny check and honor an explicit Deny in the inline session policy too. * test(sts): integration coverage for AssumeRole authorization Add an end-to-end AssumeRole authorization test (real weed mini + boto3): a non-admin caller assumes a role its trust policy admits, an explicit identity-side deny is blocked, and a session policy's explicit deny blocks role chaining. * sts: skip OIDC tokens and reject revoked sessions in the chaining deny check Review follow-ups on the session-policy deny check: - Guard session validation with !isOIDCToken so a bearer token our STS service cannot validate does not error into a false deny. - Reject a revoked session before evaluating its policy, restoring the revocation enforcement the AssumeRole path lost when it stopped routing through IsActionAllowed. |