mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-18 21:26:56 +00:00
f41595fb101feb428e3dc0bbaf8cfe5e49be67fc
9677
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
f41595fb10 |
mount: drop consumed entries when reading a directory through (#10802)
The cached readdir trims the head of the handle's entry stream as the client walks past it; the read-through path never did, so a directory too large to cache -- the only kind that takes that path -- was held whole in the handle for the length of the walk. Hoist the trim to cover both paths. |
||
|
Chris LuandGitHub
|
3cf7d306a5 |
Give the WebDav chunk reader a bounded, invalidatable location cache (#10801)
* mount: re-resolve volume locations after a failed chunk read NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A mount that cached a volume's locations while one server was down kept retrying that server after it died, then returned EIO, even though the master and filer both resolved the live replica. The S3 gateway already passes its filerClient; do the same for the mount. * test: FUSE integration tests for volume server failover One mount appends while a second tails, and a volume server is killed, started or restarted mid-stream against a 001-replicated cluster of three volume servers. Automates the scenario matrix reported for Docker Swarm mounts, including the large-file variant and a no-chaos control. * test: report the filer's own view when append content mismatches A mismatch between what the writer wrote and what the reader sees can come from either side's cache. Read the file back through the filer's HTTP handler as well, and let the mount verbosity be raised from the environment, so a failing run says which layer lost the data. * test: wait for the reader mount to converge before comparing A mount caches metadata for about a second, so reading the file the instant the writer's last close returned can legitimately come back short. Poll the reader until it matches or the timeout expires; content that is wrong rather than merely late never converges and still fails, now with the writer's mount and the filer's own view alongside it. * test: detect a failover cluster child that exited at startup Signal(0) succeeds for a zombie and nothing reaped these children until shutdown, so a process that died on startup looked alive until the readiness timeout expired. Reap each child as it is started and consult the result. * test: read a file the killed volume server actually holds Placement decides which two of three servers back each volume, so killing volume N and reading readfile-N could pass without the victim ever holding a replica of it. Resolve each file's volumes through the filer and the master, and pick one the victim backs, preferring a file the reader has not cached. * ci: stop persisting checkout credentials in the failover workflow The job does not use the token after cloning. Also tag the README's command block as bash and match the timeout the workflow actually uses. * test: discard the ignored errors errcheck flags in the failover harness * test: resolve manifests when mapping a file to its volumes A manifest chunk's own fid names the volume holding the manifest, not the volumes holding the data, so a large enough file would point the failover victim at the wrong server. * test: pin the stale-location recovery path with a primed reader Reading a file for the first time after a server dies proves nothing: the lookup is fresh and returns the survivor. Kill one holder and wait for the master to drop it, read a file on that volume so the reader caches the lone survivor, restart the first server, then kill the survivor. The reader's only cached location is now dead while the data is live elsewhere, which is the case the invalidator exists for: EIO without it, recovery with it. * filer: re-look-up a chunk's locations as soon as they all fail A read that fails against every location it was given is far more likely to be holding a stale list than to be hitting a cluster that is briefly slow, but the retry loops spent the whole backoff ladder, about 13 s, before the caller got a chance to invalidate and look the chunk up again. Give the loops a refresh hook and let the reader cache invalidate on the first fully failed pass, so recovery starts in milliseconds. Clients without an invalidator keep the old behavior. The filer's streaming read path has its own fetch loop and is not covered. * webdav: give the chunk reader a bounded, invalidatable location cache WebDav resolved chunk locations through filer.LookupFn, whose own doc asks long-running processes to prefer wdclient.FilerClient: its cache is unbounded, and it has no way to invalidate an entry, so the reader cache was constructed with a nil invalidator and a WebDav server that had cached a location kept reading from it after the volume moved or died. Use FilerClient, as the mount and the S3 gateway already do. * filer: refresh locations on the random-read path too readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches the range directly, which left it without the invalidation the cacher does: a random reader parked on a stale location had no way back at all. Hoist the refresh hook onto the reader cache so both paths share it. * filer: compare chunk locations as a set, not in order Lookups shuffle the locations they return, so comparing positionally reads a reshuffle of the very same replicas as a fresh set and spends an immediate retry on locations that just failed. weed/filer already had an order-independent comparison for this; move it next to the retry loops so both callers share one helper. |
||
|
Chris LuandGitHub
|
f3dc530919 |
Re-look-up a chunk's locations as soon as they all fail (#10800)
* mount: re-resolve volume locations after a failed chunk read NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A mount that cached a volume's locations while one server was down kept retrying that server after it died, then returned EIO, even though the master and filer both resolved the live replica. The S3 gateway already passes its filerClient; do the same for the mount. * test: FUSE integration tests for volume server failover One mount appends while a second tails, and a volume server is killed, started or restarted mid-stream against a 001-replicated cluster of three volume servers. Automates the scenario matrix reported for Docker Swarm mounts, including the large-file variant and a no-chaos control. * test: report the filer's own view when append content mismatches A mismatch between what the writer wrote and what the reader sees can come from either side's cache. Read the file back through the filer's HTTP handler as well, and let the mount verbosity be raised from the environment, so a failing run says which layer lost the data. * test: wait for the reader mount to converge before comparing A mount caches metadata for about a second, so reading the file the instant the writer's last close returned can legitimately come back short. Poll the reader until it matches or the timeout expires; content that is wrong rather than merely late never converges and still fails, now with the writer's mount and the filer's own view alongside it. * test: detect a failover cluster child that exited at startup Signal(0) succeeds for a zombie and nothing reaped these children until shutdown, so a process that died on startup looked alive until the readiness timeout expired. Reap each child as it is started and consult the result. * test: read a file the killed volume server actually holds Placement decides which two of three servers back each volume, so killing volume N and reading readfile-N could pass without the victim ever holding a replica of it. Resolve each file's volumes through the filer and the master, and pick one the victim backs, preferring a file the reader has not cached. * ci: stop persisting checkout credentials in the failover workflow The job does not use the token after cloning. Also tag the README's command block as bash and match the timeout the workflow actually uses. * test: discard the ignored errors errcheck flags in the failover harness * test: resolve manifests when mapping a file to its volumes A manifest chunk's own fid names the volume holding the manifest, not the volumes holding the data, so a large enough file would point the failover victim at the wrong server. * test: pin the stale-location recovery path with a primed reader Reading a file for the first time after a server dies proves nothing: the lookup is fresh and returns the survivor. Kill one holder and wait for the master to drop it, read a file on that volume so the reader caches the lone survivor, restart the first server, then kill the survivor. The reader's only cached location is now dead while the data is live elsewhere, which is the case the invalidator exists for: EIO without it, recovery with it. * filer: re-look-up a chunk's locations as soon as they all fail A read that fails against every location it was given is far more likely to be holding a stale list than to be hitting a cluster that is briefly slow, but the retry loops spent the whole backoff ladder, about 13 s, before the caller got a chance to invalidate and look the chunk up again. Give the loops a refresh hook and let the reader cache invalidate on the first fully failed pass, so recovery starts in milliseconds. Clients without an invalidator keep the old behavior. The filer's streaming read path has its own fetch loop and is not covered. * filer: refresh locations on the random-read path too readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches the range directly, which left it without the invalidation the cacher does: a random reader parked on a stale location had no way back at all. Hoist the refresh hook onto the reader cache so both paths share it. * filer: compare chunk locations as a set, not in order Lookups shuffle the locations they return, so comparing positionally reads a reshuffle of the very same replicas as a fresh set and spends an immediate retry on locations that just failed. weed/filer already had an order-independent comparison for this; move it next to the retry loops so both callers share one helper. |
||
|
Chris LuandGitHub
|
606a90b3b1 |
filer: close the empty-folder race by checking after each mutation (#10799)
* filer: re-list a folder after deleting it, and put it back if it is not empty The emptiness check inside the delete and the removal of the folder entry are not atomic, so an entry can land between them and be left reachable by its own path but out of every listing. Looking again after the delete catches the ones whose create event has not arrived yet, and does not depend on the event stream or on the observation window holding. * filer: create the directories holding an entry after the entry A parent checked before the insert can be taken by the empty-folder cleaner before the entry lands, which leaves the entry reachable by its own path but out of every listing. Creating the parents afterwards cannot be undone by a delete that was authorised before the insert, and pairs with the cleaner re-listing after its own delete: whichever of the two acts second sees what the other did. Going second means the entry is already stored when the parent fails, so it is taken back out and the caller still sees the error it used to get. * filer: narrow a directory that came back wider than the one it replaced A writer recreating its own missing parent has only the entry it is inserting to go on, so the directory it mints can grant access the deleted one denied - a 0700 folder comes back 0751. The cleaner read the real attributes before deleting, so its restore now puts the original mode back instead of leaving the inferred one in place. It only ever narrows, so a directory deliberately tightened since is left as it is. |
||
|
Chris LuandGitHub
|
1ddec72707 |
Recover from a dead volume server on the mount read path (#10798)
* mount: re-resolve volume locations after a failed chunk read NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A mount that cached a volume's locations while one server was down kept retrying that server after it died, then returned EIO, even though the master and filer both resolved the live replica. The S3 gateway already passes its filerClient; do the same for the mount. * test: FUSE integration tests for volume server failover One mount appends while a second tails, and a volume server is killed, started or restarted mid-stream against a 001-replicated cluster of three volume servers. Automates the scenario matrix reported for Docker Swarm mounts, including the large-file variant and a no-chaos control. * test: report the filer's own view when append content mismatches A mismatch between what the writer wrote and what the reader sees can come from either side's cache. Read the file back through the filer's HTTP handler as well, and let the mount verbosity be raised from the environment, so a failing run says which layer lost the data. * test: wait for the reader mount to converge before comparing A mount caches metadata for about a second, so reading the file the instant the writer's last close returned can legitimately come back short. Poll the reader until it matches or the timeout expires; content that is wrong rather than merely late never converges and still fails, now with the writer's mount and the filer's own view alongside it. * test: detect a failover cluster child that exited at startup Signal(0) succeeds for a zombie and nothing reaped these children until shutdown, so a process that died on startup looked alive until the readiness timeout expired. Reap each child as it is started and consult the result. * test: read a file the killed volume server actually holds Placement decides which two of three servers back each volume, so killing volume N and reading readfile-N could pass without the victim ever holding a replica of it. Resolve each file's volumes through the filer and the master, and pick one the victim backs, preferring a file the reader has not cached. * ci: stop persisting checkout credentials in the failover workflow The job does not use the token after cloning. Also tag the README's command block as bash and match the timeout the workflow actually uses. * test: discard the ignored errors errcheck flags in the failover harness * test: resolve manifests when mapping a file to its volumes A manifest chunk's own fid names the volume holding the manifest, not the volumes holding the data, so a large enough file would point the failover victim at the wrong server. * test: pin the stale-location recovery path with a primed reader Reading a file for the first time after a server dies proves nothing: the lookup is fresh and returns the survivor. Kill one holder and wait for the master to drop it, read a file on that volume so the reader caches the lone survivor, restart the first server, then kill the survivor. The reader's only cached location is now dead while the data is live elsewhere, which is the case the invalidator exists for: EIO without it, recovery with it. |
||
|
Chris LuandGitHub
|
6fda8c67f3 |
Guard the gcs credential path in FetchAndWriteNeedle like the other backends (#10796)
* volume: accept only static-key gcs credentials on the fetch request An inline credentials document of a federated type points the SDK at a url, file or executable of the caller's choosing for the token exchange, so the request-supplied value is no longer just a key. * volume: guard the gcs token endpoint like the other remote endpoints Inline credentials pick where the token request goes, so route the gcs client through the same deny-list and rebinding-safe dialer used for S3 and azure. * rust volume: pin that gcs has no credential-driven dial path * volume: only check gcs credentials on a gcs remote conf Only the gcs backend reads that field, so another backend carrying a stale value should not fail the request. * gcs: load credentials with the type the caller expects The untyped loader is deprecated because it reads whatever the document claims to be; callers handling credentials they do not control now name the types they accept. |
||
|
Chris LuandGitHub
|
e383ee47cb |
filer: use bind variables for request-controlled values in the arangodb store (#10795)
* arangodb: bind list prefix, start file name and collection into the AQL query Concatenating them into the query text let a caller-supplied prefix or start name close the string literal and append arbitrary AQL, which runs with the filer's ArangoDB credentials against any collection. * arangodb: bind the folder path and collection into the recursive delete query A trailing-slash S3 key reaches DeleteFolderChildren through the directory-marker cleanup, so quotes in the path could turn the filter into a match-everything REMOVE over the whole bucket collection. * arangodb: match the real directory prefix in the recursive delete The prefix was built by re-joining the path segments with commas, so it never matched a stored directory and the subtree sweep did nothing. |
||
|
Chris LuandGitHub
|
5d5ea63b3f |
Fix what the Go 1.26 language bump breaks (#10794)
* worker: log the balance move stage through a constant format string Go 1.26's printf analyzer now follows printf wrappers reached through an interface, so passing the stage straight to Logger.Info is a vet failure. * s3api: bracket the IPv6 host in the signature test URL A bare IPv6 literal is legal in a Host header but never in a URL. Go 1.26 stopped parsing it leniently, so carry the two forms separately and set r.Host to the value the client would actually have signed. * mini: bracket IPv6 addresses in the readiness probe URLs An IPv6-only host hands mini a bare literal, and %s:%d pasted it into a URL unbracketed. Under Go 1.26 that URL no longer parses, so waiting for the admin server never succeeds and mini refuses to start. |
||
 github-actions[bot]
|
f04da8e9ad | 4.42 | ||
|
Chris LuandGitHub
|
5c43c03b76 |
filer: restore a folder that received an entry while it was deleted (#10783)
* filer: restore a folder that received an entry while it was deleted The empty-folder cleaner checks that a folder is empty and then deletes it, and those two steps are not atomic. An entry created in between survives the delete but loses the directory holding it: still readable by its own path, yet absent from every listing until a later write happens to recreate the parent. Record the folders deleted in each pass and re-check them on the next one, putting back any that turned out to hold entries. The check waits a pass on purpose - a writer looks up the parent before inserting the child, so checking straight after the delete can still run ahead of the insert and see nothing. Restoring a directory that holds entries is always correct, and restoring one whose entry went away again just leaves an empty folder for a later pass to collect, so the repair needs no locking or coordination. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * filer: keep failed restores queued and inherit the ancestor's ownership Two gaps in the restore pass. A folder whose count or restore hit a transient store error was dropped from the tracking list and never looked at again, leaving its entries out of listings until some later write recreated the folder - the very thing the pass exists to avoid. Put those back for the next pass, still under the cap. A restored folder was minted with a fixed mode and no owner, so a directory that had been private came back world-readable and owned by root. Take the mode and ownership from the nearest ancestor still present instead. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * filer: let the redis stores keep a directory listing that still has entries On the redis stores the listing is not derived from the entries, it is the only record that they sit under that directory. DeleteEntry opened by dropping it outright, so an entry that arrived after the caller judged the directory empty lost its membership and became unreachable: readable by exact path, absent from every listing, and invisible to any later check, since counting the directory reads the listing that was just destroyed. Nothing could detect or repair it. Drop the listing in DeleteFolderChildren instead, alongside the children it describes, and leave it alone in DeleteEntry. redis3 needs it explicitly, since removeChildren clears the skip list nodes but not the list itself, and the plain redis store was leaking the key entirely. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * filer: restore folders with their own attributes, and observe them for a window Five gaps in the restore pass. The restored directory was reconstructed from whatever ancestor happened to still be present, and the mode was ORed with 0111 on the way. A private directory under a world-traversable parent came back granting traversal it had denied. Read the folder's own attributes before deleting it and put exactly those back. That also removes the ancestor walk, which treated a transient store error as "not found" and silently fell through to a broader ancestor. A single check a pass later was not a delay at all. Ticker sends coalesce, so when a pass runs long the next one starts immediately, and a writer already past its parent lookup can insert after the check has read zero - after which the folder was discarded for good. Keep each folder under observation for a bounded wall-clock window and re-check it on every pass until it expires. This narrows the exposure rather than closing it; only making the emptiness check and the delete atomic would do that. A delete that returned an error was never observed at all, though the redis stores drop the folder before its parent-list member, so a failure return is not proof the folder survived. Record the folder before the delete instead. Restores now run shallowest first, so a folder taken by the parent cascade is rebuilt with its own attributes before anything below it needs it as a parent. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * filer: recover a deleted folder from the create event for the entry that raced it Checking each deleted folder on a timer was the wrong instrument. It cost a listing per folder per pass, and it could only ever be a guess about when the racing write would land. The metadata stream already carries the answer. A folder is recorded before it is deleted, so any entry that can be orphaned is created after that record and its create event names that exact directory. Match the event against the recently deleted folders and the folder is known to need putting back, rather than inferred to. The window stops being a guess at the race and becomes what it should be: how far behind the event stream is allowed to run before a folder stops being watched. Listing is now done once, for a folder an event has already named, to skip the restore when the entry has since gone away again. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * filer: bound how long a folder is watched, and rebuild ancestors from themselves Four gaps found reviewing the restore pass. A folder whose restore kept failing was never let go: the written-to check ran before the age check, so it was picked up, retried, put back, and counted again on every pass for the life of the process. Apply the window first, whatever state the folder is in. At the cap, the folder being recorded was the one turned away, though it is the one whose race is still live - the older entries are already close to ageing out. Give up one of those instead, picked as the oldest of a small sample so the cost stays flat under heavy deletion rates. An ancestor taken by the same cascade was left to the descendant's restore to recreate, which minted it from the descendant's attributes and handed back access the ancestor never granted. Rebuild those from what they were, ahead of anything below them. Reading a directory's attributes assumed an entry came back. Some stores return nothing with no error, so treat that as not found. The mode is also taken whole rather than through Perm(), which was dropping setgid, setuid and sticky. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * redis3: take a directory listing left behind by a failed delete Removing the last name deletes the list, and if that delete fails the header survives pointing at a name that is gone. The retry finds nothing to remove, reports no changes, and returns before reaching the delete, so the key stays for good. Take it on that path too. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r |
||
|
Chris LuandGitHub
|
f530102c45 |
filer: do not sweep children when deleting a folder non-recursively (#10782)
* filer: do not sweep children when deleting a folder non-recursively doBatchDeleteFolderMetaAndData lists a folder and bails out if it has any children, then calls Store.DeleteFolderChildren unconditionally. On the non-recursive path that bulk sweep has nothing legitimate to remove: it only runs once the listing came back empty, so the sole rows it can delete are ones inserted after the check. The S3 empty-folder cleaner deletes through this path, so a PUT landing between the listing and the sweep loses its entry after the write was already acknowledged. Neither side sees an error - the client has its 200 and the cleaner logs an ordinary empty-folder deletion - and the chunks leak, since the cleaner passes shouldDeleteChunks=false and nothing was enumerated to collect. Workloads that scatter objects over many shallow prefixes empty and refill those folders constantly, which is what makes the window reachable. Sweep only when the delete is recursive, or when the whole-bucket shortcut skipped the listing and depends on it. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r * filer: pin the folder entry removal left by the racing-child test The surviving entry is reachable by path but drops out of listings until the folder comes back, and nothing in the test said so. Assert it, so the exposure that remains after this change is visible rather than implied. Claude-Session: https://claude.ai/code/session_01HdLXMUopwgofPb1ZEmiE6r |
||
|
Chris LuandGitHub
|
7522e17b6d |
iceberg: vend table-scoped credentials to clients that ask for delegation (#10777)
* iceberg: vend table-scoped credentials to clients that ask for delegation The catalog recognised X-Iceberg-Access-Delegation: vended-credentials and then deliberately said nothing, because it had nothing to vend: it withheld even the S3 endpoint so the client would keep the credentials it was configured with. That left every engine expecting the catalog to hand out access - Snowflake, Databricks, Trino with vending, any multi-tenant setup - needing static S3 keys distributed out of band. Mint an STS session per request instead, scoped by a session policy to the table's own prefix plus the bucket listing needed to resolve it, and return it in the load response config and storage-credentials. The role to assume is named by -s3.iceberg.credentialRole; its trust policy is what decides whether a caller may assume it, and vending stays off until it is set. A failed mint falls back to the old silence rather than handing back an endpoint the client cannot sign for. * iceberg: keep vended credentials inside the table prefix Review follow-ups on credential vending: Listing was granted on the bucket ARN with no condition, so a credential vended for one table could enumerate every other table's object names. Constrain s3:prefix to the table's own prefix, which the S3 gateway already populates for list requests. A table location carrying * or ? would have gone into the policy's resource pattern unescaped and widened the session to sibling prefixes. Refuse to vend for such a location rather than escaping it; nothing the catalog generates contains those characters. DurationSeconds skipped the 900..43200 bounds the other assume-role paths enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session outside them. The check is now shared by all three entry points. * iceberg: return the vended credentials from buildFileIOConfig itself buildStorageConfig was a second name for what buildFileIOConfig already did; it now returns the storage credentials alongside the properties, and callers that only want the properties drop them. * iceberg: split the vended bucket grants, and refuse a whole-bucket scope The prefix condition sat on a statement that also granted GetBucketLocation and ListBucketMultipartUploads, neither of which carries an s3:prefix to satisfy it, so both were denied for every vended credential. GetBucketLocation moves to its own unconditioned statement. ListBucketMultipartUploads is dropped: Iceberg writers complete and abort by upload id, and granting it either leaks in-flight keys bucket-wide or breaks on the same missing prefix. A table whose location has no prefix - one registered at the bucket root - would have been vended read and write over every other table in the bucket. Refuse, the way a location with wildcards is refused. |
||
|
Chris LuandGitHub
|
ec37ef5aaa |
iceberg: add view rename, scan-report and snapshots=refs to the catalog (#10776)
* iceberg: add view rename, scan-report and snapshots=refs to the catalog Three gaps against the REST spec that clients hit in normal use: Views had no rename, though tables did and views are stored the same way, so the move is the same catalog-only pointer move. Tables and views share a namespace directory, so both renames now refuse the other kind instead of moving it. Engines POST a scan or commit report after planning; a 404 there turns into an error line per query. Accept the report and discard it - the catalog keeps no metrics store. LoadTable ignored ?snapshots=refs and always returned the whole snapshot history, which is what clients use the parameter to avoid on long-lived tables. * iceberg: authorize view rename against the view ARN, tighten the metrics endpoint Review follow-ups: The shared rename checked the source against a table ARN whatever the kind, so a policy scoped to a view's own ARN never matched and one written for a table ARN was evaluated for a view. The entry kind now carries the ARN builder. The metrics endpoint truncated a report at 1 MiB and then failed to parse it, answering 400 for a query that had actually succeeded. Read one byte past the limit to tell "fits" from "cut short", and discard an oversized report instead of rejecting it. Empty bodies and reports without a report-type are now rejected, which the REST schema requires. ?snapshots= is defined for LoadTable, so it no longer filters what CreateTable echoes back. |
||
|
Chris LuandGitHub
|
d044839ab2 |
iceberg: make a table commit a compare-and-swap (#10775)
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
|
||
|
Chris LuandGitHub
|
a80259d362 |
iceberg maintenance: fix the test build master merged broken (#10780)
#10774 gave buildTestMetadata its refs and age parameters while #10773 added a caller with the old arity. Each was green against a master that did not yet have the other, and the merge of both does not compile, so vet and the unit tests fail on master. |
||
|
Chris LuandGitHub
|
5f6dd4d3e5 |
iceberg maintenance: keep the snapshots that branches and tags pin (#10774)
* iceberg maintenance: keep the snapshots that branches and tags pin expireSnapshots only ever protected the current snapshot, so a snapshot held by a tag or a non-main branch was expired once it aged out of the retention window. iceberg-go's RemoveSnapshots drops any ref whose snapshot is gone without complaint, so the tag disappeared and the files behind it were deleted as unreferenced. Protect every ref target, and honour a branch's own min-snapshots-to-keep / max-snapshot-age-ms over the ancestors behind its head. Detection skips pinned snapshots for the same reason: proposing a job whose only outcome is a no-op keeps the worker busy forever. * iceberg maintenance: re-plan when a ref appears mid-commit, and stop proposing no-op expiry Three follow-ups from review of the ref-aware expiry: The commit guard only compared the table head, so a tag created between planning and commit could pin a snapshot the plan was about to expire. Re-check the refs against the metadata the commit actually reads. Detection now asks snapshotsToExpire what execution would remove instead of approximating with its own count-and-age rules. Expiry always requires a snapshot past the retention window, so a table over the quota whose snapshots are all young was being proposed for a job that could only no-op. The branch retention test could not tell "retained the whole lineage" from "honoured min-snapshots-to-keep", because the branch had exactly as many ancestors as the count. Give it one more, and cover max-snapshot-age-ms too. Both need snapshots genuinely older than a retention window, which iceberg-go will not accept at build time, so the fixture backdates the metadata after building it. * iceberg maintenance: fold the metadata test builders back into one buildTestMetadata, buildTestMetadataWithRefs, buildTestMetadataAged and buildTestMetadataNow were four names for one thing. Keep the original and give it the refs and age it needs. |
||
|
Chris LuandGitHub
|
eef6f3d1e6 |
s3tables: add the maintenance configuration APIs (#10773)
* s3tables: add the maintenance configuration APIs Stores the configuration verbatim as the wire shape under a new s3tables.maintenance extended attribute, so Get hands back what Put took and no translation layer can drift from the AWS model. Nothing reads the configuration yet. Put merges a single type into the stored map so configuring compaction does not drop snapshot management, and asserts the attribute's prior value so two concurrent Puts cannot silently clobber each other. * iceberg: apply the maintenance configuration in the worker The worker now reads the per-table and per-bucket maintenance configuration written by the control plane, so the wildcard plugin config is a default rather than the only setting a table can have. Table properties still win by default, since a table declaring its own layout is what every engine honours and the compactor has to agree with whoever writes the files. Clearing table_properties_override makes the maintenance configuration authoritative instead. Status is not part of that contest: a disabled type drops its operations and no property can re-enable them, so the operator's kill switch always holds. Manifest and delete-file rewrites have no AWS equivalent and ride with compaction. Detection reads both attributes from entries it already lists. * s3tables: report maintenance job status The worker records the outcome of each run in its own extended attribute, separate from the configuration so operator and worker writes do not contend, and GetTableMaintenanceJobStatus reads it back. Only the types a run touched are written, so a partial run cannot erase what an earlier one recorded. The reader fills in the rest: Disabled when the configuration switched a type off, Not_Yet_Run otherwise. Status is advisory, so a lost race is logged rather than failing a job whose work already committed. * s3tables: route the maintenance APIs over REST The five actions were only reachable by X-Amz-Target dispatch, which the AWS CLI and SDK do not use for this service. They address the operations by path, so the APIs were unreachable from any official client. * s3tables: fix the table bucket ARN field name GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the wire field is tableBucketARN, as every other response in this package already spells it. Official SDK deserializers ignore the unknown key, so the required field came back unset. * s3tables: carry the compaction strategy through to the worker IcebergCompactionSettings modelled only targetFileSizeMB, so a request naming a strategy was accepted and then dropped on the way to storage. The worker now maps binpack and sort onto its own rewrite strategy and lets auto defer to the worker configuration. z-order is rejected rather than accepted and quietly binpacked. * s3tables: report bucket-level maintenance status GetTableMaintenanceJobStatus read only the table's configuration, so unreferenced file removal — which is configured on the bucket — reported Not_Yet_Run or a stale success after an operator disabled it. The merge helper now lives in this package and the worker shares it. * iceberg: delete orphans only after the non-current window AWS marks a file non-current once it has been unreferenced for unreferencedDays, then deletes it a further nonCurrentDays later. The cutoff was taken from unreferencedDays alone, so a 3/10 configuration hard-deleted on day three and threw away the ten day recovery window. remove_orphans deletes in one step rather than marking, so the cutoff is now the sum of the two. * s3tables: assert every attribute when rewriting an entry UpdateEntry writes the whole entry back from the snapshot the caller read, and its precondition only covers the keys the caller names. Both maintenance writers named one key, so a job status write could revert a maintenance configuration an operator had just disabled, turning an advisory write into a silent re-enable. Both now assert the entry's full attribute set, including the target key when absent so a concurrent create also fails the precondition. * s3tables: assert absent attributes when rewriting an entry The precondition covered the attributes present when the writer read the entry, so an attribute created between that read and the write was absent from it. A first-time PutTableMaintenanceConfiguration disabling a type therefore lands, passes the per-key checks, and is then deleted by the stale whole-entry write. Every attribute this package stores is now asserted, absent ones included. The metadata commit and planning index writers rewrite the same entries and had the same exposure, so both use the shared snapshot too. * iceberg: implement the auto compaction strategy auto was accepted, stored and read back, but left the worker on its own default, so a sorted table configured as auto was compacted with binpack. AWS defines auto as sorting tables that declare a sort order and bin-packing the rest. That needs the table metadata, so the choice is made where the rewrite plan is resolved: an unsorted table falls back to binpack rather than failing the way an explicit sort request does. * s3tables: validate the maintenance setting ranges PUT accepted zero, negative and oversized values for every numeric setting. The worker then ignores a non-positive value and saturates an oversized one, so the configuration read back was not the one that ran. AWS bounds all five to 1..2147483647, which is now enforced. The fields are pointers so an explicit zero is distinguishable from an omitted one and can be rejected rather than silently ignored. * s3tables: give every entry writer the same compare-and-swap updateExtendedAttribute asserted the entry's attributes, but the helpers behind the metadata, policy and tag handlers still wrote the whole entry unconditionally. Any of them could land on a stale snapshot and delete a maintenance configuration an operator had just written. They all share one read-modify-write loop now, so the precondition and the bounded retry apply wherever an entry is rewritten. * s3tables: move the maintenance configuration with a renamed table RenameTable carried the metadata, version, policy and tags to the new name but left the maintenance configuration and job status behind. A table with snapshot management disabled came back enabled under its new name, and the stale configuration stayed on the old name where a table created there would inherit it. The decoupled-delete cleanup left the same two attributes behind. * s3tables: accept every AWS partition in ARNs The route regexes and the ARN patterns both hardcoded arn:aws, so valid aws-cn and aws-us-gov ARNs never reached a handler. The router now shares the partition-tolerant prefix with the parser, and a generated ARN uses the partition its region belongs to so it parses back. * s3tables: generate ARNs in the region's partition The handler's own ARN generators still formatted arn:aws directly rather than going through the partition-aware builder, so a China or GovCloud deployment routed the request but then returned a commercial ARN and matched IAM policies against it. The round-trip test missed this because parsing accepts any partition, so it now asserts the prefix the region implies. * s3tables: complete the ARN partition table aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-* and eusc-* regions fell through to the commercial partition. * s3tables: do not let a rename swallow a concurrent maintenance write Rename copied the source attributes early and cleared the source at the end, so a Put landing in between missed the copy to the destination and was then deleted by the cleanup. It succeeded and vanished. The cleanup now clears the source only while it still holds exactly what was copied, and returns a conflict otherwise. Put checks the catalog identity inside the same conditional mutation, so it also cannot write to a name that a rename or delete has already soft-deleted. |
||
|
Chris LuandGitHub
|
a1d3fe236f |
iceberg: let table properties override the worker config (#10772)
* iceberg: carry snapshot retention in milliseconds Config stored retention as hours, so any sub-hour value would have to be truncated to 0 and then clamped back up to the 168 hour default. Keep the plugin config key in hours and convert once at parse time. * iceberg: let table properties override the worker config Every other Iceberg implementation lets a table's own properties win over engine defaults; the worker ignored them entirely. A writer honouring write.target-file-size-bytes and a compactor rewriting to the plugin config's size would rewrite each other's output forever. Resolved once per job rather than per operation, so compaction committing new metadata mid-job cannot change the settings underneath it. * iceberg: clamp the orphan cutoff so it cannot overflow collectOrphanCandidates converts the cutoff to a time.Duration. Past roughly 2.5 million hours that multiplication wraps negative, putting the cutoff in the future so every file walked looks like an orphan and gets deleted, including data a concurrent writer has not yet committed. Reachable today through orphan_older_than_hours. |
||
 
|
4c40ec3a9e |
master: align default volume size with EC rows (#10761)
Co-authored-by: joe <joe@gmail.com> |
||
|
Chris LuandGitHub
|
b45f8314c5 |
ec.encode: require the shards to agree on size before deleting the volume (#10769)
* ec.encode: require the shards to agree on size before deleting the volume Before an encode deletes the volume it just encoded, it asks whether enough shards exist and whether they are spread across nodes. Both are questions about presence: nothing asks whether those shards are whole. Every shard takes one piece of each block row, so they are all written to the same length. One that disagrees was truncated, half copied, or landed on a disk that filled up -- and counting cannot see it, so the source volume is deleted on the strength of a set that cannot rebuild it. Compare the sizes the cluster already reports (shard_sizes travels in the heartbeat) and hold the deletion back when they disagree, naming the odd shard and its holder. Sizes reported as zero are skipped rather than read as a disagreement: a volume server that predates shard-size reporting, or one that has not heartbeated them yet, must not strand every encode in the volume-plus-shards state this check exists to avoid. * ec.encode: judge shard sizes on the newest encode generation only The size check collected every shard the master reports for the volume, while the recoverability check beside it counts only the newest encode generation. A re-encode can change the ratio, so an orphaned older generation -- one the pre-encode sweep could not reach, but the master still hears about -- has shards of a different length by nature. Merging those into the comparison makes a healthy current set look inconsistent, and because the orphan keeps being reported, every retry fails and the encode is left holding the volume and its shards for good. Collect sizes the way CollectEcShardBitsByNode collects bits: fenced to the newest EncodeTsNs, with unstamped entries forming the one legacy generation. |
||
|
Chris LuandGitHub
|
fbd85d31b0 |
ec.decode: check the rebuilt .dat is complete before the shards can be deleted (#10768)
A decode ends by deleting the shards it read, and the only thing standing between that and a bad reconstruction is verifyDecodedVolumeBeforeDelete, which asks whether .dat and .idx are non-empty. A .dat truncated to a single byte passes, and the shards -- the only other copy of everything past the cut -- are deleted on the strength of it. The server already knows the answer it never checks: FindDatFileSize returns the extent the EC index references, and WriteDatFile rebuilds to it. Compare the two once the file is written and fail the decode instead of reporting a short volume as a good one. Longer than the extent still verifies -- padding is not missing data -- so only a genuinely short rebuild is rejected. Needle counts cannot answer this: .idx is written from .ecx, so the count matches by construction and a truncated .dat still reports every needle. |
||
|
Chris LuandGitHub
|
829064af71 |
ec.decode: finish the cleanup an interrupted decode left behind (#10767)
A decode deletes the shards only after the regenerated volume is mounted and verified, so a run interrupted in that last phase leaves the volume in place with its shards partway through deletion. The re-run then finds both, tries to collect the shards again to rebuild a volume that already exists, and fails on the first shard the interrupted run had removed: generate normal volume 3 ...: ec volume 3 missing shard 6 Nothing recovers from there: the shard set is deliberately being destroyed, so every retry fails the same way while the decoded volume sits there, already complete. Finish that cleanup instead. A volume beside the shards is not enough to act on -- an encode interrupted before it deleted the original leaves the same shape, as does a decode killed while generating, whose volume may be half written -- so require a data shard to be gone. Only the deletion phase removes one, and it is also exactly the state no decode can recover from, so finishing is the only move left rather than a choice between two. The deletion still runs behind verifyDecodedVolumeBeforeDelete, the check that guards it in a normal run. |
||
|
Chris LuandGitHub
|
97a155d14d |
admin: show capacity per storage tier and stop counting remote-tiered bytes as local disk usage (#10766)
* admin: show capacity per storage tier and stop counting remote-tiered bytes as local disk usage A remote-tiered volume reports its cloud object's size, so summing volume sizes inflated the dashboard's used-vs-capacity numbers (the local .dat is gone after volume.tier.move). Split the accounting: DiskUsage now only counts bytes on local disks, with the cloud bytes surfaced separately per server and per remote storage name. The dashboard gains a Storage Tiers table breaking volumes and EC shards down by tier (each local disk type plus each remote storage), using the per-disk-type statfs numbers already in the VolumeList response. The volumes page badges remote-tiered volumes with their storage name, and the EC shards page fills in real per-shard sizes instead of hardcoding 0. * admin: review fixes for the tier capacity display - A disk that predates disk_total_bytes now contributes its logical bytes to the tier's DiskUsed, so a tier mixing old and new volume servers doesn't underreport usage; the usage bar always reflects the displayed Disk Used value (the DataSize fallback in UsagePercent is gone, and the percent math is overflow-safe). - getTopologyViaGRPC defaults a zero VolumeSizeLimitMb to 30000 MB like GetClusterVolumeServers, keeping slot-based capacities consistent. - The dashboard volume-servers column reads Usage / Capacity to match its cell content, and the hdd disk-type default is shared between the volumes-page badge and countUniqueDiskTypes. |
||
|
Chris LuandGitHub
|
1c926e8fac |
test: systematic EC interruption verification — exhaustive model check + deterministic kill matrix (#10764)
* ec: bounded-exhaustive model check of the volume lifecycle The randomized chaos harness samples the state space; this enumerates it. The lifecycle is a state machine whose steps mirror the pipelines in this package, and the checker explores every schedule within the bound: a crash at every step boundary, an error return running the rollback (itself crashable at every step), a volume-server restart applying the startup reconciliation rules in every quiescent state, and the prescribed restart-based recovery from every crashed state. Checked in every reachable state: durability (a readable copy always exists), at most one generation mounted, and — a property the sweep discipline turns out to guarantee — at most one generation's files on disk. From every quiescent state the recovery must converge to a clean volume. Runs in well under a second. * test: deterministic EC interruption matrix Enumerate every phase of every interruptible EC operation and kill a real weed shell exactly when the phase announces itself on the command output, instead of at a random moment: four encode phases, four decode phases, and the balance's move phase (set up with -rebalance=false so a move is guaranteed). Each scenario prepares its precondition, kills at the marker, runs the prescribed recovery, and verifies every stored byte still reads back identical. The interruption recoveries move out of the randomized ops into shared chaosRun helpers both drivers use. * test: make the randomized EC chaos walk opt-in The systematic layers — the interruption matrix and the lifecycle model check — carry the CI coverage deterministically; the randomized walk stays for exploratory runs, behind EC_CHAOS_SEED. * ci: bound the EC integration suite by the job budget, not go test's default The suite with the interruption matrix runs close to the default 10m binary timeout on slower runners. * test: require every interruption-matrix marker to appear A marker that never prints means a pipeline refactor renamed or dropped the progress line; silently degenerating into a no-interruption run would let CI pass without exercising the boundary the scenario names. Also recheck the marker channel after the wait: a shell that prints and exits at once makes both channels ready, and select picking the exit case must not report a printed marker as missed. |
||
|
Chris LuandGitHub
|
602746f51d |
test: EC lifecycle chaos harness, with four fixes it found (#10763)
* ec: let the encode's balance see a migrating volume's shards across disk-type buckets Shard generation writes beside the source .dat, so a cross-tier encode (source on hdd, -diskType=ssd) leaves the fresh shards in the source disk-type bucket. The encode's internal balance ingested only the target bucket, saw no shards, and planned no moves; the spread guard then correctly aborted the encode (and before that guard existed, the shards silently stayed clumped on the generation host in the wrong tier). EcBalance now takes the encode batch as migratingVolumeIds and ingests those volumes' shards from every bucket, while everything else keeps the bucket filter so a plain ec.balance never drags deliberately tiered shards onto another disk type. The in-memory model delete also becomes bucket-agnostic: a node holds a given shard in exactly one bucket, and a bucket-scoped delete missed cross-bucket moves in the dry-run model. * volume: decode reads shard 0 from its resolved path, not the EC volume's base dir On a multi-disk server a volume's shards can sit on several disks; the store registers each shard with its own path and CollectEcShards resolves them, but FindDatFileSize derived the .ec00 path from the EcVolume's base directory. When shard 0 lived on a sibling disk, VolumeEcShardsToVolume failed with 'open ...ec00: no such file or directory' and ec.decode aborted. * ec: decode re-copies shards the topology claims but the target does not hold An interrupted earlier decode or balance can leave the master believing the decode target holds a shard whose file never landed: the mount registered but the partial copy was cleaned, or the file was swept. The collect step took the topology's word for it, excluded the shard from the copy set, and the decode failed with 'missing shard'. Probe the target's live inventory (VolumeEcShardsInfo) and treat anything it cannot serve as still-to-copy. * ec: decode discovers shards across disk-type buckets Shards sit wherever encode generation and balance left them: a cross-tier encode leaves them in the source disk-type bucket, a partial migration straddles buckets. ec.decode scoped its shard discovery to the -diskType bucket and reported a decodable volume as having no shards at all. Union across buckets, the way the encode's shard verification already does. * test: EC chaos lifecycle harness Randomized, seeded sequences of the EC lifecycle against a live cluster in the production-shaped layout: multiple data disks per server, a separate -dir.idx directory so .ecx/.ecj sidecars are shared across disks, and a tagged ssd tier. Operations cover encode (hdd and ssd targets), balance, shard damage plus rebuild, decode, re-encode, deletes, scrub, tier moves, crash-restarts, sidecar fault injections (a data-dir .vif pushed into the shared idx dir; a stale-generation shard planted beside a newer encode), and interruptions: a real weed shell subprocess killed mid-encode, mid-decode, and mid-balance, with the recovery re-run required to converge. One invariant holds after every step: every stored byte reads back identical and every deleted needle stays deleted. EC_CHAOS_SEED and EC_CHAOS_STEPS make runs reproducible and scalable. A known gap is tolerated and logged rather than fixed here: a shard mounted on two disks of one node (orphan adoption after an interrupted copy) is invisible to ec.balance's dedup and unaddressable by ec.shard.unmount's shard@address form, so no cleanup path exists yet. * test: fail payload-corruption checks on the test goroutine t.Fatalf inside require.Eventually's condition runs on the poller's goroutine, where Goexit kills only that goroutine and the corruption message can be lost behind a generic timeout. Record the mismatch, end the polling, and fail on the test goroutine. Also assert the full shard count in the cross-bucket decode-discovery test. |
||
|
Chris LuandGitHub
|
944d967502 |
refactor: extract EC orchestration into a shared weed/ec package (#10760)
* shell: move ErrorWaitGroup to weed/util * shell: remove unused CandidateEcNode and EcRack types * ec: extract EC orchestration logic from weed/shell into weed/ec Move the EC node/topology model, balance engine, encode pipeline, decode pipeline, and rebuild engine into a new weed/ec package so shell commands and maintenance workers can share the logic. Shell commands keep flag parsing and delegate through a small ec.Env (dial option, topology fetch, volume locations, lock check). Tests move along with the code. * shell: remove unused proportional-rebalance type stubs * ec: move scrub, replication check, and shard unmount engines into weed/ec * worker: share the EC generation-aware shard counter from weed/ec * ec: gofmt * shell: drop EC aliases with no remaining callers * ec: guard a missing topology hook and nil disk entries in topology helpers * ec: drop trailing newlines from decode error strings * ec: re-check the shell lock before applying shard unmounts * shell: trim -node entries in ec.scrub |
||
|
Chris LuandGitHub
|
f66d6ffc4a |
s3: option to disable bucket auto-creation on upload (#10759)
* s3: add option to disable bucket auto-creation on upload * command: expose -autoCreateBucket in s3, filer, server, and mini * s3: apply the bucket auto-create policy to directory marker uploads * s3: validate the bucket name before the auto-create disabled check * s3: cover the disabled auto-create gate at all three upload entry points |
||
|
Chris LuandGitHub
|
02b3ec6e90 |
sftp: url-encode the upload path (#10758)
sftp: url-encode the upload path so filenames can't inject filer query commands
The SFTP put handler concatenated the user-controlled filename straight into
the filer upload URL, so a name containing "?" was parsed as a query string.
Build the URL via url.URL{Path: ...} so "?" becomes %3F and stays a literal
path character.
|
||
|
Chris LuandGitHub
|
c2ea452b9d |
skiplist: fix TestFindGreaterOrEqual flake (compare against largest key, not value) (#10757)
skiplist: compare against the largest key, not its value, in TestFindGreaterOrEqual |
||
|
Chris LuandGitHub
|
d713ab49f9 |
volume: validate replica targets and restrict gcs credentials in FetchAndWriteNeedle (#10755)
* volume: validate replica upload targets in FetchAndWriteNeedle The replica leg forwarded the fetched needle to a caller-supplied address without checking it, so a malformed target could redirect the upload to an unintended host or path. Require each replica target to be a bare host:port whose host is not loopback / link-local / unspecified, reusing the address deny-list; cluster peers legitimately sit on private networks, so RFC 1918 / CGNAT stay allowed and -volume.allowUntrustedRemoteEndpoints still opts out. Validate every target up front so a bad one fails the request before the local write, and upload through a client that re-checks the resolved address at connect time so a replica hostname cannot rebind to a blocked address after validation. Mirrored in Rust (validation moved ahead of the local write; the Rust S3 path's connect-time re-check is still a follow-up there). * volume: only accept inline gcs credentials in FetchAndWriteNeedle The gcs credentials value on this request could name a local filesystem path, which the SDK reads from disk. Accept only inline JSON here; the server-side GOOGLE_APPLICATION_CREDENTIALS env var still supplies a path. The Rust volume server has no gcs backend, so there is nothing to mirror. |
||
|
Chris LuandGitHub
|
9125b9c835 |
volume: extend the remote-endpoint guard to the azure backend (#10754)
* remote_storage/azure: allow a per-request HTTP client Thread an optional *http.Client through NewAzBlobClient and add azure.MakeWithHTTPClient, mirroring the S3 backend. When set, the client overrides the azblob transport so a caller can pin the dial path. The existing makers pass nil, so behavior is unchanged. * volume: extend the remote-endpoint guard to the azure backend The endpoint validation and rebinding-safe dialer in FetchAndWriteNeedle covered the S3-SDK backends. The azure backend also dials a caller-supplied AzureEndpoint, so route both families through a single guardedRemoteClient helper that returns the endpoint each backend dials and a constructor bound to the guarded HTTP client. azure is guarded only when AzureEndpoint is set; an empty endpoint derives the public host from the account. -volume.allowUntrustedRemoteEndpoints still opts out. * rust volume: assert the azure endpoint has no remote-client path The Rust volume server has no azure backend, so make_remote_storage_client rejects the type before any client is built. Add a regression test pinning that invariant. |
||
|
Chris LuandGitHub
|
94f8e2caf9 |
EC: handle zero-sized shard files uniformly (moves, rebuilds, startup cleanup) (#10753)
* volume_move: treat zero-sized EC shards as absent in move verification A zero-sized shard file is residue of a failed operation (issue 10730), not a shard - but VerifyEcShards only checked presence, so a copy that landed as an empty file passed verification and the source was deleted behind it. Size zero now reads as absent, with a distinct error naming the zero-sized shard so the operator can tell a broken copy from a missing one. * storage: exclude zero-sized EC shards from rebuilds and clean up stale ones The reproducer in issue 10730: a zero-sized shard file left by a failed operation was selected as a Reed-Solomon input and failed the whole rebuild with an input size mismatch, because input discovery checked existence, not substance. - RebuildEcFiles treats a zero-sized shard file as missing and regenerates over it in place (the reclassified-corrupt path: temp file beside the residue, atomic rename). - The startup/rescan shard loader, which always skipped zero-sized files, now deletes them once they are older than an hour - young enough files can be an in-flight copy's just-created file, since the same scan runs from LoadNewVolumes while serving. Regression tests: a rebuild with one emptied shard regenerates it byte-identical; the loader deletes a stale zero-sized shard and leaves a fresh one alone. * storage: age-check each zero-shard cleanup candidate individually The shard scan merges the data and idx directory listings, so the age-checked entry and a deletion candidate can be different files sharing one name - a stale zero-sized file in one directory next to a fresh same-named file in the other (possibly an in-flight copy's just-created one) could get the fresh file deleted. Each candidate's own modification time now decides, both directories are handled in one pass, and the split-directory case is pinned by a test. |
||
 
|
9386a25a4a |
feat(shell): parallelize volume balance moves (#10737)
* chore: volume.balance parallelization * chore: volume.balance add ioBytePerSecond --------- Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work> |
||
|
Chris LuandGitHub
|
0de7ff5eb8 |
ci: run the gated redis store tests (#10746)
* redis2: route the orphan cleanup existence checks to the master * scaffold: the redis_cluster2 read routing key is useReadOnly * ci: run the gated redis store tests * redis2: poll for the redis expiry instead of a fixed sleep * redis2: assert the value key exists before testing its expiry |
||
|
Chris LuandGitHub
|
0481f712b1 |
redis2: orphan cleanup existence checks must not read replicas (#10745)
* redis2: route the orphan cleanup existence checks to the master * scaffold: the redis_cluster2 read routing key is useReadOnly |
||
|
Chris LuandGitHub
|
7f27c572c4 |
log_buffer: end bounded reads that find the buffer empty (#10750)
A bounded LoopProcessLogData (stopTsNs set) on a buffer that never took a write since process start fell into the ResumeFromDiskError branch, which never checks stopTsNs when ReadFromDiskFn is nil and HasData() is false. The read parked on the notification loop forever while the subscription's idle heartbeats kept the stream looking alive, so a bounded SubscribeMetadata pass on a freshly restarted idle filer never completed. Terminate like the caught-up path does, returning a nil error: leaking the pending ResumeFromDiskError would latch the filer's outer loop into its gap machinery, which parks the bounded subscriber all over again. |
||
|
Chris LuandGitHub
|
4f50c5b0d4 |
feat: throughput limits for replicate, EC shard, and worker-driven moves (#10749)
* feat: throughput limits for replicate, EC shard, and worker-driven moves VolumeCopy was the only rate-limitable transfer; EC shard copies, replica creation, and worker-driven moves all ran at whatever the receiving server's maintenance rate allowed, with no per-operation control. - proto: VolumeEcShardsCopyRequest and the balance / ec_balance task params and configs gain io_byte_per_second; 0 keeps today's behavior (the volume server's own maintenance rate governs). - volume server: VolumeEcShardsCopy throttles with one WriteThrottler per request, shared across the shard, .ecx, .ecj, .vif, and .ecsum copies so the limit caps the transfer as a whole - the same shape as VolumeCopy. - volume_move: ReplicateVolume accepts the limit; EcMoveOptions carries it through MoveEcShards/CopyAndMountEcShards into the copy request, with fake-client tests asserting propagation. - shell: ec.balance gains -ioBytePerSecond; volume.tier.move's replication top-up honors the command's existing -ioBytePerSecond instead of running unthrottled. - worker: balance and ec_balance configs gain io_byte_per_second (surfaced in the admin config schema), carried through detection and plugin job parameters into task params and handed to the shared mover; batch balance jobs inherit the limit from their detection results. The limit is per copy stream, so maxParallelization multiplies the aggregate ceiling. * worker plugins: expose io_byte_per_second in the plugin config and derive it The plugin-driven detection path derives its task Config from the plugin configuration values, and both balance and ec_balance left IoBytePerSecond at zero there - a configured limit silently reverted to the server maintenance rate. Both derive functions now read the field (clamped at zero), and the plugin descriptors expose it with defaults so the configuration form carries it. |
||
|
Chris LuandGitHub
|
7d0fff32db |
redis2: expire entries without destroying a concurrent recreate (#10744)
* redis2: expire entries without destroying a concurrent recreate * redis2: repair the member when redis expiry wins the compare-and-delete race |
||
|
Chris LuandGitHub
|
4500bdf88e |
iceberg: accept lowercase parquet file format when planning compaction (#10751)
* iceberg: accept lowercase parquet file format when planning compaction * iceberg: expect absolute added-file paths in compaction integration test |
||
|
Chris LuandGitHub
|
76d3fd0e9d |
grpc: optional client_cert/client_key for outgoing mTLS connections (#10747)
* grpc: optional client_cert/client_key for outgoing mTLS connections * scaffold: list client_cert/client_key in each grpc section |
||
|
Chris LuandGitHub
|
abd36cbf92 |
redis2: harden the orphaned index member cleanup (#10743)
* redis2: derive the orphan cleanup keys inside the helper * redis2: skip orphan cleanup in super large directories * redis2: detach orphan cleanup from the request context and log a failed restore * redis2: keep a directory member whose child index is still live * redis2: run restore-path tests under both key prefixes and fix the test harness * redis2: check cleanup errors in tests |
||
|
Chris LuandGitHub
|
4fb5d15019 |
redis: remove orphaned directory index members on listing (#10742)
* redis: remove orphaned directory index members on listing * redis: check cleanup errors in tests |
||
 
|
f7ae2d4dd5 |
fix(redis2): remove orphaned directory index members on listing (#10735)
* fix(redis2): remove orphaned directory index members on listing ListDirectoryEntries skipped index members whose value key was gone and left them in the ZSET, so the per-directory child index grew without bound under any TTL workload. Mirror the ZRem the logical-expiry branch already performs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(redis2): keep the index member when a concurrent insert recreates the value The orphan cleanup removed the member unconditionally, so an InsertEntry landing between FindEntry and the ZRem left a live value with no index member, invisible to listings until another InsertEntry on that path. UpdateEntry does not re-add it, so the loss persisted. Restore the member when the value is present again after the removal. The value key and the directory index key hash to different slots, so a Lua script or MULTI over both is not available to the cluster store. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Chris LuandGitHub
|
8714f42abf |
erasure_coding: share the EC shard teardown primitive (#10740)
The unmount+full-teardown of EC shards was duplicated: the plugin-worker EC task had unmountAndDeleteEcShards and the shell had unmountAndDeleteEcShardsQuiet, byte-identical apart from a fence parameter and a sentinel error. That duplication is how the teardown fence semantics drifted between the two paths. Distribute, mount and verify already live in weed/storage/erasure_coding and are shared by both callers; move the teardown there too, as UnmountAndDeleteEcShards plus the shared ErrFullTeardownNotAcked sentinel. Both paths now call the one function, so the fence semantics cannot diverge again. The shell keeps a thin type-converting wrapper and aliases the sentinel; behavior is unchanged. |
||
|
Chris LuandGitHub
|
6408f32232 |
EC worker: clear stale/interrupted shards at task start and on failure (#10738)
* EC worker: clear stale/interrupted shards at task start and on failure
The EC encode task cleared stale shards from a prior interrupted encode only
at 55% progress (after mark-readonly, copy, and generate), and used a
generation-fenced teardown. Two gaps left orphan shards behind:
- a retried encode's prior attempt carries the same admin-issued encodeTsNs,
and the server's teardown fence preserves same-or-newer generations, so the
prior attempt's shards were never cleared;
- shards left by an interrupted distribute often have an unreadable .vif
generation (the sidecar never landed), which the fence also preserves.
Both survive the next volume-server restart as orphans and make detection
refuse the volume (Manual intervention required).
Move the cleanup to a Step 0 preflight that runs before any destructive step,
and switch it to the server's blanket (generation-independent) teardown -- the
same wipe the shell ec.encode pre-cleanup uses. The admin dedupe key already
prevents a concurrent newer encode of the volume, and the blanket path aborts
rather than clobber a live newer mount.
Add rollbackDistribute: a failure after distribute begins but before verify
commits the EC copy now tears down the shards it wrote and restores the sources
to writable, so a terminally-failed encode (a single-attempt job, or the last
of a retry series, which has no successor preflight) leaves nothing behind.
The preflight also rejects a plan with no targets or no source before marking
the source readonly.
* EC worker: reject malformed targets and keep source readonly on incomplete teardown
Address review feedback:
- ensureCleanEcStart only rejected an empty target slice; a target with an
empty Node (or no shard ids) passed the length check, was then silently
skipped by cleanupStaleEcShards, and let Execute mark the source readonly
with nothing to distribute to. Validate each target before the first
destructive step. Add regression cases.
- rollbackDistribute marked the source writable even when the shard teardown
returned an error, exposing a writable source beside stale (possibly mounted)
shards -- reads/writes could diverge and orphan cleanup will not remove a
writable source. On an incomplete teardown, leave the source readonly for the
next preflight or an operator to reconcile.
|
||
|
Chris LuandGitHub
|
fa48ce20fc |
shell: roll back a failed ec.encode instead of leaving readonly volumes and orphan shards (#10741)
* shell: roll back a failed ec.encode instead of leaving readonly volumes and orphan shards ec.encode marks the source volumes readonly and generates EC shards before it verifies the shards and deletes the originals. If any step in between failed, the command just returned the error: the volumes were left readonly and the partially-produced EC shards survived as orphans, cleaned up only by the next ec.encode run (via clearPreexistingEcShards) if the operator retried. Add a deferred rollback that runs when the batch fails before the originals are deleted: it tears down the EC shards produced this run and restores the sources to writable, reusing the existing clearPreexistingEcShards and markVolumeReplicaWritable helpers. Once the shards are verified recoverable the batch is committed to the EC copy and does not roll back. Both rollback steps are idempotent, so a failure before the volumes were marked readonly is safe. * shell: re-read volume locations when restoring writable in ec.encode rollback Address review: rollbackFailedEcEncode restored writable using the location snapshot taken before doEcEncode, but doEcEncode re-reads locations and marks every replica of that later snapshot readonly. A replica added or moved in between would be left readonly. Re-read locations in the rollback and fall back to the pre-encode snapshot only if the re-read fails. |
||
|
Chris LuandGitHub
|
db5a086d04 |
read cold remote objects straight from the origin while caching (#10731)
* refactor: extract remote mount resolution into shared helpers * refactor: share the adaptive remote cache wait policy * filer: stream cold remote reads from the origin while caching * s3: stream cold remote reads from the origin instead of 503 retries * test: cover the S3 origin stream-through path * remote mounts: match on path components and prefer the longest mount * fail short origin streams instead of silently truncating * s3: try the origin before failing a cold read on a local cache error * s3: gate origin streaming on the entry's resolved version * return the cache RPC's NotFound as a canonical status and classify it everywhere * filer: keep multipart-range cold reads on the retry path |
||
|
Chris LuandGitHub
|
a0347ca545 |
test: assert EC shard identity and empty-view in multi-disk lifecycle tests (#10723)
test: assert EC shard identity and empty-view, not just counts, in lifecycle Follow-up to the multi-disk EC lifecycle tests (#10721), addressing review feedback. The phase checks compared shard counts. A reconcile that put a shard on the wrong disk, or loaded a different shard than the file on disk, keeps 6/5/3 right while corrupting the mapping. Compare the exact registered shard set per disk at every phase instead, via a shared assertRegistered helper. The cross-disk mount phase now also pins that shard 0 landed on disk2 with the existing shards, not merely that it is findable. The sidecar-disk-lost scenario only logged the registered view, so a change that registered shards without reachable sidecars would pass despite the documented expectation that the view stays empty. It now asserts countRegistered == 0: a registered-but-unreadable shard is worse than an unregistered one, because the master advertises it. The first store's closer is now deferred as a closure the moment the store is created, so a Fatalf in an early phase no longer leaks it and its notification-drainer goroutine; the closure reads the reassigned variable so it also covers the post-restart store. |
||
|
Chris LuandGitHub
|
78e7e04377 |
plugin scheduler: drain started jobs past the window close instead of killing them (#10728)
* plugin scheduler: drain started jobs past the window close instead of killing them * plugin scheduler: never drain-cap an attempt below its declared estimated runtime * plugin scheduler: cap estimated_runtime_seconds before the Duration conversion |
||
|
Chris LuandGitHub
|
0799084e98 |
refactor: share volume and EC shard move logic between shell and workers (#10727)
* operation: add shared volume_move package for volume and EC shard moves The shell commands (volume.move, volume.balance, ec.balance, tier moves) and the maintenance workers (balance, ec_balance) each carried their own copy of the move RPC sequences, and the copies had drifted: the worker verified the target before deleting the source but dropped the disk type and IO throttle; the shell passed those but deleted the source unverified. volume_move.Mover carries the merged sequences, keeping the stricter behavior from each side: - LiveMoveVolume: check-then-hard-freeze the source (VolumeStatus's IsReadOnly also covers low-disk and readonly-but-can-delete states, which still accept needle deletes), copy with disk type and IO throttle, tail, verify the target is not behind the source before the destructive source delete (a target that is ahead holds writes it accepted during the tail and the move commits to keep them), and restore the source's writability when a failure precedes the delete and this move did the freezing. Aborts clean up the incomplete target copy; a failed cleanup or an ambiguous source delete keeps the source readonly (ErrSourceKeptReadonly) so callers do not thaw a source next to a possibly-authoritative copy. With a readonly source, an existing or unknown-state target refuses the move outright: no client-side observation can prove such a copy is a stale remnant rather than the authoritative copy of an unfinished move. - MoveEcShards: copy with the .ecx/.ecj/.vif/.ecsum sidecars, mount, verify the target registered every shard before unmount+delete on the source, and reject same-server moves (the EC delete is server-wide). Server identity is the grpc endpoint (SameServer), so node:8080 and node:8080.18080 compare equal while test servers sharing a degenerate HTTP address stay distinct; addresses are validated non-fatally before dialing and before being embedded in copy/tail requests, since both the client dialer and the receiving server normalize them through a parser that aborts the process on a malformed port. The Rust volume server's codes.NotFound counts as a definitively absent probe answer alongside the Go server's plain-error code Unknown. All RPCs go through an injectable ClientFunc, so the sequences are unit tested against a fake volume server client: RPC order, request fields, and that verification failures keep the source intact. * shell, worker: delegate volume and EC shard moves to operation/volume_move LiveMoveVolume and the copy/tail/delete/mark-writable helpers become thin wrappers over the shared mover, keeping their signatures; the EC helpers keep their per-step output and delegate the RPCs. BalanceTask and ECBalanceTask keep their parameter validation, progress reporting, and guards (same-node cross-disk rejection, dedup keep-node verification, shard ids range-checked before the uint8 narrowing) and hand the RPC sequences to the mover. volume.tier.move skips its thaw-on-failure when the mover deliberately kept the source readonly, since reopening the replicas beside a possibly-authoritative target copy would fork the volume. The tail-failure tolerance moves inside the mover: a failed tail is tolerated only when the volume was already readonly before the move began, backstopped by a stability re-read across the idle window, so volume.balance's -skipTailError-by-readonly heuristic and tier-move's unconditional skip both become the same authoritative rule. * volume_move: keep the source readonly when a failed copy leaves a target of unknown origin A failed copy can leave a complete, mounted copy on the target (the server finishes after the client loses the stream). The abort probed the target only when its pre-copy state was known-absent; an unknown prior state skipped both the probe and the cleanup and then reopened the source - two writable replicas of one volume, diverging from the next write on. The abort now probes the target on every failed copy and restores the source only when the target provably holds nothing. A copy whose provenance cannot be proven (unknown prior state, a pre-existing replica, or an unreachable target) is never deleted, and the source stays readonly with ErrSourceKeptReadonly naming the recovery. * test: teach the plugin worker harness the shared move sequence The fake volume server lacked VolumeStatus, which the shared mover now issues before freezing the source, and the batch execution test's status-read accounting predates the pre-copy target probe and the verification reads. Mirrors the harness the enterprise tree already carries. |