Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: let the sweep release volumes that stay empty and quiet
Vacuuming reclaims bytes but not slots: a fully emptied volume stays
registered to its collection forever, and since growth is gated only on
slot count a store at 99% free disk can still refuse writes to other
collections (#11429). volume.deleteEmpty exists but is manual-only.
With -vacuumDeleteEmptyAfterSeconds (or master.vacuumDeleteEmptyAfterSeconds
under weed server/mini; default 0, off) the automatic sweep now deletes
replica copies that have stayed empty and quiet for that long, the same
rule volume.deleteEmpty applies on demand: remote-backed copies are
skipped, and every delete carries the volume server's onlyEmpty /
onlyGarbage guards so a copy written since the last report is refused
rather than removed. Copies that still hold data or were written
recently stay; only a volume whose every copy is deleted leaves the
sweep's work map, sparing a compaction of bytes that are all deleted.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: harden empty-volume sweep against partial and racing deletes
Review follow-up on #11477:
- delete a volume only when every replica copy is a verifiable
empty-and-quiet candidate; deleting the empty copy of a volume whose
sibling holds live files would silently cut its replica count
(greptile P1).
- drain the volume out of the writable list before deleting, the same
drain the compact pass uses, so PickForWrite stops assigning it and
pending writes settle (devin).
- bound the VolumeDelete RPC so one stalled server cannot hold the
vacuum lock indefinitely (greptile P1, reusing allocateVolumeTimeout).
The vid2location panic scenario raised in review does not exist:
VolumeLocationList methods are nil-receiver safe and a missing vid just
fails enoughCopies, so a partially deleted volume skips compaction
instead of crashing the sweep.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: unregister deleted empty replicas and prune the sweep list
A successful VolumeDelete only updates the volume server; the master
still tracked the replica and kept it in the sweep's location list for
the compaction pass (coderabbit on #11477). Unregister the replica right
after its delete succeeds and drop it from the sweep copy, so a partially
deleted volume only compacts copies that still exist.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: pin deleting volumes out of the writable list across heartbeats
Review follow-up on #11477 (greptile): DrainAndRemoveFromWritable only
removed the volume once; a heartbeat landing between the drain and the
replica deletes re-evaluated writability and re-added it, so a client
write could reach a replica whose siblings were already gone and leave
the volume under-replicated when the last copy refused its onlyEmpty
delete.
MarkDeleting records the vid in deletingVolumes — checked inside
setVolumeWritable so heartbeat, capacity-recovery, and admin re-add
paths all hold it out — and UnmarkDeleting releases it once the sweep
finishes the copy pass. A partially deleted volume's surviving replicas
then return to writable through the normal heartbeat path.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* vacuum: restore writability when a sweep delete survives
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(master): honour -volume.fileSizeLimitMB on the master's /submit - #6748
`weed server -volume.fileSizeLimitMB=2048` still refused anything over
256MB, and the reason is not the one the report assumes: the option does
reach the volume server. The master does not use it. Uploads through the
master's /submit are buffered by submitForClientHandler, which passed a
hardcoded 256MB to needle.ParseUpload, so the master rejected what the
volume server it started would have accepted.
The limit is now passed in. `weed master` gains its own -fileSizeLimitMB
with the same 256 default, so a standalone master behaves exactly as
before, and `weed server` and `weed mini` hand it the value their volume
server already got.
* master.follower: take the same upload limit, and say which flag to match
Review found the follower left behind. It serves /submit like the leader
and buffers uploads under the same limit, but kept the fixed 256MB, so a
cluster raised above that would accept an upload through the leader and
refuse the identical one through a follower.
Two smaller points from the same review: the master's flag description
named only the standalone volume server's spelling, and now names the
weed server and weed mini form too; and the under-limit test asserted on
the error message alone, so it would have passed had the limit rejected
that payload with different wording. It now requires the request to get
past parsing.
* master: bound each volume server DeleteCollection, and finish the fan-out
A collection delete fanned out to every volume server holding it with
context.Background(), so a server that accepted the connection and then
went quiet held the whole delete open with nothing to end it. Each RPC is
bounded now, on the same budget allocateVolumeTimeout gives the other
master-to-volume-server admin RPC. The volume server runs the delete to
completion regardless of the request context, so giving up costs the
confirmation and not the deletion.
The walk itself is the caller's, not a per-server one:
- It outlives the caller. A cancelled request must not abandon a
destructive fan-out part-done, with volumes left behind and no request
still running to come back for them.
- It no longer stops at the first server that refuses, which left the
collection on every server after it in the list. The first failure is
still what is reported, and the collection stays in the topology so a
later delete comes back for the rest.
- It sends one RPC per server rather than one per replica.
ListVolumeServers reports a node once for every replica it holds, while
DeleteCollection removes the whole collection from the server it
reaches, so a collection with thousands of volumes repeated the same
whole-collection delete thousands of times over.
Both passes run too. Returning after a failed normal pass left the
collection's EC shards in place with nothing left to retry them.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* master: delete the EC shards behind /col/delete too
The HTTP handler carried its own copy of the volume-server walk and only
ever ran the normal pass, so a collection deleted through it kept its EC
shards. It shares the gRPC path now, which also gets it the bounded RPCs
and the one-per-server fan-out.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* filer: bound the collection delete a bucket delete leaves behind
Deleting a bucket entry deletes its collection afterwards, deliberately
detached from the request so a client that hangs up cannot strand the
bucket's volumes. Detached meant unbounded, though: with the master down
or mid-election the wait for a leader has nothing to end it, so the
handler parks, and the client retrying behind it parks another.
It keeps outliving the request and now carries a deadline of its own. The
budget bounds the wait, not the work: the master keeps deleting on its own
fan-out once asked, so giving up costs the confirmation.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3api: bound the collection RPCs a bucket creation and deletion issue
Neither carried a deadline, so a transient failure anywhere down the chain
held the S3 request open until the client gave up on it. Both budgets are
taken outside the filer failover walk, so one budget covers the whole walk
rather than granting each filer a fresh one.
The walk itself stops when that budget is spent, and stops without blaming
anyone: the caller's own expiry is not evidence against the filer that was
answering, and the next filer has no time left to answer in either.
Recorded as a filer failure, a slow master upstream would flag every filer
in the walk, and the three failures that open the circuit take unrelated
object reads down with them.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* s3api: a failed collection listing no longer fails a bucket creation
PutBucket lists collections to notice a leftover one it is about to reuse.
The result feeds a warning and nothing else -- s3a.exists is what decides
whether the bucket already exists -- yet a transient failure of that
listing returned 500 and refused the creation. It is advisory now, so a
failure is logged and the creation continues, exactly as it does when the
listing returns false.
Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
* Added context for the MasterClient's methods to avoid endless loops
* Returned WithClient function. Added WithClientCustomGetMaster function
* Hid unused ctx arguments
* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions
* Changed the context termination check in the tryConnectToMaster function
* Added a child context to the tryConnectToMaster function
* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
Junker der ProvinzandGitHub
chrislu
Riccardo BertossaandGitHub
vadimartynovandGitHub
Konstantin LebedevandGitHub
ningfd
qieqieplus
Lei Liu
j.laycock