mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-30 04:37:07 +00:00
Chris LuandGitHub
ba5b14b457
* master: bound each volume server DeleteCollection, and finish the fan-out A collection delete fanned out to every volume server holding it with context.Background(), so a server that accepted the connection and then went quiet held the whole delete open with nothing to end it. Each RPC is bounded now, on the same budget allocateVolumeTimeout gives the other master-to-volume-server admin RPC. The volume server runs the delete to completion regardless of the request context, so giving up costs the confirmation and not the deletion. The walk itself is the caller's, not a per-server one: - It outlives the caller. A cancelled request must not abandon a destructive fan-out part-done, with volumes left behind and no request still running to come back for them. - It no longer stops at the first server that refuses, which left the collection on every server after it in the list. The first failure is still what is reported, and the collection stays in the topology so a later delete comes back for the rest. - It sends one RPC per server rather than one per replica. ListVolumeServers reports a node once for every replica it holds, while DeleteCollection removes the whole collection from the server it reaches, so a collection with thousands of volumes repeated the same whole-collection delete thousands of times over. Both passes run too. Returning after a failed normal pass left the collection's EC shards in place with nothing left to retry them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * master: delete the EC shards behind /col/delete too The HTTP handler carried its own copy of the volume-server walk and only ever ran the normal pass, so a collection deleted through it kept its EC shards. It shares the gRPC path now, which also gets it the bounded RPCs and the one-per-server fan-out. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * filer: bound the collection delete a bucket delete leaves behind Deleting a bucket entry deletes its collection afterwards, deliberately detached from the request so a client that hangs up cannot strand the bucket's volumes. Detached meant unbounded, though: with the master down or mid-election the wait for a leader has nothing to end it, so the handler parks, and the client retrying behind it parks another. It keeps outliving the request and now carries a deadline of its own. The budget bounds the wait, not the work: the master keeps deleting on its own fan-out once asked, so giving up costs the confirmation. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: bound the collection RPCs a bucket creation and deletion issue Neither carried a deadline, so a transient failure anywhere down the chain held the S3 request open until the client gave up on it. Both budgets are taken outside the filer failover walk, so one budget covers the whole walk rather than granting each filer a fresh one. The walk itself stops when that budget is spent, and stops without blaming anyone: the caller's own expiry is not evidence against the filer that was answering, and the next filer has no time left to answer in either. Recorded as a filer failure, a slow master upstream would flag every filer in the walk, and the three failures that open the circuit take unrelated object reads down with them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: a failed collection listing no longer fails a bucket creation PutBucket lists collections to notice a leftover one it is about to reuse. The result feeds a warning and nothing else -- s3a.exists is what decides whether the bucket already exists -- yet a transient failure of that listing returned 500 and refused the creation. It is advisory now, so a failure is logged and the creation continues, exactly as it does when the listing returns false. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP