mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-16 12:16:36 +00:00
ab79d1f680
* operation: re-assign chunk upload when replica volume is full When a volume reaches MaxPossibleVolumeSize, needle writes return 'Volume Size Exceeded' and the fan-out in uploadChunkToHolders fails. Previously the error propagated immediately, killing the entire chunked upload even though the master has other writable volumes. Fix: detect 'Volume Size' errors on the fan-out path, call AssignFunc again to get a fresh volume, and retry (up to 3 attempts). This avoids backup failures while a single replica volume is full and waiting for GC. Also add 'volume size exceeded' to the transient error messages so any retry path that checks IsTransientError recognises it. * util: fix transient error pattern for volume size errors The actual error message is 'Volume Size 34361499680 Exceeded 34359738368' where the numeric size separates 'Volume Size' from 'Exceeded'. The previous pattern 'volume size exceeded' would never match. Change to 'volume size' which correctly matches any capacity-full error. * operation: fix reassignment loop nits - Case-insensitive volume size match (strings.ToLower) - Propagate AssignFunc error so caller sees reassignment failure - Reset JWT fallback before each reassignment to avoid retaining stale auth * util: stop classifying a full volume as a transient error A volume at capacity does not become writable on the next attempt, so the entry only bought a retry loop's worth of sleeping before the same failure. It also reached four consumers that all retry the same target — the deletion queue, the replication sink, volume lookups, and Retry/MultiRetry — none of which reassign, and the widened "volume size" substring swallowed the replica-receive rejection from WriteNeedleBlob too. The chunked upload path recovers by asking for a different volume instead. * operation: reassign a chunk with the shared upload gate shouldReassignUpload already answers this question for the non-chunked path, keyed off the status the volume server returned rather than its message text. Reusing it covers a lost replica peer and an unreachable target as well as a full volume, and it stops a 4xx from being retried on a second volume that would reject it identically. Pulling the single attempt out into uploadChunk keeps the retry loop readable now that it wraps both the fan-out and the relay path. * test: cover chunk reassignment across volumes Pins the three outcomes the gate decides: a full volume moves the chunk to a fresh assignment, a 4xx stays put, and the loop gives up after chunkAssignAttempts volumes. * operation: roll back the fid a reassigned chunk abandons ReplicatedWrite commits the needle locally before it replicates, so a 5xx can leave a copy behind on a volume the chunk is about to walk away from. Nothing will ever reference that fid, and an unreferenced needle is not garbage vacuum can find — it is dead space until the volume is destroyed. uploadChunkToHolders rolls back only the holders that reported success, which misses the one whose write landed but whose response did not, and the relay path had no rollback at all. Delete from every holder of the abandoned assignment instead; deleting a needle that never landed is a no-op. * operation: stop the reassignment loop from multiplying work retriedUploadData already retries a chunk three times against the same URL, so wrapping it in three assignments made nine POSTs for one chunk. On the relay path that inner retry is redundant — the loop retries everything it would, and on a different volume — so cap it at one attempt per assignment and leave the budget where it was. The fan-out path keeps its inner retries: absorbing a blip on one holder beats cancelling the rest and re-uploading the whole chunk. Nothing bounded any of it by time. weed/s3api passes context.Background() so a chunk survives client disconnect, which also means no deadline cuts the loop short, and a chunk goroutine holds one of four buffer slots while it spins. Break out once another chunk has already failed the object. * operation: keep the reassignment gate's inputs deterministic uploadChunkToHolders reported whichever holder error won the channel race. That was cosmetic while the value was only logged; now it decides whether the chunk moves to another volume, so a 400 and a 500 arriving in either order made the retry behavior depend on scheduling. Prefer an error the caller can act on, and the same failure always retries the same way. A failed reassignment also overwrote the upload error that prompted it, which buried a full volume behind whatever the filer happened to say. Keep both in the chain. The tests grew a JWT per assignment, since the loop re-derives one and nothing covered it, and the bound is now spelled out rather than compared against the constant that defines it. * operation: roll back the last abandoned fid too The rollback ran only on the path that goes on to reassign, so the attempt that exhausts the budget — or stops because another chunk already failed the object, or because the error is not one a different volume fixes — left its fid behind. That is the case that matters most: no chunk names it, the caller gets no fid to clean up, and a 5xx can still mean the needle was committed. Roll back on every failed attempt instead, before deciding whether to retry. --------- Co-authored-by: timolow <timolow@users.noreply.github.com> Co-authored-by: timolow <tim@timolow.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>