mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-28 20:06:14 +00:00
Chris LuandGitHub
44115c1051
* filer: store TUS sub-chunks through the regular chunk writer A TUS sub-chunk was written with one assigned file id, retried up to three times against that same id, and abandoned on failure: an attempt that had landed on some replicas left a needle no session record and no entry ever references, unreclaimable by vacuum. dataToChunkWithSSE, which the regular write path uses per chunk, assigns a fresh file id per attempt and hands back the file ids of failed attempts, which are now freed the way the regular write path frees them. * filer: retry a chunk write on a fresh volume when the server 5xxs The filer's chunk writer assigns a fresh file id per attempt but only retried transient network errors, so a volume filling up and turning read-only mid-write failed the whole request even though the very next assignment would have landed elsewhere. Every other write client already routes this through ShouldReassignUpload; the filer's own write path now does the same, for regular uploads and TUS sub-chunks alike. * filer: export the chunk deletion queue The filer test harness in weed/server builds filer.Filer as a struct literal, so any code path reaching DeleteChunks dereferenced a nil queue. Exported like the neighboring DeletionRetryQueue so the harness can arm it. * filer: complete a TUS upload whose chunk records overlap A PATCH retried while its predecessor was still storing a sub-chunk - a proxy timeout with an immediate retry is enough - records the same range twice. HEAD computes Upload-Offset as the covered watermark and reported the upload fully received, but completion demanded exactly adjacent records and failed every attempt: the client concluded success from offset == length, no entry was created, and the session eventually expired, turning the entire upload into deleted needles for the vacuum to chew through. Completion now validates gapless coverage with the same watermark HEAD uses. A record extending coverage joins the entry - the read path resolves partial overlaps by ModifiedTsNs, and the raced copies carry identical bytes - while a fully covered duplicate is freed once the entry lands. * filer: allow one mutating TUS request per session at a time Nothing stopped two PATCHes from writing the same range concurrently: both loaded the same offset, both passed the conflict check, and both recorded their sub-chunks. A client whose request timed out in a proxy retries immediately while the server side is still storing the buffered sub-chunk, which is exactly that race. A session now accepts one PATCH or DELETE at a time, the way tusd locks uploads; a concurrent one is refused with 423 Locked, which TUS clients retry, and HEAD keeps answering so progress polling is unaffected. The chunk state is loaded under the claim, so a retried PATCH sees every record its predecessor left and conflicts cleanly instead of duplicating data. * test: cover a TUS PATCH raced by its own retry Stalls a PATCH mid-body over a raw connection, retries the same range while it is in flight, and expects the retry refused with 423 Locked; the upload then resumes from the reported offset and the final content must be intact. * filer: never free a TUS duplicate the entry still references Coverage is computed from ranges, so a record fully covered by another is treated as a duplicate no matter which needle it names. A malformed record naming a file id the entry keeps would have had that needle freed right after the entry landed - the corruption this change set exists to stop. The duplicates are now freed in one batch, skipping any file id the entry references; their records go with the session directory. * test: bound the raw TUS connection reads http.ReadResponse on the stalled PATCH's connection blocked until the whole go test timeout if the filer never answered. * filer: free the needles of chunk write attempts a retry replaced A volume server stores the needle locally and only then fans out to the replicas, so a replication failure 5xxs with the data already written. Each attempt assigns its own file id, so once a later attempt lands elsewhere nothing references the earlier ones: the caller only sees the chunk that succeeded, and the failed ids were dropped. They are now freed the way the caller frees them when the whole write fails. Retrying on a 5xx makes this reachable on every read-only or full volume, which is exactly the condition that filled the reporter's volumes.