mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-19 05:36:58 +00:00
Chris LuandGitHub
c917110124
* fix(volume): pre-size ParseUpload buffer to request ContentLength The volume server's PostHandler reads the multipart upload body via bytes.Buffer.ReadFrom inside parseUpload. The buffer comes from a sync.Pool and may have cap=0 when the pool dropped the prior entry, which makes ReadFrom geometric-grow on each chunk: a 64 MiB upload allocates roughly 1+2+4+...+64 ≈ 128 MiB just to receive the body. Under concurrent uploads (every s3 chunk-copy lands here on the destination volume) this is one of the main contributors to the runaway-RSS pattern in #6541 — pprof shows ~458 MiB cum in parseUpload's bytes.Buffer.ReadFrom under Harbor-style assemble load. Grow the buffer once up front, bounded by the existing sizeLimit so a misreported Content-Length can't over-allocate. The receive then fills in place. Add a regression test that drives ParseUpload with a 16 MiB multipart body and bounds TotalAlloc at 1.5x the chunk size (pre-fix measures ~4x, so the bound trips deterministically). * fix(volume): guard ParseUpload pre-grow against int overflow on 32-bit Address PR review feedback: r.ContentLength is int64, and on 32-bit platforms int is 32 bits wide, so int(r.ContentLength) for a value above math.MaxInt32 wraps negative and bytes.Buffer.Grow panics with "bytes.Buffer.Grow: negative count". Skip the pre-grow optimization in that range; the existing geometric-grow path remains correct, just slightly more allocator pressure for that one call. 64-bit platforms (math.MaxInt == math.MaxInt64) are unaffected — the guard only kicks in for 32-bit builds with very large sizeLimit. * fix(volume): cap ParseUpload pre-grow at 4 MiB to bound DoS surface Address PR review: pre-growing the receive buffer to r.ContentLength trusts the header before any body bytes arrive. A bad header or slow / idle client could declare a large Content-Length up to sizeLimit (256 MiB by default for volume writes) and force per-request preallocation without sending data, turning many concurrent slow connections into avoidable memory pressure. Cap eager pre-grow at maxEagerPreGrow (4 MiB). Larger uploads still benefit from the higher starting cap and fall back to ReadFrom's grow path for the remainder. Per-request waste from a misreported Content-Length is now bounded at 4 MiB regardless of sizeLimit. Extract the policy as eagerPreGrow so the unit test can exercise the gates structurally — replaces the prior TotalAlloc bound (which became uninformative once savings were capped at 4 MiB).