perf(volume): stream-count the gzip size when no Content-MD5 is set (#9433)

Browse Source

ParseUpload runs util.DecompressData on every gzipped multipart upload
just to record OriginalDataSize. The decompress materializes the full
uncompressed slice via bytes.Buffer.ReadFrom inside util.GunzipStream;
for a 64 MiB chunk that's a ~128 MiB heap spike per call (geometric
grow). On 6-way concurrent UploadPartCopy the spike dominated the
remaining heap profile after #9420/#9421/#9422/#9424/#9425.

When no Content-MD5 verification is requested the uncompressed bytes
aren't needed — only the length is. Stream the gunzip through
io.Discard and count: the pooled gzip.Reader's working set replaces
the materialized slice.

Unlike the previous attempt in #9426 the size still comes from the
real bytes, not from a client-set header.

  TotalAlloc per call, 4 MiB uncompressed body:
    materialize (was, still runs when MD5 is set):  ~16.8 MiB
    stream-count (no MD5):                             ~28 KiB

Refs #6541, #9426 (reverted in #9432).

...

This commit is contained in:

Chris Lu

2026-05-11 10:56:11 -07:00

committed by GitHub

parent 6001d65206

commit defe047e1a

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

Diff Content Not Available