Final piece for #6541. The buffered chunk-copy path holds two
chunk-sized buffers per copy in flight (download buffer + multipart-
encoded upload buffer). Under concurrent UploadPartCopy that put a
floor on RSS at concurrency × 2 × chunk_size — about 768 MiB for the
6-way / 64 MiB Harbor-style assemble repro, even after the previous
pool/retention fixes.
Replace the buffered path with an io.Pipe between the source GET and
the destination POST: ReadUrlAsStream pumps data into the pipe via a
multipart.Writer, the http.Client reads from the pipe end and POSTs
the body. In-flight per copy is now ~32 KiB (pipe hand-off + http
buffers), regardless of chunk size.
The streaming path is gated by canStreamCopyChunk: only used when no
in-transit transformation is needed (no per-chunk CipherKey, no SSE).
SSE-C / SSE-KMS / SSE-S3 paths still go through the buffered path,
which already handles re-encryption correctly.
Benchmarks (Apple M4, httptest source/dest, B/op = bytes per copy):
Buffered 1 MiB: 6.0 MB B/op, 443 MB/s
Streamed 1 MiB: 374 KB B/op, 727 MB/s
Buffered 8 MiB: 56 MB B/op, 559 MB/s
Streamed 8 MiB: 379 KB B/op, 1138 MB/s
Buffered 64 MiB: 455 MB B/op, 718 MB/s
Streamed 64 MiB: 304 KB B/op, 1387 MB/s
End-to-end repro (512 MiB src, 6 parallel UploadPartCopy):
pre-#9420 RSS round 2: 3134 MiB
+ #9420/#9421/#9422 : 2236 MiB
+ this PR : 1521 MiB
heap inuse_space : 350 MiB (was 1422 / 1187 MiB)
HeapSys (MemStats) : 1.74 GiB (was 2.49 GiB)
see https://blog.aqwari.net/xml-schema-go/
1. go get aqwari.net/xml/cmd/xsdgen
2. Add EncodingType element for ListBucketResult in AmazonS3.xsd
3. xsdgen -o s3api_xsd_generated.go -pkg s3api AmazonS3.xsd
4. Remove empty Grantee struct in s3api_xsd_generated.go
5. Remove xmlns: sed s'/http:\/\/s3.amazonaws.com\/doc\/2006-03-01\/\ //' s3api_xsd_generated.go
Chris Lu
6f2bf40d5a