Files
seaweedfs/weed/s3api
Chris LuandGitHub 542495f1f1 S3: fold large chunk lists into manifest chunks on the direct write path (#10383)
s3: fold large chunk lists into manifest chunks on the direct write path

The S3 gateway uploads chunks itself and hands the filer a fully prepared
entry. On the routed write path (ObjectTransaction) the filer stores that
entry as-is, so a large PutObject or CompleteMultipartUpload persisted its
whole flat chunk list - a 900GB object carries 120k chunk references in one
entry. Manifestize on the gateway before the entry is written, the same way
mount, WebDAV, and filer.copy prepare theirs.

Multipart part boundaries also record byte offsets now: the stored chunk
indexes stop matching the entry once the list is folded, and partNumber
reads plus GetObjectAttributes prefer the offsets. Legacy index-only
records still work, with bounds checks instead of a possible panic.

Copy paths resolve a manifested source into data chunks before their
per-chunk copy loops - copying a manifest chunk raw would store its blob
as object data still pointing at the source - and a large copied list is
folded again on the destination. Completion likewise resolves manifest
chunks a part entry may carry (the filer folds an oversized UploadPartCopy
range) before rebasing part offsets.
2026-07-21 00:08:05 -07:00
..
2026-01-28 14:34:07 -08:00
2026-02-20 18:40:47 -08:00

see https://blog.aqwari.net/xml-schema-go/

1. go get aqwari.net/xml/cmd/xsdgen
2. Add EncodingType element for ListBucketResult in AmazonS3.xsd
3. xsdgen -o s3api_xsd_generated.go -pkg s3api AmazonS3.xsd
4. Remove empty Grantee struct in s3api_xsd_generated.go
5. Remove xmlns: sed s'/http:\/\/s3.amazonaws.com\/doc\/2006-03-01\/\ //' s3api_xsd_generated.go