feat: support sha512, md5, xxhash3, xxhash64, xxhash128 data integrity checksums

Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.

These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`

Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.

References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
This commit is contained in:
niksis02
2026-05-04 08:50:39 -07:00
committed by Ben McClelland
parent fbe2a4ba10
commit d2fa265fb8
39 changed files with 1811 additions and 1025 deletions
+11 -1
View File
@@ -79,6 +79,11 @@ const (
checksumTypeSha1 checksumType = "x-amz-checksum-sha1"
checksumTypeSha256 checksumType = "x-amz-checksum-sha256"
checksumTypeCrc64nvme checksumType = "x-amz-checksum-crc64nvme"
checksumTypeSha512 checksumType = "x-amz-checksum-sha512"
checksumTypeMd5 checksumType = "x-amz-checksum-md5"
checksumTypeXxhash64 checksumType = "x-amz-checksum-xxhash64"
checksumTypeXxhash3 checksumType = "x-amz-checksum-xxhash3"
checksumTypeXxhash128 checksumType = "x-amz-checksum-xxhash128"
)
func (c checksumType) isValid() bool {
@@ -86,7 +91,12 @@ func (c checksumType) isValid() bool {
c == checksumTypeCrc32c ||
c == checksumTypeSha1 ||
c == checksumTypeSha256 ||
c == checksumTypeCrc64nvme
c == checksumTypeCrc64nvme ||
c == checksumTypeSha512 ||
c == checksumTypeMd5 ||
c == checksumTypeXxhash64 ||
c == checksumTypeXxhash3 ||
c == checksumTypeXxhash128
}
// Extracts and validates the checksum type from the 'X-Amz-Trailer' header