Integrate the new S3 checksum types in the gateway, including `SHA512`, `MD5`, `XXHASH64`, `XXHASH3`, and `XXHASH128`. This adds checksum calculation, validation, schema handling, and test coverage for the expanded checksum support.
These external packages have been used:
- `github.com/zeebo/xxh3` for `XXHASH3` and `XXHASH128`
- `github.com/cespare/xxhash/v2` for `XXHASH64`
Adjust integration tests because `aws-sdk-go-v2/service/s3` does not support automatic checksum calculation for the new checksum algorithms and returns an SDK-level error when only the checksum algorithm is provided. Only precalculated checksum values are acceptable for these checksum types.
References:
- `https://github.com/aws/aws-sdk-go-v2/issues/3404`
- `https://github.com/aws/aws-sdk-go-v2/issues/3403`
Fixes#2052Fixes#2056Fixes#2057
Previously, GetObject and HeadObject used the request's `Range` header to determine the response status code, which caused incorrect 206 responses for invalid Range header values.
The status is now driven by whether res.ContentRange is set in the response, rather than by the presence of a range in the request. Backends (posix and azure) now set Content-Range for PartNumber=1 on non-multipart objects, skipping zero-size objects where no range applies.
HeadObject was also fixed to return 206 when Content-Range is present, and to only return checksums when the full object is requested.
Closes#1064
Use the multipart ETag as the in-progress directory suffix instead of the static `.inprogress` marker so that concurrent CompleteMultipartUpload calls for the same upload ID are all treated as successful (idempotent) rather than racing, where only one succeeded and the rest returned NoSuchUpload.
After finalizing the multipart upload, store an `mp-metadata` xattr on the assembled object that records the upload ID and cumulative byte offsets for each part. GetObject and HeadObject now use this metadata to serve individual part ranges via the `partNumber` query parameter, returning a successful response instead of returning NotImplemented.
Add two new S3 error codes:
- `ErrInvalidPartNumberRange` (416 RequestedRangeNotSatisfiable) — returned
when the requested part number exceeds the number of parts in the upload.
- `ErrRangeAndPartNumber` (400 BadRequest) — returned when both a Range header
and a partNumber query parameter are specified on the same request.
Closes#1813
We use a specific `versionId` format(`ulid` package) to generate versionIds in posix, which is not compatible to S3. The versionId validation was performed in frontend which is a potential source of failure for s3 proxy configured on an s3 service which doesn't use ulid for versionId generation(e.g. aws S3). These changes move the specific `ulid` versionId validation to posix to not force any specific versionId format in the gateway.
Fixes#1809Fixes#1806Fixes#1804Fixes#1794
This PR focuses on correcting so-called "list-limiter" parsing and validation. The affected limiters include: `max-keys`, `max-uploads`, `max-parts`, `max-buckets`, `max-uploads` and `part-number-marker`. When a limiter value is outside the integer range, a specific `InvalidArgument` error is now returned. If the value is a valid integer but negative, a different `InvalidArgument` error is produced.
`max-buckets` has its own validation rules: completely invalid values and values outside the allowed range (`1 <= input <= 10000`) return distinct errors. For `ListObjectVersions`, negative `max-keys` values follow S3’s special-case behavior and return a different `InvalidArgument` error message.
Additionally, `GetObjectAttributes` now follows S3 semantics for `x-amz-max-parts`: S3 ignores invalid values, so the gateway now matches that behavior.
Fixes#1698
`PutObjectTagging`, `GetObjectTagging` and `DeleteObjectTagging` return the `x-amz-version-id` in the response headers. The PR adds this header in the responses.
Fixes#1630
S3 returns `InvalidArgument: Invalid version id specified` for invalid version IDs in object-level actions that accept `versionId` as a query parameter. The `versionId` in S3 follows a specific structure, and if the input string doesn’t match this structure, the error is returned. In the gateway, the `versionId` is generated using the `ulid` package, which also has a defined structure. This PR adds validation for object-level operations that work with object versions by using the ULID parser.
These actions include: `HeadObject`, `GetObject`, `PutObjectTagging`, `GetObjectTagging`, `DeleteObjectTagging`, `PutObjectLegalHold`, `GetObjectLegalHold`, `PutObjectRetention`, `GetObjectRetention`, `DeleteObject`, `CopyObject`, `UploadPartCopy`, and `GetObjectAttributes`.
Closes#1343
Object version tagging support was previously missing in the gateway. The support is added with this PR. If versioning is not enabled at the gateway level and a user attempts to put, get, or delete object version tags, the gateway returns an `InvalidArgument`(Invalid versionId)
Fixes#1520
Removes the incorrect logic for HeadObject returning successful response, when querying an incomplete multipart upload.
Implements the logic to return `NotImplemented` error if `GetObject`/`HeadObject` is attempted with `partNumber` in azure and posix backends. The front-end part is preserved to be used in s3 proxy backend.
