Fixes#1342
This PR includes two main changes:
1. It fixes the case where `x-amz-checksum-x` (precalculated checksum headers) are not provided for `UploadPart`, and the checksum type for the multipart upload is `FULL_OBJECT`. In this scenario, the server no longer returns an error.
2. When no `x-amz-checksum-x` is provided for `UploadPart`, and `x-amz-sdk-checksum-algorithm` is also missing, the gateway now calculates the part checksum based on the multipart upload's checksum algorithm and stores it accordingly.
Additionally, the PR adds integration tests for:
* The two cases above
* The case where only `x-amz-sdk-checksum-algorithm` is provided
Fixes#1339
`x-amz-checksum-type` and `x-amz-checksum-algorithm` request headers should be case insensitive in `CreateMultipartUpload`.
The changes include parsing the header values to upper case before validating and passing to back-end. `x-amz-checksum-type` response header was added in`CreateMultipartUpload`, which was missing before.
Fixes#1352
Adds a validation check step in `SigV4` authentication for `x-amz-content-sh256` to check it to be either a valid sha256 hash or a special payload type(UNSIGNED-PAYLOAD, STREAMING-UNSIGNED-PAYLOAD-TRAILER...).
Fixes#1385
When accessing a specific object version, the user must have the `s3:GetObjectVersion` permission in the bucket policy. The `s3:GetObject` permission alone is not sufficient for a regular user to query object versions using `HeadObject`.
This PR fixes the issue and adds integration tests for both `HeadObject` and `GetObject`. It also includes cleanup in the integration tests by refactoring the creation of user S3 clients, and moves some test user data to the package level to avoid repetition across tests.
Fixes#1388Fixes#1389Fixes#1390Fixes#1401
Adds the `x-amz-copy-source` header validation for `CopyObject` and `UploadPartCopy` in front-end.
The error:
```
ErrInvalidCopySource: {
Code: "InvalidArgument",
Description: "Copy Source must mention the source bucket and key: sourcebucket/sourcekey.",
HTTPStatusCode: http.StatusBadRequest,
},
```
is now deprecated.
The conditional read/write headers validation in `CopyObject` should come with #821 and #822.
Fixes#1398
The `x-amz-mp-object-size` request header can have two erroneous states: an invalid value or a negative integer. AWS returns different error descriptions for each case. This PR fixes the error description for the invalid header value case.
The invalid case can't be integration tested as SDK expects `int64` as the header value.
Fixes#1036
Fixes the issue when calling a non-existing root endpoint(POST /) the gateway returns `NoSuchBucket`. Now it returns the correct `MethodNotAllowed` error.
Implements a middleware that validates incoming bucket and object names before authentication. This helps prevent malicious attacks that attempt to access restricted or unreachable data in `POSIX`.
Adds test cases to cover such attack scenarios, including false negatives where encoded paths are used to try accessing resources outside the intended bucket.
Removes bucket validation from all other layers—including `controllers` and both `POSIX` and `ScoutFS` backends — by moving the logic entirely into the middleware layer.
This implementation introduces **public buckets**, which are accessible without signature-based authentication.
There are two ways to grant public access to a bucket:
* **Bucket ACLs**
* **Bucket Policies**
Only `Get` and `List` operations are permitted on public buckets. All **write operations** require authentication, regardless of whether public access is granted through an ACL or a policy.
The implementation includes an `AuthorizePublicBucketAccess` middleware, which checks if public access has been granted to the bucket. If so, authentication middlewares are skipped. For unauthenticated requests, appropriate errors are returned based on the specific S3 action.
---
**1. Bucket-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
}
]
}
```
**2. Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
**3. Both Bucket and Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
---
```sh
aws s3api create-bucket --bucket test --object-ownership BucketOwnerPreferred
aws s3api put-bucket-acl --bucket test --acl public-read
```
Closes#803
Implements host-style bucket addressing in the gateway. This feature can be enabled by running the gateway with the `--virtual-domain` flag and specifying a virtual domain name.
Example:
```bash
./versitygw -a user -s secret --virtual-domain localhost:7070 posix /tmp/vgw
```
The implementation follows this approach: it introduces a middleware (`HostStyleParser`) that parses the bucket name from the `Host` header and appends it to the URL path. This effectively transforms the request into a path-style bucket addressing format, which the gateway already supports. With this design, the gateway can handle both path-style and host-style requests when running in host-style mode.
For local testing, one can either set up a local DNS server to wildcard-match all subdomains of a specified domain and resolve them to the local IP address, or manually add entries to `/etc/hosts` to resolve bucket-prefixed hosts to the server IP (e.g., `127.0.0.1`).
`context` gets cancelled early before reading the full body in the `GetObject_with_range` integration test.
This change defers the context cancelation to make sure the full body is ready and the context isn't canceled in the middle of the request body read.
Fixes#1288
If the checksum algorithm/type is not specified during multipart upload initialization, it is considered `null`, and the `ListParts` result should also set it to `null`.
Fixes#1271
In the `Versioning_GetObject_success` integration test the contexts are canceled before reading the full request body after `GetObject`.
Changes the behaviour to defer the context cancelation, to be sure it's canceled after the full request body is read.
Fixes#1258Fixes#1257Closes#1244
Adds range queries support for `HeadObject`.
Fixes the range parsing logic for `GetObject`, which is used for `HeadObject` as well. Both actions follow the same rules for range parsing.
Fixes the error message returned by `GetObject`.
Fixes#1215Fixes#1216
`PutObject`, `CopyObject` and `CreateMultipartUpload` accept tag string as an http request header which should be url-encoded. The tag string should be a valid url-encoded string and each key/value pair should be valid, otherwise they should fail with `APIError`.
If the provided tag set contains duplicate `keys` the calls should fail with the same `InvalidURLEncodedTagging` error.
Not all url-encoded characters are supported by `S3`. The tagging string should contain only `letters`, `digits` and the following special chars:
- `-`
- `.`
- `/`
- `_`
- `+`
- ` `(space)
And their url-encoded versions: e.g. `%2F`(/), `%2E`(.) ... .
If the provided tagging string contains invalid `key`/`value`, the calls should fail with the following errors respectively:
`invalid key` - `(InvalidTag) The TagKey you have provided is invalid`
`invalid value` - `(InvalidTag) The TagValue you have provided is invalid`
Fixes#1214Fixes#1231Fixes#1232
Implements `utils.ParseTagging` which is a generic implementation of parsing tags for both `PutObjectTagging` and `PutBucketTagging`.
- The actions now return `MalformedXML` if the provided request body is invalid.
- Adds validation to return `InvalidTag` if duplicate keys are present in tagging.
- For invalid tag keys, it creates a new error: `ErrInvalidTagKey`.
Fixes#1204Fixes#1205
Tag count in `PutBucketTagging` and `PutObjectTagging` is limited.
`PutBucketTagging`: 50
`PutObjectTagging`: 10
Adds the changes to return errors respectively
Fixes#816
`ListObjects(V2)` used to return truncated response, if delimiter is provided and the result is limited by max-keys and the number of common prefixes is the same as `max-keys`.
e.g
PUT -> `foo/bar`
PUT -> `foo/quxx`
LIST: `max-keys=1;delim=/` -> foo/
`ListObjects(V2)` should return `foo/` as common prefix and `truncated` should be `false`.
The PR makes this fix to return `non-truncated` response for the above described case.
Fixes#1171
As signature v2 is depracated the gateway doesn't support it.
AWS S3 supports signature version 2 in some regions. For some regions the request fails with error:
```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
```
The PR makes this change to return unsupported authorization mechanism for `sigV2` requests.
Fixes#1186Fixes#1188Fixes#1189
If multiple checksum headers are provided, no matter if they are empty or not, the gateway should return `(InvalidRequest): Expecting a single x-amz-checksum- header. Multiple checksum Types are not allowed.`
An empty checksum header is considered as invalid, because it's not valid crc32, crc32c ...
Fixes#1182
S3 calculates the `CRC64NVME` checksum of an object on object upload(`PutObject`), when no checksum algorithm or precalculated checksum header is provided.
Makes the `CRC64NVME` checksum as default for `PutObject`, when no checksum is provided.
Fixes#1181
`DeleteObjects` should remove non-empty directory objects, which has been uploaded as a separate object.
e.g
Upload -> `foo/bar`
Upload -> `foo/`
Delete -> `foo/`
The last action call should succeed.
The PR introduces changes which removes `ETag` from the directory object attempting to `delete`, which has been uploaded as a separate object.
Closes#819
ListObjects returns object owner data in each object entity in the result, while ListObjectsV2 has fetch-owner query param, which indicates if the objects owner data should be fetched.
Adds these changes in the gateway to add `Owner` data in `ListObjects` and `ListObjectsV2` result. In aws the objects can be owned by different users in the same bucket. In the gateway all the objects are owned by the bucket owner.
Fixes#1021
`foo` and `foo/` object paths were considered as the same in `CopyObject` source and destination object paths in posix.
Implements the `joinPathWithTrailer` function, which calls `filepath.Join` and adds trailing `/` if it existed in the original path. This way the implementation puts separation
between directory and file objects with the same name.
If `Statement` field is missing in the json document in `PutBucketPolicy` body, the gateway returns `Missing required field Statement` error description alongside with `MalformedPolicy` error code.
Fixes#1000
`GetObjectAttributes` returned `InvalidRequest` instead of `InvalidArgument` with description `Invalid attribute name specified.`.
Fixes the logic in `ParseObjectAttributes` to ignore empty values for `X-Amz-Object-Attributes` headers to return `InvalidArgument` if all the specified object attributes are invalid.
Fixes#965
Changes the returned error description to `Policies must be valid JSON and the first byte must be '{'` for invalid bucket policy json documents, which doesn't start with `{`.
The gateway returns `This policy contains invalid Json` error description, if the document starts with `{`, but still isn't valid json.
Implements the `policyErr` string type which implements the `error` interface, to handle the policy json document validation errors, by avoiding staticchecker warnings.
Removes the `HeadObject_with_contenttype` and `GetObject_with_meta` integration test as the logic is covered in `HeadObject_success` and `GetObject_success` accordingly.
Fixes#1141Fixes#1142
Changes the error type to `InvalidArgument` for `x-amz-object-lock-legal-hold` and `x-amz-object-lock-mode` headers invalid values.
Fixes#998Closes#1125Closes#1126Closes#1127
Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.
Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
Closes#1128
Adds `Content-Disposition`, `Content-Language`, `Cache-Control` and `Expires` object meta properties support in posix and azure backends.
Changes the `PutObject` and `CreateMultipartUpload` actions backend input type to custom `s3response` types to be able to store `Expires` as any string.
Fixes#1004Fixes#1122Fixes#1120
Separates `GetObject` and `UploadPartCopy` range parsing/validation.
`GetObject` returns a successful response if acceptRange is invalid.
Adjusts the range upper limit, if it exceeds the actual objects size for `GetObject`.
Corrects the `ContentRange` in the `GetObject` response.
Fixes the `UploadPartCopy` action copy source range parsing/validation.
`UploadPartCopy` returns `InvalidArgument` if the copy source range is not valid.
