This implementation introduces **public buckets**, which are accessible without signature-based authentication.
There are two ways to grant public access to a bucket:
* **Bucket ACLs**
* **Bucket Policies**
Only `Get` and `List` operations are permitted on public buckets. All **write operations** require authentication, regardless of whether public access is granted through an ACL or a policy.
The implementation includes an `AuthorizePublicBucketAccess` middleware, which checks if public access has been granted to the bucket. If so, authentication middlewares are skipped. For unauthenticated requests, appropriate errors are returned based on the specific S3 action.
---
**1. Bucket-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
}
]
}
```
**2. Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
**3. Both Bucket and Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
---
```sh
aws s3api create-bucket --bucket test --object-ownership BucketOwnerPreferred
aws s3api put-bucket-acl --bucket test --acl public-read
```
Closes#803
Implements host-style bucket addressing in the gateway. This feature can be enabled by running the gateway with the `--virtual-domain` flag and specifying a virtual domain name.
Example:
```bash
./versitygw -a user -s secret --virtual-domain localhost:7070 posix /tmp/vgw
```
The implementation follows this approach: it introduces a middleware (`HostStyleParser`) that parses the bucket name from the `Host` header and appends it to the URL path. This effectively transforms the request into a path-style bucket addressing format, which the gateway already supports. With this design, the gateway can handle both path-style and host-style requests when running in host-style mode.
For local testing, one can either set up a local DNS server to wildcard-match all subdomains of a specified domain and resolve them to the local IP address, or manually add entries to `/etc/hosts` to resolve bucket-prefixed hosts to the server IP (e.g., `127.0.0.1`).
`context` gets cancelled early before reading the full body in the `GetObject_with_range` integration test.
This change defers the context cancelation to make sure the full body is ready and the context isn't canceled in the middle of the request body read.
Fixes#1288
If the checksum algorithm/type is not specified during multipart upload initialization, it is considered `null`, and the `ListParts` result should also set it to `null`.
Fixes#1271
In the `Versioning_GetObject_success` integration test the contexts are canceled before reading the full request body after `GetObject`.
Changes the behaviour to defer the context cancelation, to be sure it's canceled after the full request body is read.
Fixes#1258Fixes#1257Closes#1244
Adds range queries support for `HeadObject`.
Fixes the range parsing logic for `GetObject`, which is used for `HeadObject` as well. Both actions follow the same rules for range parsing.
Fixes the error message returned by `GetObject`.
Fixes#1215Fixes#1216
`PutObject`, `CopyObject` and `CreateMultipartUpload` accept tag string as an http request header which should be url-encoded. The tag string should be a valid url-encoded string and each key/value pair should be valid, otherwise they should fail with `APIError`.
If the provided tag set contains duplicate `keys` the calls should fail with the same `InvalidURLEncodedTagging` error.
Not all url-encoded characters are supported by `S3`. The tagging string should contain only `letters`, `digits` and the following special chars:
- `-`
- `.`
- `/`
- `_`
- `+`
- ` `(space)
And their url-encoded versions: e.g. `%2F`(/), `%2E`(.) ... .
If the provided tagging string contains invalid `key`/`value`, the calls should fail with the following errors respectively:
`invalid key` - `(InvalidTag) The TagKey you have provided is invalid`
`invalid value` - `(InvalidTag) The TagValue you have provided is invalid`
Fixes#1214Fixes#1231Fixes#1232
Implements `utils.ParseTagging` which is a generic implementation of parsing tags for both `PutObjectTagging` and `PutBucketTagging`.
- The actions now return `MalformedXML` if the provided request body is invalid.
- Adds validation to return `InvalidTag` if duplicate keys are present in tagging.
- For invalid tag keys, it creates a new error: `ErrInvalidTagKey`.
Fixes#1204Fixes#1205
Tag count in `PutBucketTagging` and `PutObjectTagging` is limited.
`PutBucketTagging`: 50
`PutObjectTagging`: 10
Adds the changes to return errors respectively
Fixes#816
`ListObjects(V2)` used to return truncated response, if delimiter is provided and the result is limited by max-keys and the number of common prefixes is the same as `max-keys`.
e.g
PUT -> `foo/bar`
PUT -> `foo/quxx`
LIST: `max-keys=1;delim=/` -> foo/
`ListObjects(V2)` should return `foo/` as common prefix and `truncated` should be `false`.
The PR makes this fix to return `non-truncated` response for the above described case.
Fixes#1171
As signature v2 is depracated the gateway doesn't support it.
AWS S3 supports signature version 2 in some regions. For some regions the request fails with error:
```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
```
The PR makes this change to return unsupported authorization mechanism for `sigV2` requests.
Fixes#1186Fixes#1188Fixes#1189
If multiple checksum headers are provided, no matter if they are empty or not, the gateway should return `(InvalidRequest): Expecting a single x-amz-checksum- header. Multiple checksum Types are not allowed.`
An empty checksum header is considered as invalid, because it's not valid crc32, crc32c ...
Fixes#1182
S3 calculates the `CRC64NVME` checksum of an object on object upload(`PutObject`), when no checksum algorithm or precalculated checksum header is provided.
Makes the `CRC64NVME` checksum as default for `PutObject`, when no checksum is provided.
Fixes#1181
`DeleteObjects` should remove non-empty directory objects, which has been uploaded as a separate object.
e.g
Upload -> `foo/bar`
Upload -> `foo/`
Delete -> `foo/`
The last action call should succeed.
The PR introduces changes which removes `ETag` from the directory object attempting to `delete`, which has been uploaded as a separate object.
Closes#819
ListObjects returns object owner data in each object entity in the result, while ListObjectsV2 has fetch-owner query param, which indicates if the objects owner data should be fetched.
Adds these changes in the gateway to add `Owner` data in `ListObjects` and `ListObjectsV2` result. In aws the objects can be owned by different users in the same bucket. In the gateway all the objects are owned by the bucket owner.
Fixes#1021
`foo` and `foo/` object paths were considered as the same in `CopyObject` source and destination object paths in posix.
Implements the `joinPathWithTrailer` function, which calls `filepath.Join` and adds trailing `/` if it existed in the original path. This way the implementation puts separation
between directory and file objects with the same name.
If `Statement` field is missing in the json document in `PutBucketPolicy` body, the gateway returns `Missing required field Statement` error description alongside with `MalformedPolicy` error code.
Fixes#1000
`GetObjectAttributes` returned `InvalidRequest` instead of `InvalidArgument` with description `Invalid attribute name specified.`.
Fixes the logic in `ParseObjectAttributes` to ignore empty values for `X-Amz-Object-Attributes` headers to return `InvalidArgument` if all the specified object attributes are invalid.
Fixes#965
Changes the returned error description to `Policies must be valid JSON and the first byte must be '{'` for invalid bucket policy json documents, which doesn't start with `{`.
The gateway returns `This policy contains invalid Json` error description, if the document starts with `{`, but still isn't valid json.
Implements the `policyErr` string type which implements the `error` interface, to handle the policy json document validation errors, by avoiding staticchecker warnings.
Removes the `HeadObject_with_contenttype` and `GetObject_with_meta` integration test as the logic is covered in `HeadObject_success` and `GetObject_success` accordingly.
Fixes#1141Fixes#1142
Changes the error type to `InvalidArgument` for `x-amz-object-lock-legal-hold` and `x-amz-object-lock-mode` headers invalid values.
Fixes#998Closes#1125Closes#1126Closes#1127
Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.
Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
Closes#1128
Adds `Content-Disposition`, `Content-Language`, `Cache-Control` and `Expires` object meta properties support in posix and azure backends.
Changes the `PutObject` and `CreateMultipartUpload` actions backend input type to custom `s3response` types to be able to store `Expires` as any string.
Fixes#1004Fixes#1122Fixes#1120
Separates `GetObject` and `UploadPartCopy` range parsing/validation.
`GetObject` returns a successful response if acceptRange is invalid.
Adjusts the range upper limit, if it exceeds the actual objects size for `GetObject`.
Corrects the `ContentRange` in the `GetObject` response.
Fixes the `UploadPartCopy` action copy source range parsing/validation.
`UploadPartCopy` returns `InvalidArgument` if the copy source range is not valid.
