166 Commits

Author SHA1 Message Date
niksis02
f831578d51 fix: handles tag parsing error cases for PutBucketTagging and PutObjectTagging
Fixes #1214
Fixes #1231
Fixes #1232

Implements `utils.ParseTagging` which is a generic implementation of parsing tags for both `PutObjectTagging` and `PutBucketTagging`.

- The actions now return `MalformedXML` if the provided request body is invalid.
- Adds validation to return `InvalidTag` if duplicate keys are present in tagging.
- For invalid tag keys, it creates a new error: `ErrInvalidTagKey`.
2025-04-23 20:35:19 +04:00
niksis02
66b979ee86 fix: Sets limit to tag set count to 10 for PutObjectTagging and 50 for PutBucketTagging
Fixes #1204
Fixes #1205

Tag count in `PutBucketTagging` and `PutObjectTagging` is limited.
`PutBucketTagging`: 50
`PutObjectTagging`: 10

Adds the changes to return errors respectively
2025-04-11 21:07:08 +04:00
niksis02
aaee7afa98 fix: Changes the backend.Walk implementation to return non-truncated result if both delimiter and max-keys are provided
Fixes #816

`ListObjects(V2)` used to return truncated response, if delimiter is provided and the result is limited by max-keys and the number of common prefixes is the same as `max-keys`.
e.g
PUT -> `foo/bar`
PUT -> `foo/quxx`
LIST: `max-keys=1;delim=/` -> foo/

`ListObjects(V2)` should return `foo/` as common prefix and `truncated` should be `false`.

The PR makes this fix to return `non-truncated` response for the above described case.
2025-04-09 21:29:58 +04:00
Ben McClelland
eb72905401 Merge pull request #1200 from versity/sis/obj-lock-actions-invalid-body
fix: Changes the error type to MalformedXML for PutObjectRetention and PutObjectLegalHold empty or invalid body
2025-04-08 20:55:15 -07:00
Ben McClelland
4f70a16245 Merge pull request #1198 from versity/sis/tests-context-deadline
fix: Increases the context cancelation deadline in integration tests.
2025-04-08 10:19:53 -07:00
Ben McClelland
7866c326e3 Merge pull request #1190 from versity/sis/empty-checksum-headers
fix: Handles the error cases for empty checksum headers for PutObject and UploadPart
2025-04-08 10:19:34 -07:00
niksis02
20d00f7f6d fix: Changes the error type to MalformedXML for PutObjectRetention and PutObjectLegalHold empty or invalid body
Fixes #1185
Fixes #1191

`PutObjectLegalHold` and `PutObjectRetention` should return `MalformedXML` if the request body is empty or invalid.`
2025-04-08 19:01:00 +04:00
niksis02
fca1cf9fd6 fix: Increases the context cancelation deadline in integration tests.
Increases the context cancelation deadline to `30` seconds to avoid timeouts in relatively large tests.
2025-04-08 18:12:36 +04:00
niksis02
5560e5ebc4 fix: Returns ErrSignatureVersionNotSupported for sigV2 signed requests.
Fixes #1171

As signature v2 is depracated the gateway doesn't support it.
AWS S3 supports signature version 2 in some regions. For some regions the request fails with error:
```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
```

 The PR makes this change to return unsupported authorization mechanism for `sigV2` requests.
2025-04-08 01:05:59 +04:00
niksis02
ed44fe1969 fix: Handles the error cases for empty checksum headers for PutObject and UploadPart
Fixes #1186
Fixes #1188
Fixes #1189

If multiple checksum headers are provided, no matter if they are empty or not, the gateway should return `(InvalidRequest): Expecting a single x-amz-checksum- header. Multiple checksum Types are not allowed.`
An empty checksum header is considered as invalid, because it's not valid crc32, crc32c ...
2025-04-04 23:17:22 +04:00
niksis02
f627b2efec fix: Makes the crc64nvme checksum as default for PutObject, even if no checksum is provided
Fixes #1182

S3 calculates the `CRC64NVME` checksum of an object on object upload(`PutObject`), when no checksum algorithm or precalculated checksum header is provided.
Makes the `CRC64NVME` checksum as default for `PutObject`, when no checksum is provided.
2025-04-04 00:08:54 +04:00
Ben McClelland
90a67272a7 Merge pull request #1184 from versity/sis/deleteobject-non-empty-dir-obj
fix: Fixes non empty directory objects deletion
2025-04-02 13:09:48 -07:00
niksis02
8705905319 fix: Fixes non empty directory objects deletion
Fixes #1181

`DeleteObjects` should remove non-empty directory objects, which has been uploaded as a separate object.
e.g
Upload -> `foo/bar`
Upload -> `foo/`

Delete -> `foo/`

The last action call should succeed.

The PR introduces changes which removes `ETag` from the directory object attempting to `delete`, which has been uploaded as a separate object.
2025-04-02 22:31:05 +04:00
Ben McClelland
dadfc97d46 Merge pull request #1183 from versity/sis/listobjectsv2-owner
feat: Adds Owner data in ListObjects(V2) result
2025-04-02 10:52:14 -07:00
niksis02
cb97fb589b feat: Adds Ownder data in ListObjects(V2) result.
Closes #819

ListObjects returns object owner data in each object entity in the result, while ListObjectsV2 has fetch-owner query param, which indicates if the objects owner data should be fetched.
Adds these changes in the gateway to add `Owner` data in `ListObjects` and `ListObjectsV2` result. In aws the objects can be owned by different users in the same bucket. In the gateway all the objects are owned by the bucket owner.
2025-04-02 18:28:32 +04:00
niksis02
03c7c432ad fis: Fixes the trailing slash issue in CopyObject destination and source object paths in posix.
Fixes #1021

`foo` and `foo/` object paths were considered as the same in `CopyObject` source and destination object paths in posix.
Implements the `joinPathWithTrailer` function, which calls `filepath.Join` and adds trailing `/` if it existed in the original path. This way the implementation puts separation
 between directory and file objects with the same name.
2025-04-01 17:31:02 +04:00
niksis02
30f319bc92 fix: Hanldes the case when Statement is missing in PutBucketPolicy json document.
If `Statement` field is missing in the json document in `PutBucketPolicy` body, the gateway returns `Missing required field Statement` error description alongside with `MalformedPolicy` error code.
2025-03-31 18:23:33 +04:00
niksis02
832371afb1 fix: Fixes the case for GetObjectAttributes to return InvalidArgument if a single invalid object attribute is provided.
Fixes #1000

`GetObjectAttributes` returned `InvalidRequest` instead of `InvalidArgument` with description `Invalid attribute name specified.`.
Fixes the logic in `ParseObjectAttributes` to ignore empty values for `X-Amz-Object-Attributes` headers to return `InvalidArgument` if all the specified object attributes are invalid.
2025-03-28 07:27:35 +04:00
niksis02
d82da64942 fix: Fixes the bucket policy json document validation errors for invalid jsons.
Fixes #965

Changes the returned error description to `Policies must be valid JSON and the first byte must be '{'` for invalid bucket policy json documents, which doesn't start with `{`.
The gateway returns `This policy contains invalid Json` error description, if the document starts with `{`, but still isn't valid json.

Implements the `policyErr` string type which implements the `error` interface, to handle the policy json document validation errors, by avoiding staticchecker warnings.
2025-03-20 01:08:47 +04:00
Ben McClelland
4dad42e4e8 Merge pull request #1150 from versity/sis/unnecessary-int-tests
fix: Removes 2 unnecessary integration tests for HeadObject and GetOb…
2025-03-18 08:25:16 -07:00
niksis02
607c6d2308 fix: Removes 2 unnecessary integration tests for HeadObject and GetObject.
Removes the `HeadObject_with_contenttype` and `GetObject_with_meta` integration test as the logic is covered in `HeadObject_success` and `GetObject_success` accordingly.
2025-03-18 18:50:01 +04:00
niksis02
9e0f56f807 fix: Fixes the returned error type for object legal hold status and object lock mode in PutObject, CopyObject and CreateMultipartUpload.
Fixes #1141
Fixes #1142

Changes the error type to `InvalidArgument` for `x-amz-object-lock-legal-hold` and `x-amz-object-lock-mode` headers invalid values.
2025-03-18 13:58:49 +04:00
niksis02
cfb2d6d87d feat: Implements object meta properties for CopyObject in azure and posix backends.
Fixes #998
Closes #1125
Closes #1126
Closes #1127

Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.

Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
2025-03-17 09:37:05 -07:00
niksis02
65261a9753 feat: Adds the Content-Disposition, Content-Language, Cache-Control and Expires object meta properties support in the gateway.
Closes #1128

Adds `Content-Disposition`, `Content-Language`, `Cache-Control` and `Expires` object meta properties support in posix and azure backends.
Changes the `PutObject` and `CreateMultipartUpload` actions backend input type to custom `s3response` types to be able to store `Expires` as any string.
2025-03-12 16:01:56 +04:00
niksis02
96af2b6471 fix: Fixes GetObject and UploadPartCopy actions data range parsing.
Fixes #1004
Fixes #1122
Fixes #1120

Separates `GetObject` and `UploadPartCopy` range parsing/validation.

`GetObject` returns a successful response if acceptRange is invalid.
Adjusts the range upper limit, if it exceeds the actual objects size for `GetObject`.
Corrects the `ContentRange` in the `GetObject` response.

Fixes the `UploadPartCopy` action copy source range parsing/validation.
`UploadPartCopy` returns `InvalidArgument` if the copy source range is not valid.
2025-03-08 01:39:21 +04:00
niksis02
4b0dd64836 fix: removes ETag quotes for in GetObjectAttributes response for azure and posix backends 2025-03-06 00:28:55 +04:00
niksis02
d59ee87b10 fix: Fixes ListObjects direcotry objects listing issue. Fixes DeleteObject directory objects parents deletion 2025-03-04 21:29:08 +04:00
niksis02
92dee62298 fix: Allows checksum-type to be empty in CompleteMultipartUpload input parameters 2025-03-03 22:14:47 +04:00
niksis02
9a02c474b1 fix: Implements a generic wildcard match for bucket policy resources 2025-03-03 20:04:31 +04:00
niksis02
30f3fac4e1 fix: Prioritize explicit deny in bucket policy statements 2025-03-01 01:14:12 +04:00
niksis02
d13791f5ce fix: Adds quotes for object Etags 2025-02-28 00:43:51 +04:00
niksis02
fcafb57abc feat: Implements checksums for ListObjectVersions and CopyObject actions. 2025-02-27 00:24:38 +04:00
niksis02
1bb07fd78e fix: Changes HeadObject returned error type from MethodNotAllowed to NotFound if the latest object version is a delete marker. 2025-02-24 20:44:26 +04:00
niksis02
b53bbb025f fix: Adds crc64nvme checksum cases in the integration tests 2025-02-21 21:01:41 +04:00
Ben McClelland
e256a1aa0d Merge pull request #1079 from versity/fix/versioning-nested-object-deletion
fix: Adds removeParents in DeleteObject in case of deleting an object…
2025-02-20 18:01:20 -08:00
niksis02
a30b8c2320 fix: Adds removeParents in DeleteObject in case of deleting an object in nested directories 2025-02-21 00:20:49 +04:00
Ben McClelland
549289c581 Merge pull request #1078 from versity/fix/listparts-issues
ListParts refactoring
2025-02-20 08:12:36 -08:00
niksis02
e5811e4ce7 fix: Fixes the entity limiter validation for ListObjects(V2), ListParts, ListMultipartUploads, ListBuckets actions 2025-02-20 15:45:42 +04:00
niksis02
64a72a2dee feat: Adds 'x-amz-mp-object-size' request header support for CompleteMultipartUpload 2025-02-19 19:26:03 +04:00
Ben McClelland
ff0cf29d0a Merge pull request #1075 from versity/fix/complete-mp-parts-order
fix: Adds PartNumber validation for CompleteMultipartUploads parts. A…
2025-02-18 09:51:02 -08:00
niksis02
3cae3fced9 fix: Adds PartNumber validation for CompleteMultipartUploads parts. Adds a check to validate the parts order to be ascending. 2025-02-18 20:02:01 +04:00
niksis02
4517b292b9 fix: Changes UploadPart returned error from ErrInvalidPart to ErrInvalidPartNumber 2025-02-17 19:38:50 +04:00
niksis02
132d0ae631 feat: Adds the CRC64NVME checksum support in the gateway. Adds checksum-type support for the checksum implementation 2025-02-16 17:10:06 +04:00
niksis02
6956757557 feat: Integrates object integrity checksums(CRC32, CRC32C, SHA1, SHA256) into the gateway 2025-02-14 14:14:00 +04:00
niksis02
b5b418e4b3 fix: Adds the minimum allowed size(5 Mib) check for mp parts sizes in CompleteMultipartUpload 2025-02-03 21:49:15 +04:00
niksis02
f35cdfb20c feat: Adds a flag on the test command to run the integration tests in insecure mode: by skiping the tls verification 2025-02-03 09:13:01 -08:00
niksis02
c094086d83 fix: Fixes the response body streaming for GetObject, implementing a chunk streamer 2025-01-15 23:11:04 +04:00
niksis02
c37a22ffe1 fix: Fixes the AccessControlPolicy Grantee Type unmarshalling, Adds request body validation for the PutBucketAcl action 2025-01-13 23:44:46 +04:00
niksis02
7c5258e6e9 fix: Adds a check to ensure that the CompleteMultipartUpload parts are not empty. 2024-12-17 18:50:57 +04:00
niksis02
2a61489e4c fix: Uncommentes two WORM protection integration tests that were accidentally commented out. 2024-12-11 00:33:45 +04:00