Commit Graph

110 Commits

Author SHA1 Message Date
Ben McClelland
97b5424e07 feat: move metrics actions and service to tags 2024-05-29 15:42:04 -07:00
jonaustin09
dbfd9e5171 fix: Removed required request body check for PutBucketAcl action 2024-05-29 14:13:38 -07:00
Ben McClelland
983da28a7e feat: define action names in metrics module 2024-05-28 15:46:37 -07:00
jonaustin09
be6f9a86cd feat: Integrated metric manager into the gateway 2024-05-28 12:50:50 -07:00
jonaustin09
9e8458a09f feat: Added integration tests for bypass governance retention functionality 2024-05-28 15:17:25 -04:00
jonaustin09
fb27e2703e feat: Implemented to logic to bypass governance retention 2024-05-24 13:50:41 -04:00
jonaustin09
6fb102056d fix: Changed WORM protection implementation to prevent root/admin users to overwrite objects in governance mode or if legal hold is set up 2024-05-23 16:56:21 -04:00
jonaustin09
43f509d971 fix: Added missing properties support for CreateMultipartUpload action: ContentType, ObjectLock, Tagging, Metadata 2024-05-22 12:16:55 -07:00
jonaustin09
e38c63448d fix: Changed GetObject by range success status code from 200 to 206 2024-05-20 14:48:02 -04:00
jonaustin09
4bde84eafd fix: Added status property validation for PutObjectLegalHold action 2024-05-17 12:39:27 -04:00
jonaustin09
fbb7c4a888 fix: Fixed CopyObject action response body type 2024-05-07 11:57:36 -04:00
jonaustin09
2a2f9c827c feat: Closes #484. Added support to run the gateway on read only mode 2024-05-06 16:41:39 -04:00
jonaustin09
481c9246c6 feat: HeadObject ation multipart upload case 2024-05-03 18:10:32 -04:00
jonaustin09
b4cd35f60b feat: error refactoring and enable object lock in backends
Added support to enable object lock on bucket creation in posix and azure
backends.
Implemented the logic to add object legal hold and retention on object creation
in azure and posix backends.
Added the functionality for HeadObject to return object lock related headers.
Added integration tests for these features.
2024-05-02 15:23:48 -07:00
jonaustin09
0c3771ae2d feat: Added GetObjectAttributes actions implementation in posix, azure and s3 backends. Added integration tests for GetObjectAttributes action 2024-04-29 15:31:53 -04:00
jonaustin09
6f9c6fde37 feat: Added DeleteObjects event support in bucket event notifications 2024-04-25 16:18:02 -04:00
jonaustin09
89755ea5aa feat: Changed object lock actions interface to put/get []byte 2024-04-22 13:19:09 -07:00
jonaustin09
fbaba0b944 feat: Added object WORM protection by object-lock feature from AWS with the following actions support: PutObjectLockConfiguration, GetObjectLockConfiguration, PutObjectRetention, GetObjectRetention, PutObjectLegalHold, GetObjectLegalHold 2024-04-22 13:13:40 -07:00
jonaustin09
84ce40fb54 fix: Fixes #487, added response headers for HeadBucket action 2024-04-18 13:27:45 -04:00
Ben McClelland
4be4dc2971 feat: add more debug logging for api handler errors
There are a few cases where parsing, validations checks, etc
error details are getting lost with the more generic error
responses. This add some opt-in debug logging to log more
info for these various error cases.
2024-04-06 20:08:16 -07:00
jonaustin09
dbc0ad4325 feat: Closes #475, Implemented filters for s3 bucket event notifications, created a utility CLI command to create config file 2024-04-04 13:25:01 -07:00
Ben McClelland
4d02ac21c5 Merge pull request #460 from versity/bucket-policies
Bucket Policy
2024-03-25 15:19:46 -07:00
jonaustin09
754c221c4d feat: Added bucket policy access verifier function implementation. Changed the default behaviour of bucket ACLs. Fixed the supported actions list for bucket policy. Implemented Copy* actions access checker function 2024-03-25 16:00:35 -04:00
jonaustin09
d469a72213 feat: Implemented Put/Get/DeletBucketPolicy s3 actions in posix backend. Implemented policy document validation function 2024-03-15 15:47:10 -04:00
Ben McClelland
b801a700d5 Merge pull request #449 from versity/ben/input_tag_format
fix: remove namespace restrictions on tag xml input
2024-03-12 11:46:36 -07:00
Ben McClelland
08e5c568d5 fix: zero len put error when content length value not defined
Fixes #444. For some clients using chunked uploads with a zero
length file, the content length value from the request headers
was coming back as an empty string. If this happens, just set
it to "0" so that we can successfully parse this to int value.
2024-03-11 21:15:34 -07:00
Ben McClelland
0d8a4f5791 fix: remove namespace restrictions on tag xml input
Fixes #447. Previously we required XML namespace and got these
errors with this input:
DEBUG:  <Tagging><TagSet><Tag><Key>mykey</Key><Value>myvalue</Value></Tag></TagSet></Tagging>
DEBUG: expected element <Tagging> in name space http://s3.amazonaws.com/doc/2006-03-01/ but have no name space
2024-03-11 21:01:40 -07:00
jonaustin09
d4f17bf32f feat: Added bucket policy actions implementation in FE 2024-03-06 13:56:29 -05:00
jonaustin09
e6852b3a99 feat: Closes 417, Added the following versioning related actions: PutBucketVersioning, GetBucketVersioning, ListObjectVersions. Added versionId support in FE for the following actions: GetObject, DeleteObject 2024-02-28 09:48:05 -05:00
Ben McClelland
0760467c3d fix: correct xml response encoding for list-buckets and tagging
fixes #395
2024-02-12 16:20:07 -08:00
Jon Austin
0c056f935b ListObjectsV2 start-after prop (#388)
* fix: Fixes #138, Added StartAfter property in ListObjectsV2 action, added couple of integration tests for ListObjectsV2
2024-02-01 11:04:52 -08:00
jonaustin09
3cce3a5201 feat: Added unit and integration test cases for posix bucket tagging related actions 2024-01-31 10:09:48 -08:00
jonaustin09
d70ea61830 feat: Added the following actions support in posix backend: PutBucketTagging, GetBucketTagging, DeleteBucketTagging 2024-01-31 10:09:48 -08:00
jonaustin09
0d3a238ceb feat: Implemented logic to add bucket ACL on bucket creation 2024-01-31 09:49:56 -08:00
Ben McClelland
6481e2aac5 fix: cleanup backend ACLs
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.

This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
2024-01-10 09:36:00 -08:00
Ben McClelland
ba501e482d feat: steaming requests for put object and put part
This builds on the previous work that sets up the body streaming
for the put object and put part requests. This adds the auth and
checksum readers to postpone the v4auth checks and the content
checksum until the end of the body stream.

This means that the backend with start reading the data from the
body stream before the request is fully validated and signatures
checked. So the backend must check the error returned from the
body reader for the final auth and content checks. The backend
is expected to discard the data upon error.

This should increase performance and reduce memory utilization
to no longer require caching the entire request body in memory
for put object and put part.
2023-12-14 19:19:46 -08:00
jonaustin09
27eb43d089 feat: Closes #290, implemented request body stream reading for PutObject and UploadPart actions. 2023-12-14 18:27:51 -08:00
Ben McClelland
c6cbd82f8b fix: Fixes 330, Fixes 334, Fixed copySrcModifSince bug in CopyObject action
commit 48d951edfe
Author: jonaustin09 <jonaustin460@gmail.com>
Date:   Wed Dec 13 09:06:44 2023 -0500
2023-12-13 15:35:46 -08:00
Jon Austin
e9f01c8cce S3 proxy ListParts max parts issue (#338)
* feat: implemented the logic to run integration tests separately

* fix: Resolved tests closer bug

* fix: Fixes #329, Fixed ListParts max-parts property issue

* fix: removed max-parts max int value check
2023-12-13 10:22:34 -08:00
jonaustin09
48818927bb feat: Fixes #286, Created a struct which handles s3 select event streaming and event message construction 2023-12-06 14:02:36 -08:00
Ben McClelland
cd8ad7d482 fix: breaking changes with aws sdk updates 2023-11-28 13:51:32 -08:00
jonaustin09
4f8e4714ee fix: Fixes #286, Fixed SelectObjectContent CLI bug, added Content-Length header in response 2023-10-13 13:23:19 -04:00
jonaustin09
52674ab0c5 feat: Closes #282, created a new integration test group and test cases for posix specific behaviours testing 2023-10-11 08:17:10 -07:00
Ben McClelland
4661af11dd feat: replace access/role context locals with full account info 2023-10-02 10:59:59 -07:00
Jon Austin
9cb357ecc5 CopyObject metadata (#265)
* fix: Object tag actions cleanup

* fix: Fixes #249, Changed ListObjects default max-keys from -1 to 1000

* fix: Fixes #250, Added support to provide a marker not from the objects list and list the objects after the provided marker in ListObjects(V2) actions

* feat: Closes #256, Addded a check step, to compare object metadatas and allow the copying to itself, if the metadata has been changed

* fix: Simplified range assignment in CopyObject posix function
2023-09-26 18:09:09 -07:00
jonaustin09
099ac39f22 fix: Fixes #259, Changed delete actions response statuses from 200 to 204 2023-09-23 21:21:47 -07:00
jonaustin09
4be5d64c8b fix: Object tag actions cleanup 2023-09-23 21:00:45 -07:00
jonaustin09
e0c09ad4d9 fix: Fixes #247, Changed DeleteObjectTagging action successful response status from 200 to 204 2023-09-20 12:07:52 -04:00
jonaustin09
e392ac940a fix: VerifyACL function clenup: removed unused bucket argument from the function declaration 2023-09-19 16:36:42 -07:00
jonaustin09
a77954a307 fix: Fixes #246, Added max length check for tag keys and values in PutObjectTagging and PutObject actions 2023-09-19 14:42:30 -04:00