Commit Graph

164 Commits

Author SHA1 Message Date
Ben McClelland
d4d511cf98 feat: compile support for 32bit and bsd platforms 2024-03-11 09:42:36 -07:00
Ben McClelland
5b2beb8fc0 Merge pull request #443 from versity/ben/cleanup
chore: cleanup redundant nil checks and unused variable args
2024-03-11 09:01:17 -07:00
Ben McClelland
39663724a6 chore: cleanup redundant nil checks and unused variable args 2024-03-09 10:34:04 -08:00
Ben McClelland
f7655dab9b fix: delete object xml response should be DeleteResult instead of DeleteObjectsResult 2024-03-09 10:20:15 -08:00
jonaustin09
d4f17bf32f feat: Added bucket policy actions implementation in FE 2024-03-06 13:56:29 -05:00
jonaustin09
e6852b3a99 feat: Closes 417, Added the following versioning related actions: PutBucketVersioning, GetBucketVersioning, ListObjectVersions. Added versionId support in FE for the following actions: GetObject, DeleteObject 2024-02-28 09:48:05 -05:00
jonaustin09
4d8352225b feat: Added bucket tagging actions in azure backend 2024-02-15 15:23:55 -05:00
Jon Austin
0c056f935b ListObjectsV2 start-after prop (#388)
* fix: Fixes #138, Added StartAfter property in ListObjectsV2 action, added couple of integration tests for ListObjectsV2
2024-02-01 11:04:52 -08:00
Ben McClelland
68794518af fix: remove special proxy handling for bucket acls in posix backend 2024-01-31 10:10:12 -08:00
jonaustin09
3cce3a5201 feat: Added unit and integration test cases for posix bucket tagging related actions 2024-01-31 10:09:48 -08:00
jonaustin09
d70ea61830 feat: Added the following actions support in posix backend: PutBucketTagging, GetBucketTagging, DeleteBucketTagging 2024-01-31 10:09:48 -08:00
Jon Austin
cf92b6fd80 Fix/azure copy object (#382)
* fix: Added destination bucket acl check and metadata comparision for CopyObject action in azure backend

---------

Co-authored-by: Ben McClelland <ben.mcclelland@versity.com>
2024-01-22 10:01:16 -08:00
Jon Austin
68e800492e Fix/azure list objects (#379)
* fix: Added pagination to ListObjects and ListObjectsV2 actions, fixed multipart upload non existing key error handling
2024-01-22 09:54:45 -08:00
jonaustin09
03e4a28d57 fix: Fixed couple of bugs regarding to GetObject range errors, blob metadata reference losing 2024-01-17 08:27:37 -08:00
jonaustin09
240db54feb feat: Added ChangeBucketOwner, ListBucketsAndOwners action implementation in azure backend. Fixed acl key bug in getting container metadata. Added container owner in ListBuckets action 2024-01-17 08:27:37 -08:00
Ben McClelland
d404f96320 fix: translate azure errors to s3 for compatibility 2024-01-17 08:27:37 -08:00
Ben McClelland
6481e2aac5 fix: cleanup backend ACLs
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.

This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
2024-01-10 09:36:00 -08:00
Ben McClelland
3db43b7206 feat: add azure local env auth
This is the recommended auth from the following:
https://github.com/Azure-Samples/storage-blobs-go-quickstart/blob/master/storage-quickstart.go
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-go?toc=%2Fazure%2Fdeveloper%2Fgo%2Ftoc.json&bc=%2Fazure%2Fdeveloper%2Fgo%2Fbreadcrumb%2Ftoc.json&tabs=roles-azure-portal#authenticate-to-azure-and-authorize-access-to-blob-data
2024-01-09 22:21:39 -08:00
jonaustin09
e5fc12042b feat: Added sas token authentication for azure backend 2024-01-09 22:03:13 -08:00
Ben McClelland
a86a8cbce5 fix: add azure CreateMultipartUpload to allow clients to work as expected
The azure sdk doesnt use a separate function to initialize a
multipart upload, so CreateMultipartUpload becomes a no-op.
But we still need to have it return success so that clients
wont get an unexpected error.
2024-01-08 13:40:20 -08:00
Ben McClelland
bf38a03af9 chore: remove azure bug comment
This comment references a bug that was fixed in the v1.2.1 sdk
update:
https://github.com/Azure/azure-sdk-for-go/issues/22171
2024-01-08 13:11:41 -08:00
Jon Austin
9bfec719f3 Azure ACL (#364)
feat: Addded GetBucketAcl and PutBucketAcl actions implementation in azure backend. ACL is stored in the container metadata
2024-01-03 11:15:53 -08:00
jonaustin09
cafa45760c feat: Added pagination for ListParts azure action and added get range support for GetObject azure action 2023-12-29 21:55:32 -08:00
jonaustin09
8cc89fa713 feat: Azure backend implementation 2023-12-29 21:55:32 -08:00
Ben McClelland
3b945f72fc feat: azure blob backend initial pass 2023-12-29 21:54:56 -08:00
Ben McClelland
c1587e4c1c fix: make s3 PutBucketAcl have receiver pointer like others for consistency 2023-12-26 10:31:56 -08:00
Ben McClelland
6e37096b35 fix: s3 backend performance increase with client reuse
The previous way of initializing the s3 client in each call was
adding a lot of overhead and would tank performance beyond about
20 simultaneous requests.

Since the backend access is through a single account, we can init
and store this client for use from each api call.
2023-12-22 15:09:44 -08:00
Ben McClelland
2fb5ecfbc4 chore: clarify s3 backend struct with better name 2023-12-16 09:50:26 -08:00
Ben McClelland
ba501e482d feat: steaming requests for put object and put part
This builds on the previous work that sets up the body streaming
for the put object and put part requests. This adds the auth and
checksum readers to postpone the v4auth checks and the content
checksum until the end of the body stream.

This means that the backend with start reading the data from the
body stream before the request is fully validated and signatures
checked. So the backend must check the error returned from the
body reader for the final auth and content checks. The backend
is expected to discard the data upon error.

This should increase performance and reduce memory utilization
to no longer require caching the entire request body in memory
for put object and put part.
2023-12-14 19:19:46 -08:00
jonaustin09
27eb43d089 feat: Closes #290, implemented request body stream reading for PutObject and UploadPart actions. 2023-12-14 18:27:51 -08:00
Jon Austin
90bb43f7c9 S3 Proxy PutBucketAcl reference bug (#341)
* feat: implemented the logic to run integration tests separately

* fix: Fixes #336, fixed s3 proxy PutBucketAcl action's referance bug
2023-12-14 09:43:02 -08:00
Jon Austin
206231f27b S3 Proxy DeleteObjects empty input (#340)
* feat: implemented the logic to run integration tests separately

* fix: Fixes #335, fixed s3 proxy DeleteObject empty input bug
2023-12-14 09:42:22 -08:00
jonaustin09
ef5a94420c feat: Created admin CLI actions in s3 proxy, Created iam proxy for proxy server 2023-12-11 11:05:08 -08:00
jonaustin09
48818927bb feat: Fixes #286, Created a struct which handles s3 select event streaming and event message construction 2023-12-06 14:02:36 -08:00
jonaustin09
c5007a68aa eat: Closes #317, Changed s3 proxy behavior to run in single user mode with CLI provided credentials 2023-12-04 10:06:59 -05:00
Ben McClelland
c48f126557 Merge pull request #318 from versity/fix/issue-300-proxy-listbuckets-panic
S3 proxy ListBucket panic
2023-11-29 09:51:17 -08:00
Jon Austin
315cb0ea81 fix: Fixes #307, Fixed ListMultipartUploads and ListParts parts timestamp in s3 proxy (#316) 2023-11-29 08:54:12 -08:00
jonaustin09
c4b019f6ee fix: Fixes #300, Removed Displayname field from ListBuckets output as it's not supported in the gateway 2023-11-29 11:49:27 -05:00
jonaustin09
2b22509a90 fix: Fixes #302, #308. Added error handling function in s3 proxy implementation, which converts sdk error to API Errors 2023-11-29 08:40:06 -08:00
Ben McClelland
cd8ad7d482 fix: breaking changes with aws sdk updates 2023-11-28 13:51:32 -08:00
jonaustin09
3a60dcd88f feat: Added proxy backend actions: DeleteObjectTagging, GetObjectTagging, PutObjectTagging, PutBucketAcl, GetBucketAcl 2023-10-11 09:00:02 -07:00
Ben McClelland
f58646b58d feat: s3proxy backend
This backend redirects incoming requests to another s3 service.
This will use the incoming credentials to setup the client
requests to the external s3 service. So the IAM accounts (or
root account) must match what the external s3 service expects.
2023-10-11 09:00:02 -07:00
jonaustin09
b8140fe3ed fix: Fixed error case when overwriting a directory object with file one 2023-10-10 18:20:42 -04:00
Ben McClelland
23281774aa fix: allow posix GET of 0-len directory type object 2023-10-07 15:57:31 -07:00
Ben McClelland
5ca44e7c2f fix: prevent directory type object uploads containing data
Since objects with trailing "/" are mapped to directories in the
posix filesystem, they must not contain data since there is no
place to store that data.

This checks both PutObject and CreateMultipartUpload for invalid
directory object types containing data.
2023-10-07 15:36:03 -07:00
jonaustin09
856d79d385 fix: Fixes #274, Fixed putting and getting object metadata case normalization issue 2023-10-05 15:33:03 -04:00
Ben McClelland
ae4e382e61 feat: refactor internal iam service
This moves the internal iam service from the posix backend so
that we can start implementing new iam services right in the auth
module.

The internal iam service has same behavior as before, but now
must be enabled with the --iam-dir cli option.

New single user service is the default when no other iam service
is selected. This just runs the gateway in single user mode with
just the root account.
2023-10-02 11:12:18 -07:00
Ben McClelland
4661af11dd feat: replace access/role context locals with full account info 2023-10-02 10:59:59 -07:00
Jon Austin
9cb357ecc5 CopyObject metadata (#265)
* fix: Object tag actions cleanup

* fix: Fixes #249, Changed ListObjects default max-keys from -1 to 1000

* fix: Fixes #250, Added support to provide a marker not from the objects list and list the objects after the provided marker in ListObjects(V2) actions

* feat: Closes #256, Addded a check step, to compare object metadatas and allow the copying to itself, if the metadata has been changed

* fix: Simplified range assignment in CopyObject posix function
2023-09-26 18:09:09 -07:00
jonaustin09
5c48fcd443 fix: Fixes #250, Added support to provide a marker not from the objects list and list the objects after the provided marker in ListObjects(V2) actions 2023-09-23 21:10:20 -07:00