Ben McClelland
4429570388
fix: use xattr.ENOATTR check for posix xattrs
...
The xattr package has a more universal error type for xattrs
not existing. Use this for better platform compatibility.
This also adds the xattr.XATTR_SUPPORTED check for platform
xattr suport in xattr package.
Fixes #527
2024-04-18 18:20:43 -07:00
Ben McClelland
c81403fe90
feat: add metadata storage abstraction layer
...
Closes #511 . This adds an abstraction layer to the metadata
storage to allow for future non-xattr metadata storage
implementations.
2024-04-15 13:57:31 -07:00
Ben McClelland
aeea61544b
feat: add s3err QuotaExceeded for posix/scoutfs
...
When fileystem quota exceeded, the gateway will now return the
error:
S3 error: 403 (QuotaExceeded):
Your request was denied due to quota exceeded.
This will help clients to better detect upload errors due to
quota exceeded.
Fixes #483
2024-04-06 11:53:40 -07:00
Ben McClelland
c228bbfd79
feat: add option to change ownership of dir/files to acct settings
...
When enabled, any new directories or files created through the
gateway will change ownership based on the account uid/gid.
Fixes #238 .
2024-03-30 22:26:29 -07:00
Ben McClelland
f72d6349fe
feat: consolidate scoutfs/posix mkdir in backend
...
We had some duplicated code that we can bring into the backend
package so that we can remove duplications. This moves the mkdir
implementation into backend so that both posix and scoutfs can
call the same implementation.
2024-03-30 22:26:29 -07:00
jonaustin09
e6203c5765
feat: Closes #441 , Added access control integration tests, fixed some bugs in bucket policy and acl access checking flow
2024-03-28 14:52:56 -04:00
Ben McClelland
4d02ac21c5
Merge pull request #460 from versity/bucket-policies
...
Bucket Policy
2024-03-25 15:19:46 -07:00
jonaustin09
754c221c4d
feat: Added bucket policy access verifier function implementation. Changed the default behaviour of bucket ACLs. Fixed the supported actions list for bucket policy. Implemented Copy* actions access checker function
2024-03-25 16:00:35 -04:00
Ben McClelland
0ea5db228d
fix: scoutfs return correct ContentRange for get request
2024-03-25 09:36:38 -07:00
Ben McClelland
7ff89af6b5
fix: scoutfs backend needs to dereference types.CompletedPart.PartNumber
...
There was a change a while back in the aws sdk that made these
pointers instead of ints. We somehow missed the pointer deref in
the scoutfs backend.
2024-03-25 08:29:03 -07:00
jonaustin09
af641e5368
feat: Added integration test cases for Put/Get/DeleteBucketPolicy actions. Made some bug fixes in these actions implementations
2024-03-20 17:31:52 -04:00
jonaustin09
d469a72213
feat: Implemented Put/Get/DeletBucketPolicy s3 actions in posix backend. Implemented policy document validation function
2024-03-15 15:47:10 -04:00
Ben McClelland
8144d90e25
feat: check for xattr support on posix init
2024-03-14 08:16:52 -07:00
Ben McClelland
d4d511cf98
feat: compile support for 32bit and bsd platforms
2024-03-11 09:42:36 -07:00
Ben McClelland
5b2beb8fc0
Merge pull request #443 from versity/ben/cleanup
...
chore: cleanup redundant nil checks and unused variable args
2024-03-11 09:01:17 -07:00
Ben McClelland
39663724a6
chore: cleanup redundant nil checks and unused variable args
2024-03-09 10:34:04 -08:00
Ben McClelland
f7655dab9b
fix: delete object xml response should be DeleteResult instead of DeleteObjectsResult
2024-03-09 10:20:15 -08:00
jonaustin09
d4f17bf32f
feat: Added bucket policy actions implementation in FE
2024-03-06 13:56:29 -05:00
jonaustin09
e6852b3a99
feat: Closes 417, Added the following versioning related actions: PutBucketVersioning, GetBucketVersioning, ListObjectVersions. Added versionId support in FE for the following actions: GetObject, DeleteObject
2024-02-28 09:48:05 -05:00
jonaustin09
4d8352225b
feat: Added bucket tagging actions in azure backend
2024-02-15 15:23:55 -05:00
Jon Austin
0c056f935b
ListObjectsV2 start-after prop ( #388 )
...
* fix: Fixes #138 , Added StartAfter property in ListObjectsV2 action, added couple of integration tests for ListObjectsV2
2024-02-01 11:04:52 -08:00
Ben McClelland
68794518af
fix: remove special proxy handling for bucket acls in posix backend
2024-01-31 10:10:12 -08:00
jonaustin09
3cce3a5201
feat: Added unit and integration test cases for posix bucket tagging related actions
2024-01-31 10:09:48 -08:00
jonaustin09
d70ea61830
feat: Added the following actions support in posix backend: PutBucketTagging, GetBucketTagging, DeleteBucketTagging
2024-01-31 10:09:48 -08:00
Jon Austin
cf92b6fd80
Fix/azure copy object ( #382 )
...
* fix: Added destination bucket acl check and metadata comparision for CopyObject action in azure backend
---------
Co-authored-by: Ben McClelland <ben.mcclelland@versity.com >
2024-01-22 10:01:16 -08:00
Jon Austin
68e800492e
Fix/azure list objects ( #379 )
...
* fix: Added pagination to ListObjects and ListObjectsV2 actions, fixed multipart upload non existing key error handling
2024-01-22 09:54:45 -08:00
jonaustin09
03e4a28d57
fix: Fixed couple of bugs regarding to GetObject range errors, blob metadata reference losing
2024-01-17 08:27:37 -08:00
jonaustin09
240db54feb
feat: Added ChangeBucketOwner, ListBucketsAndOwners action implementation in azure backend. Fixed acl key bug in getting container metadata. Added container owner in ListBuckets action
2024-01-17 08:27:37 -08:00
Ben McClelland
d404f96320
fix: translate azure errors to s3 for compatibility
2024-01-17 08:27:37 -08:00
Ben McClelland
6481e2aac5
fix: cleanup backend ACLs
...
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.
This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
2024-01-10 09:36:00 -08:00
Ben McClelland
3db43b7206
feat: add azure local env auth
...
This is the recommended auth from the following:
https://github.com/Azure-Samples/storage-blobs-go-quickstart/blob/master/storage-quickstart.go
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-go?toc=%2Fazure%2Fdeveloper%2Fgo%2Ftoc.json&bc=%2Fazure%2Fdeveloper%2Fgo%2Fbreadcrumb%2Ftoc.json&tabs=roles-azure-portal#authenticate-to-azure-and-authorize-access-to-blob-data
2024-01-09 22:21:39 -08:00
jonaustin09
e5fc12042b
feat: Added sas token authentication for azure backend
2024-01-09 22:03:13 -08:00
Ben McClelland
a86a8cbce5
fix: add azure CreateMultipartUpload to allow clients to work as expected
...
The azure sdk doesnt use a separate function to initialize a
multipart upload, so CreateMultipartUpload becomes a no-op.
But we still need to have it return success so that clients
wont get an unexpected error.
2024-01-08 13:40:20 -08:00
Ben McClelland
bf38a03af9
chore: remove azure bug comment
...
This comment references a bug that was fixed in the v1.2.1 sdk
update:
https://github.com/Azure/azure-sdk-for-go/issues/22171
2024-01-08 13:11:41 -08:00
Jon Austin
9bfec719f3
Azure ACL ( #364 )
...
feat: Addded GetBucketAcl and PutBucketAcl actions implementation in azure backend. ACL is stored in the container metadata
2024-01-03 11:15:53 -08:00
jonaustin09
cafa45760c
feat: Added pagination for ListParts azure action and added get range support for GetObject azure action
2023-12-29 21:55:32 -08:00
jonaustin09
8cc89fa713
feat: Azure backend implementation
2023-12-29 21:55:32 -08:00
Ben McClelland
3b945f72fc
feat: azure blob backend initial pass
2023-12-29 21:54:56 -08:00
Ben McClelland
c1587e4c1c
fix: make s3 PutBucketAcl have receiver pointer like others for consistency
2023-12-26 10:31:56 -08:00
Ben McClelland
6e37096b35
fix: s3 backend performance increase with client reuse
...
The previous way of initializing the s3 client in each call was
adding a lot of overhead and would tank performance beyond about
20 simultaneous requests.
Since the backend access is through a single account, we can init
and store this client for use from each api call.
2023-12-22 15:09:44 -08:00
Ben McClelland
2fb5ecfbc4
chore: clarify s3 backend struct with better name
2023-12-16 09:50:26 -08:00
Ben McClelland
ba501e482d
feat: steaming requests for put object and put part
...
This builds on the previous work that sets up the body streaming
for the put object and put part requests. This adds the auth and
checksum readers to postpone the v4auth checks and the content
checksum until the end of the body stream.
This means that the backend with start reading the data from the
body stream before the request is fully validated and signatures
checked. So the backend must check the error returned from the
body reader for the final auth and content checks. The backend
is expected to discard the data upon error.
This should increase performance and reduce memory utilization
to no longer require caching the entire request body in memory
for put object and put part.
2023-12-14 19:19:46 -08:00
jonaustin09
27eb43d089
feat: Closes #290 , implemented request body stream reading for PutObject and UploadPart actions.
2023-12-14 18:27:51 -08:00
Jon Austin
90bb43f7c9
S3 Proxy PutBucketAcl reference bug ( #341 )
...
* feat: implemented the logic to run integration tests separately
* fix: Fixes #336 , fixed s3 proxy PutBucketAcl action's referance bug
2023-12-14 09:43:02 -08:00
Jon Austin
206231f27b
S3 Proxy DeleteObjects empty input ( #340 )
...
* feat: implemented the logic to run integration tests separately
* fix: Fixes #335 , fixed s3 proxy DeleteObject empty input bug
2023-12-14 09:42:22 -08:00
jonaustin09
ef5a94420c
feat: Created admin CLI actions in s3 proxy, Created iam proxy for proxy server
2023-12-11 11:05:08 -08:00
jonaustin09
48818927bb
feat: Fixes #286 , Created a struct which handles s3 select event streaming and event message construction
2023-12-06 14:02:36 -08:00
jonaustin09
c5007a68aa
eat: Closes #317 , Changed s3 proxy behavior to run in single user mode with CLI provided credentials
2023-12-04 10:06:59 -05:00
Ben McClelland
c48f126557
Merge pull request #318 from versity/fix/issue-300-proxy-listbuckets-panic
...
S3 proxy ListBucket panic
2023-11-29 09:51:17 -08:00
Jon Austin
315cb0ea81
fix: Fixes #307 , Fixed ListMultipartUploads and ListParts parts timestamp in s3 proxy ( #316 )
2023-11-29 08:54:12 -08:00
