jonaustin09
b4cd35f60b
feat: error refactoring and enable object lock in backends
...
Added support to enable object lock on bucket creation in posix and azure
backends.
Implemented the logic to add object legal hold and retention on object creation
in azure and posix backends.
Added the functionality for HeadObject to return object lock related headers.
Added integration tests for these features.
2024-05-02 15:23:48 -07:00
jonaustin09
0c3771ae2d
feat: Added GetObjectAttributes actions implementation in posix, azure and s3 backends. Added integration tests for GetObjectAttributes action
2024-04-29 15:31:53 -04:00
jonaustin09
5126aedeff
feat: Added bucket policy and object lock actions implementation in s3 proxy
2024-04-24 15:49:02 -04:00
jonaustin09
4a56d570ad
feat: Added object lock actions implementation in azure backend
2024-04-23 17:05:59 -04:00
jonaustin09
89755ea5aa
feat: Changed object lock actions interface to put/get []byte
2024-04-22 13:19:09 -07:00
jonaustin09
00476ef70c
feat: Closes #490 , Added integration tests for object lock actions
2024-04-22 13:13:40 -07:00
jonaustin09
fbaba0b944
feat: Added object WORM protection by object-lock feature from AWS with the following actions support: PutObjectLockConfiguration, GetObjectLockConfiguration, PutObjectRetention, GetObjectRetention, PutObjectLegalHold, GetObjectLegalHold
2024-04-22 13:13:40 -07:00
Ben McClelland
7ab0e3ebbe
Merge pull request #530 from versity/azure-policy-actions
...
Azure bucket policy actions
2024-04-20 10:00:23 -07:00
jonaustin09
5c835c5c74
feat: Implemented GetBucketPolicy, PutBucketPolicy action in azure backend
2024-04-19 16:36:42 -04:00
Ben McClelland
4429570388
fix: use xattr.ENOATTR check for posix xattrs
...
The xattr package has a more universal error type for xattrs
not existing. Use this for better platform compatibility.
This also adds the xattr.XATTR_SUPPORTED check for platform
xattr suport in xattr package.
Fixes #527
2024-04-18 18:20:43 -07:00
Ben McClelland
c81403fe90
feat: add metadata storage abstraction layer
...
Closes #511 . This adds an abstraction layer to the metadata
storage to allow for future non-xattr metadata storage
implementations.
2024-04-15 13:57:31 -07:00
Ben McClelland
aeea61544b
feat: add s3err QuotaExceeded for posix/scoutfs
...
When fileystem quota exceeded, the gateway will now return the
error:
S3 error: 403 (QuotaExceeded):
Your request was denied due to quota exceeded.
This will help clients to better detect upload errors due to
quota exceeded.
Fixes #483
2024-04-06 11:53:40 -07:00
Ben McClelland
c228bbfd79
feat: add option to change ownership of dir/files to acct settings
...
When enabled, any new directories or files created through the
gateway will change ownership based on the account uid/gid.
Fixes #238 .
2024-03-30 22:26:29 -07:00
Ben McClelland
f72d6349fe
feat: consolidate scoutfs/posix mkdir in backend
...
We had some duplicated code that we can bring into the backend
package so that we can remove duplications. This moves the mkdir
implementation into backend so that both posix and scoutfs can
call the same implementation.
2024-03-30 22:26:29 -07:00
jonaustin09
e6203c5765
feat: Closes #441 , Added access control integration tests, fixed some bugs in bucket policy and acl access checking flow
2024-03-28 14:52:56 -04:00
Ben McClelland
4d02ac21c5
Merge pull request #460 from versity/bucket-policies
...
Bucket Policy
2024-03-25 15:19:46 -07:00
jonaustin09
754c221c4d
feat: Added bucket policy access verifier function implementation. Changed the default behaviour of bucket ACLs. Fixed the supported actions list for bucket policy. Implemented Copy* actions access checker function
2024-03-25 16:00:35 -04:00
Ben McClelland
0ea5db228d
fix: scoutfs return correct ContentRange for get request
2024-03-25 09:36:38 -07:00
Ben McClelland
7ff89af6b5
fix: scoutfs backend needs to dereference types.CompletedPart.PartNumber
...
There was a change a while back in the aws sdk that made these
pointers instead of ints. We somehow missed the pointer deref in
the scoutfs backend.
2024-03-25 08:29:03 -07:00
jonaustin09
af641e5368
feat: Added integration test cases for Put/Get/DeleteBucketPolicy actions. Made some bug fixes in these actions implementations
2024-03-20 17:31:52 -04:00
jonaustin09
d469a72213
feat: Implemented Put/Get/DeletBucketPolicy s3 actions in posix backend. Implemented policy document validation function
2024-03-15 15:47:10 -04:00
Ben McClelland
8144d90e25
feat: check for xattr support on posix init
2024-03-14 08:16:52 -07:00
Ben McClelland
d4d511cf98
feat: compile support for 32bit and bsd platforms
2024-03-11 09:42:36 -07:00
Ben McClelland
5b2beb8fc0
Merge pull request #443 from versity/ben/cleanup
...
chore: cleanup redundant nil checks and unused variable args
2024-03-11 09:01:17 -07:00
Ben McClelland
39663724a6
chore: cleanup redundant nil checks and unused variable args
2024-03-09 10:34:04 -08:00
Ben McClelland
f7655dab9b
fix: delete object xml response should be DeleteResult instead of DeleteObjectsResult
2024-03-09 10:20:15 -08:00
jonaustin09
d4f17bf32f
feat: Added bucket policy actions implementation in FE
2024-03-06 13:56:29 -05:00
jonaustin09
e6852b3a99
feat: Closes 417, Added the following versioning related actions: PutBucketVersioning, GetBucketVersioning, ListObjectVersions. Added versionId support in FE for the following actions: GetObject, DeleteObject
2024-02-28 09:48:05 -05:00
jonaustin09
4d8352225b
feat: Added bucket tagging actions in azure backend
2024-02-15 15:23:55 -05:00
Jon Austin
0c056f935b
ListObjectsV2 start-after prop ( #388 )
...
* fix: Fixes #138 , Added StartAfter property in ListObjectsV2 action, added couple of integration tests for ListObjectsV2
2024-02-01 11:04:52 -08:00
Ben McClelland
68794518af
fix: remove special proxy handling for bucket acls in posix backend
2024-01-31 10:10:12 -08:00
jonaustin09
3cce3a5201
feat: Added unit and integration test cases for posix bucket tagging related actions
2024-01-31 10:09:48 -08:00
jonaustin09
d70ea61830
feat: Added the following actions support in posix backend: PutBucketTagging, GetBucketTagging, DeleteBucketTagging
2024-01-31 10:09:48 -08:00
Jon Austin
cf92b6fd80
Fix/azure copy object ( #382 )
...
* fix: Added destination bucket acl check and metadata comparision for CopyObject action in azure backend
---------
Co-authored-by: Ben McClelland <ben.mcclelland@versity.com >
2024-01-22 10:01:16 -08:00
Jon Austin
68e800492e
Fix/azure list objects ( #379 )
...
* fix: Added pagination to ListObjects and ListObjectsV2 actions, fixed multipart upload non existing key error handling
2024-01-22 09:54:45 -08:00
jonaustin09
03e4a28d57
fix: Fixed couple of bugs regarding to GetObject range errors, blob metadata reference losing
2024-01-17 08:27:37 -08:00
jonaustin09
240db54feb
feat: Added ChangeBucketOwner, ListBucketsAndOwners action implementation in azure backend. Fixed acl key bug in getting container metadata. Added container owner in ListBuckets action
2024-01-17 08:27:37 -08:00
Ben McClelland
d404f96320
fix: translate azure errors to s3 for compatibility
2024-01-17 08:27:37 -08:00
Ben McClelland
6481e2aac5
fix: cleanup backend ACLs
...
This adds the default ACL to the CreateBucket backend method so
that the backend doesn't need to know how to construct and ACL.
This also moves the s3proxy ACLs to a tag key/value because the
gateway ACLs are not the same accounts as the backend s3 server.
TODO: we may need to mask this tag key/value if we add support
for the Get/PutBucketTagging API.
2024-01-10 09:36:00 -08:00
Ben McClelland
3db43b7206
feat: add azure local env auth
...
This is the recommended auth from the following:
https://github.com/Azure-Samples/storage-blobs-go-quickstart/blob/master/storage-quickstart.go
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-go?toc=%2Fazure%2Fdeveloper%2Fgo%2Ftoc.json&bc=%2Fazure%2Fdeveloper%2Fgo%2Fbreadcrumb%2Ftoc.json&tabs=roles-azure-portal#authenticate-to-azure-and-authorize-access-to-blob-data
2024-01-09 22:21:39 -08:00
jonaustin09
e5fc12042b
feat: Added sas token authentication for azure backend
2024-01-09 22:03:13 -08:00
Ben McClelland
a86a8cbce5
fix: add azure CreateMultipartUpload to allow clients to work as expected
...
The azure sdk doesnt use a separate function to initialize a
multipart upload, so CreateMultipartUpload becomes a no-op.
But we still need to have it return success so that clients
wont get an unexpected error.
2024-01-08 13:40:20 -08:00
Ben McClelland
bf38a03af9
chore: remove azure bug comment
...
This comment references a bug that was fixed in the v1.2.1 sdk
update:
https://github.com/Azure/azure-sdk-for-go/issues/22171
2024-01-08 13:11:41 -08:00
Jon Austin
9bfec719f3
Azure ACL ( #364 )
...
feat: Addded GetBucketAcl and PutBucketAcl actions implementation in azure backend. ACL is stored in the container metadata
2024-01-03 11:15:53 -08:00
jonaustin09
cafa45760c
feat: Added pagination for ListParts azure action and added get range support for GetObject azure action
2023-12-29 21:55:32 -08:00
jonaustin09
8cc89fa713
feat: Azure backend implementation
2023-12-29 21:55:32 -08:00
Ben McClelland
3b945f72fc
feat: azure blob backend initial pass
2023-12-29 21:54:56 -08:00
Ben McClelland
c1587e4c1c
fix: make s3 PutBucketAcl have receiver pointer like others for consistency
2023-12-26 10:31:56 -08:00
Ben McClelland
6e37096b35
fix: s3 backend performance increase with client reuse
...
The previous way of initializing the s3 client in each call was
adding a lot of overhead and would tank performance beyond about
20 simultaneous requests.
Since the backend access is through a single account, we can init
and store this client for use from each api call.
2023-12-22 15:09:44 -08:00
Ben McClelland
2fb5ecfbc4
chore: clarify s3 backend struct with better name
2023-12-16 09:50:26 -08:00
