mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-02-03 08:52:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,473 Commits 23 Branches 45 Tags
ben/plugin
Commit Graph

36 Commits

Author SHA1 Message Date
niksis02
cb97fb589b feat: Adds Ownder data in ListObjects(V2) result. 
Closes #819

ListObjects returns object owner data in each object entity in the result, while ListObjectsV2 has fetch-owner query param, which indicates if the objects owner data should be fetched.
Adds these changes in the gateway to add `Owner` data in `ListObjects` and `ListObjectsV2` result. In aws the objects can be owned by different users in the same bucket. In the gateway all the objects are owned by the bucket owner.
 2025-04-02 18:28:32 +04:00
niksis02
d82da64942 fix: Fixes the bucket policy json document validation errors for invalid jsons. 
Fixes #965

Changes the returned error description to `Policies must be valid JSON and the first byte must be '{'` for invalid bucket policy json documents, which doesn't start with `{`.
The gateway returns `This policy contains invalid Json` error description, if the document starts with `{`, but still isn't valid json.

Implements the `policyErr` string type which implements the `error` interface, to handle the policy json document validation errors, by avoiding staticchecker warnings.
 2025-03-20 01:08:47 +04:00
niksis02
cfb2d6d87d feat: Implements object meta properties for CopyObject in azure and posix backends. 
Fixes #998
Closes #1125
Closes #1126
Closes #1127

Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.

Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
 2025-03-17 09:37:05 -07:00
niksis02
d13791f5ce fix: Adds quotes for object Etags 2025-02-28 00:43:51 +04:00
niksis02
fcafb57abc feat: Implements checksums for ListObjectVersions and CopyObject actions. 2025-02-27 00:24:38 +04:00
niksis02
b53bbb025f fix: Adds crc64nvme checksum cases in the integration tests 2025-02-21 21:01:41 +04:00
niksis02
132d0ae631 feat: Adds the CRC64NVME checksum support in the gateway. Adds checksum-type support for the checksum implementation 2025-02-16 17:10:06 +04:00
niksis02
6956757557 feat: Integrates object integrity checksums(CRC32, CRC32C, SHA1, SHA256) into the gateway 2025-02-14 14:14:00 +04:00
niksis02
f35cdfb20c feat: Adds a flag on the test command to run the integration tests in insecure mode: by skiping the tls verification 2025-02-03 09:13:01 -08:00
dependabot[bot]
adb69ed041 chore(deps): bump the dev-dependencies group across 1 directory with 16 updates 
Bumps the dev-dependencies group with 9 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.32.5` | `1.32.6` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.67.1` | `1.70.0` |
| [github.com/aws/aws-sdk-go-v2/feature/ec2/imds](https://github.com/aws/aws-sdk-go-v2) | `1.16.20` | `1.16.21` |
| [github.com/aws/aws-sdk-go-v2/service/sso](https://github.com/aws/aws-sdk-go-v2) | `1.24.6` | `1.24.7` |
| [github.com/aws/aws-sdk-go-v2/service/ssooidc](https://github.com/aws/aws-sdk-go-v2) | `1.28.5` | `1.28.6` |
| [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) | `1.33.1` | `1.33.2` |
| [github.com/nats-io/nkeys](https://github.com/nats-io/nkeys) | `0.4.7` | `0.4.8` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.28.5` | `1.28.6` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.17.39` | `1.17.42` |

Updates `github.com/aws/aws-sdk-go-v2` from 1.32.5 to 1.32.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.32.5...v1.32.6)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.67.1 to 1.70.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.67.1...service/s3/v1.70.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.16.20 to 1.16.21
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/ram/v1.16.20...service/ram/v1.16.21)

Updates `github.com/aws/aws-sdk-go-v2/service/sso` from 1.24.6 to 1.24.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pi/v1.24.6...service/pi/v1.24.7)

Updates `github.com/aws/aws-sdk-go-v2/service/ssooidc` from 1.28.5 to 1.28.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.28.5...config/v1.28.6)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.33.1 to 1.33.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.33.1...service/sfn/v1.33.2)

Updates `github.com/nats-io/nkeys` from 0.4.7 to 0.4.8
- [Release notes](https://github.com/nats-io/nkeys/releases)
- [Changelog](https://github.com/nats-io/nkeys/blob/main/.goreleaser.yml)
- [Commits](https://github.com/nats-io/nkeys/compare/v0.4.7...v0.4.8)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.28.5 to 1.28.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.28.5...config/v1.28.6)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.17.46 to 1.17.47
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.46...credentials/v1.17.47)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.39 to 1.17.42
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.39...credentials/v1.17.42)

Updates `github.com/aws/aws-sdk-go-v2/internal/configsources` from 1.3.24 to 1.3.25
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/ini/v1.3.24...internal/ini/v1.3.25)

Updates `github.com/aws/aws-sdk-go-v2/internal/endpoints/v2` from 2.6.24 to 2.6.25
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/endpoints/v2.6.24...internal/endpoints/v2.6.25)

Updates `github.com/aws/aws-sdk-go-v2/internal/v4a` from 1.3.24 to 1.3.25
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/ini/v1.3.24...internal/ini/v1.3.25)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.4.5 to 1.4.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/m2/v1.4.5...service/m2/v1.4.6)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/presigned-url` from 1.12.5 to 1.12.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/credentials/v1.12.6/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/sso/v1.12.5...credentials/v1.12.6)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/s3shared` from 1.18.5 to 1.18.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.18.6/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.18.5...config/v1.18.6)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sso
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ssooidc
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/nats-io/nkeys
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/configsources
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/v4a
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/presigned-url
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/s3shared
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-12-04 14:48:03 -08:00
Ben McClelland
d9591f694e fix: azure admin list-buckets 
There were two issues that were preventing correct behavior here.
One was that we need to specifically request the container metadata
when listing containers, and then we also need to handle the case
where the container does not include the acl metadata.

This fixes both of these cases by adding in the metadata request
option for this container listing, and will return a default acl
if not provided in the container metadaata.

Fixes #948
 2024-12-02 16:57:49 -08:00
Ben McClelland
568f8346bf feat: add list-buckets test for no buckets 
We need to check to make sure list-buckets returns a correct
empty list when no buckets created.
 2024-11-18 21:35:19 -08:00
jonaustin09
4d6ec783bf feat: Implements pagination for ListBuckets 2024-10-28 16:26:08 -04:00
jonaustin09
c6359a7050 feat: Refactoring admin APIs: changes i/o data transfer encoding to xml, implements traditional aws-like error handling, adds admin role checker middleware. Refactoring admin CLI actions to handle aws-like error responses 2024-10-25 11:40:23 -04:00
jonaustin09
f03d600b56 fix: The implementation handles null versionId objects for versioning enabled buckets within the versioning directory 2024-10-11 15:39:49 -04:00
Ben McClelland
b7a2e8a2c3 fix: unexpected errors during upload races 
This fixes the cases for racing uploads with the same object names.
Before we were making some bad assumptions about what would cause
an error when trying to link/rename the final object name into
the namespace, but missed the case that another upload for the
same name could be racing with this upload and causing an incorrect
error.

This also changes the order of setting metadata to prevent
accidental setting of metadata for the current upload to another
racing upload.

This also fix auth.CheckObjectAccess() when objects are removed
while this runs.

Fixes #854
 2024-10-07 17:24:44 -07:00
jonaustin09
b39b5e2373 fix: Fixed object versions listing with ListObjectVersions when versioning is not configured for the gateway 2024-10-07 18:16:11 -04:00
Ben McClelland
3a9cbfcbd6 fix: azure tests and add azure ci test 2024-09-23 09:10:52 -07:00
jonaustin09
6d4ff09d6f feat: Added integration tests for bucket object versioning. Made a couple of bug fixes in the versioning implementation 2024-09-19 13:29:14 -07:00
Ben McClelland
d9d3a16051 fix: azure list objects trim common prefixes that match marker prefix 2024-09-18 12:03:21 -07:00
Ben McClelland
180df62134 fix: list objects trim common prefixes that match marker prefix 
This checks to see if the common prefix is before the marker and
thus would have been returned in earlier list objects request.

The error case was aws cli listing multiple entries for the same
common prefix when the listing required multiple pagination
requests.

Fixes #778
 2024-09-18 12:02:25 -07:00
jonaustin09
8a46de8e3b fix: change multipart upload implementation with .sgwtmp namespace in azure 2024-09-10 13:07:15 -07:00
Ben McClelland
ccd4166b2e fix: copy object missing prefix on metadata delete 
When using the REPLACE directive, we were incorrectly removing the
old metadata on the object due to missing the metadata prefix on
the key.  Fix this to remove the correct metadata before setting
new metadata.

Fixes #787
 2024-09-07 17:06:45 -07:00
Jon Austin
d79f978df9 feat: Added the standard storage class to all the available get/list actions responses in posix. (#765) 2024-08-27 15:28:40 -07:00
Ben McClelland
54490f55cc chore: cleanup staticcheck errors 2024-08-13 11:09:14 -07:00
jonaustin09
828eb93bee fix: Added 'Type' property support in bucket ACL Grantee schema 2024-07-16 18:17:16 -04:00
jonaustin09
7545e6236c feat: Implement bucket ownership controls 
Bucket ACLs are now disabled by default the same as AWS.
By default the object ownership is BucketOwnerEnforced
which means that bucket ACLs are disabled. If one attempts
to set bucket ACL the following error is returned both in
the gateway and on AWS:
	ErrAclNotSupported: {
		Code:           "AccessControlListNotSupported",
		Description:    "The bucket does not allow ACLs",
		HTTPStatusCode: http.StatusBadRequest,
	},

ACls can be enabled with PutBucketOwnershipControls

Changed bucket canned ACL translation

New backend interface methods:
PutBucketOwnershipControls
GetBucketOwnershipControls
DeleteBucketOwnershipControls

Added these to metrics
 2024-06-28 21:03:09 -07:00
jonaustin09
e38c63448d fix: Changed GetObject by range success status code from 200 to 206 2024-05-20 14:48:02 -04:00
jonaustin09
5acf1f332a fix: Fixes #555, Added the logic to return InvalidBucketState when calling PutObjectLockConfiguration action on not object lock enabled bucket 2024-05-16 11:30:35 -04:00
Ben McClelland
f722f515ae chore: add missing copyright headers to files 2024-05-06 16:16:31 -07:00
jonaustin09
0c3771ae2d feat: Added GetObjectAttributes actions implementation in posix, azure and s3 backends. Added integration tests for GetObjectAttributes action 2024-04-29 15:31:53 -04:00
jonaustin09
00476ef70c feat: Closes #490, Added integration tests for object lock actions 2024-04-22 13:13:40 -07:00
Ben McClelland
17b1dbe025 fix: return non 0 exit status for cli admin error 
Fixes #505. This returns the body as an error when the http status
for the admin request is non-success.
 2024-04-08 17:29:02 -07:00
jonaustin09
e6203c5765 feat: Closes #441, Added access control integration tests, fixed some bugs in bucket policy and acl access checking flow 2024-03-28 14:52:56 -04:00
jonaustin09
af641e5368 feat: Added integration test cases for Put/Get/DeleteBucketPolicy actions. Made some bug fixes in these actions implementations 2024-03-20 17:31:52 -04:00
Ben McClelland
8ad9c4834b chore: cleanup top level repo by moving test related dirs to tests 2024-03-10 09:15:22 -07:00