mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-01-10 13:27:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,423 Commits 21 Branches 44 Tags
f1c8efdaf633ffb76a5fc167b2f23f36548728ac
Commit Graph

1423 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ben McClelland
f1c8efdaf6 Merge pull request #1156 from versity/sis/putbucketpolicy-invalid-json 
fix: Fixes the bucket policy json document validation errors for inva…
 2025-03-19 15:22:04 -07:00
niksis02
d82da64942 fix: Fixes the bucket policy json document validation errors for invalid jsons. 
Fixes #965

Changes the returned error description to `Policies must be valid JSON and the first byte must be '{'` for invalid bucket policy json documents, which doesn't start with `{`.
The gateway returns `This policy contains invalid Json` error description, if the document starts with `{`, but still isn't valid json.

Implements the `policyErr` string type which implements the `error` interface, to handle the policy json document validation errors, by avoiding staticchecker warnings.
 2025-03-20 01:08:47 +04:00
Ben McClelland
1eb905bf14 Merge pull request #1153 from versity/ben/ldap-err 
fix: add err to ldap user/group conversion errors
 2025-03-19 13:41:47 -07:00
Ben McClelland
9433986121 Merge pull request #1155 from versity/test/chunked_incorrect_middle 
Test/chunked incorrect final signature
 2025-03-19 12:34:33 -07:00
Luke McCrone
75c25ec789 test: chunked upload - more tests, final signature test 2025-03-19 15:36:40 -03:00
Ben McClelland
22cecea1c0 fix: add err to ldap user/group conversion errors 2025-03-19 08:48:01 -07:00
Ben McClelland
39edb0b3b9 Merge pull request #1151 from versity/sis/posix-tag-parsing-refactoring 
fix: Reuses the ParseObjectTags method to parse the object tags in Pu…
v1.0.12 		2025-03-18 13:08:20 -07:00
niksis02
1cea4c8247 fix: Reuses the ParseObjectTags method to parse the object tags in PutObject and CreateMultipartUpload. 
Puts the `backend.ParseObjectTags` method in use to parse the object tagging for `PutObject` and `CreateMultipartUpload` to not repeat the same logic.
 2025-03-18 19:31:52 +04:00
Ben McClelland
4dad42e4e8 Merge pull request #1150 from versity/sis/unnecessary-int-tests 
fix: Removes 2 unnecessary integration tests for HeadObject and GetOb…
 2025-03-18 08:25:16 -07:00
Ben McClelland
0acceaeeab Merge pull request #1148 from versity/sis/object-lock-mode-validation 
fix: Fixes the returned error type for object legal hold status and o…
 2025-03-18 08:24:31 -07:00
niksis02
607c6d2308 fix: Removes 2 unnecessary integration tests for HeadObject and GetObject. 
Removes the `HeadObject_with_contenttype` and `GetObject_with_meta` integration test as the logic is covered in `HeadObject_success` and `GetObject_success` accordingly.
 2025-03-18 18:50:01 +04:00
niksis02
9e0f56f807 fix: Fixes the returned error type for object legal hold status and object lock mode in PutObject, CopyObject and CreateMultipartUpload. 
Fixes #1141
Fixes #1142

Changes the error type to `InvalidArgument` for `x-amz-object-lock-legal-hold` and `x-amz-object-lock-mode` headers invalid values.
 2025-03-18 13:58:49 +04:00
Ben McClelland
c08fd6f064 Merge pull request #1146 from versity/dependabot/go_modules/dev-dependencies-74fe486678 
chore(deps): bump the dev-dependencies group with 3 updates
 2025-03-17 18:02:01 -07:00
Ben McClelland
e635306bfb Merge pull request #1143 from versity/test/rest_signature_invalid_user 
Test/rest signature invalid user
 2025-03-17 18:01:36 -07:00
dependabot[bot]
e8e6356174 chore(deps): bump the dev-dependencies group with 3 updates 
Bumps the dev-dependencies group with 3 updates: [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/service/internal/checksum](https://github.com/aws/aws-sdk-go-v2).


Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.78.1 to 1.78.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.78.1...service/s3/v1.78.2)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.65 to 1.17.66
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.17.65...feature/s3/manager/v1.17.66)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.6.2 to 1.7.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/v1.7.0/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pi/v1.6.2...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-03-17 22:15:49 +00:00
Ben McClelland
28c3b9b95e Merge pull request #1145 from versity/sis/copyobject-missing-props 
feat: Implements object meta properties for CopyObject in azure and p…
 2025-03-17 10:03:44 -07:00
niksis02
cfb2d6d87d feat: Implements object meta properties for CopyObject in azure and posix backends. 
Fixes #998
Closes #1125
Closes #1126
Closes #1127

Implements objects meta properties(Content-Disposition, Content-Language, Content-Encoding, Cache-Control, Expires) and tagging besed on the directives(metadata, tagging) in CopyObject in posix and azure backends. The properties/tagging should be coppied from the source object if "COPY" directive is provided and it should be replaced otherwise.

Changes the object copy principle in azure: instead of using the `CopyFromURL` method from azure sdk, it first loads the object then creates one, to be able to compare and store the meta properties.
 2025-03-17 09:37:05 -07:00
Ben McClelland
38768a92b8 Merge pull request #1139 from versity/ben/scoutfs-updates-from-posix 
feat: sync recent posix changes to scoutfs
 2025-03-17 09:29:38 -07:00
Ben McClelland
092d3b0384 feat: sync recent posix changes to scoutfs 
This syncs recent updates to posix for scoutfs backend including
the extra metadata such as Content-Disposition,
Content-Language, Cache-Control and Expires. This also fixes the
directory object listings that have a double trailing slash due
to the change in the backend.Walk().

This also simplifies head-object to call the posix on and then
post process for glacier changes. This allows keeping in closer
sync with posix head-object over time.
 2025-03-14 15:58:31 -07:00
Luke McCrone
ff305d0efc test: user REST permission, signature tests 2025-03-14 18:45:34 -03:00
Ben McClelland
73b07eb8c5 Merge pull request #1140 from versity/test/expires_header 
Test/expires header
 2025-03-13 16:27:42 -07:00
Luke McCrone
976e44bb57 test: expires header test 2025-03-12 18:16:51 -03:00
Ben McClelland
ff80b90d32 Merge pull request #1138 from versity/sis/putobject-missing-meta 
feat: Adds the Content-Disposition, Content-Language, Cache-Control and Expires object meta properties support in the gateway.
 2025-03-12 08:44:21 -07:00
niksis02
65261a9753 feat: Adds the Content-Disposition, Content-Language, Cache-Control and Expires object meta properties support in the gateway. 
Closes #1128

Adds `Content-Disposition`, `Content-Language`, `Cache-Control` and `Expires` object meta properties support in posix and azure backends.
Changes the `PutObject` and `CreateMultipartUpload` actions backend input type to custom `s3response` types to be able to store `Expires` as any string.
 2025-03-12 16:01:56 +04:00
Ben McClelland
39c8eddadd Merge pull request #1132 from versity/ben/scoutfs-config 
chore: update example config with scoutfs VGW_DISABLE_NOARCHIVE
 2025-03-10 17:38:14 -07:00
Ben McClelland
31e748277f chore: update example config with scoutfs VGW_DISABLE_NOARCHIVE 2025-03-10 16:14:37 -07:00
Ben McClelland
7fd3ca27a7 Merge pull request #1121 from versity/test/rest_checksum_nomode 
Test/rest checksum nomode
v1.0.11 		2025-03-10 15:56:07 -07:00
Luke McCrone
acb33f608e test: checksum mode test 2025-03-10 19:37:22 -03:00
Ben McClelland
1421bc111a Merge pull request #1129 from versity/ben/scoutfs-noarchive 
feat: add noarchive to scoutfs part files
 2025-03-10 15:24:32 -07:00
Ben McClelland
f15f783633 Merge pull request #1131 from versity/dependabot/go_modules/dev-dependencies-c21d9491cd 
chore(deps): bump the dev-dependencies group with 14 updates
 2025-03-10 15:24:15 -07:00
dependabot[bot]
b20b4362d7 chore(deps): bump the dev-dependencies group with 14 updates 
Bumps the dev-dependencies group with 14 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.78.0` | `1.78.1` |
| [github.com/urfave/cli/v2](https://github.com/urfave/cli) | `2.27.5` | `2.27.6` |
| [golang.org/x/sync](https://github.com/golang/sync) | `0.11.0` | `0.12.0` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.30.0` | `0.31.0` |
| [github.com/aws/aws-sdk-go-v2/service/sso](https://github.com/aws/aws-sdk-go-v2) | `1.25.0` | `1.25.1` |
| [github.com/aws/aws-sdk-go-v2/service/ssooidc](https://github.com/aws/aws-sdk-go-v2) | `1.29.0` | `1.29.1` |
| [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) | `1.33.16` | `1.33.17` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.35.0` | `0.36.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.35.0` | `0.37.0` |
| [golang.org/x/text](https://github.com/golang/text) | `0.22.0` | `0.23.0` |
| [golang.org/x/time](https://github.com/golang/time) | `0.10.0` | `0.11.0` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.29.8` | `1.29.9` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.17.61` | `1.17.62` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.17.64` | `1.17.65` |


Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.78.0 to 1.78.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.78.0...service/s3/v1.78.1)

Updates `github.com/urfave/cli/v2` from 2.27.5 to 2.27.6
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.27.5...v2.27.6)

Updates `golang.org/x/sync` from 0.11.0 to 0.12.0
- [Commits](https://github.com/golang/sync/compare/v0.11.0...v0.12.0)

Updates `golang.org/x/sys` from 0.30.0 to 0.31.0
- [Commits](https://github.com/golang/sys/compare/v0.30.0...v0.31.0)

Updates `github.com/aws/aws-sdk-go-v2/service/sso` from 1.25.0 to 1.25.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.25.0...v1.25.1)

Updates `github.com/aws/aws-sdk-go-v2/service/ssooidc` from 1.29.0 to 1.29.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.29.0...config/v1.29.1)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.33.16 to 1.33.17
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/sns/v1.33.16...service/sns/v1.33.17)

Updates `golang.org/x/crypto` from 0.35.0 to 0.36.0
- [Commits](https://github.com/golang/crypto/compare/v0.35.0...v0.36.0)

Updates `golang.org/x/net` from 0.35.0 to 0.37.0
- [Commits](https://github.com/golang/net/compare/v0.35.0...v0.37.0)

Updates `golang.org/x/text` from 0.22.0 to 0.23.0
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.22.0...v0.23.0)

Updates `golang.org/x/time` from 0.10.0 to 0.11.0
- [Commits](https://github.com/golang/time/compare/v0.10.0...v0.11.0)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.29.8 to 1.29.9
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.29.8...config/v1.29.9)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.17.61 to 1.17.62
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.61...credentials/v1.17.62)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.64 to 1.17.65
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.17.64...feature/s3/manager/v1.17.65)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sso
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ssooidc
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/net
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/text
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/time
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-03-10 22:00:42 +00:00
Ben McClelland
55999ec94a Merge pull request #1115 from versity/test/rest_checksum_crc32c 
Test/rest checksum crc32c
 2025-03-10 14:55:06 -07:00
Ben McClelland
d034f87f60 feat: add noarchive to scoutfs part files 
The part files for multipart uploads are considered temporary
files and should not be archived by default. This adds the
noarchive attribute to the part files to prevent scoutam from
trying to archive these.

There is a new parameter, disablenoarchive, that will prevent
adding the noarchive attribute to these files for the case
where there is a desire to archive these temp files.
 2025-03-10 14:52:20 -07:00
Ben McClelland
c3c201d5b6 Merge pull request #1130 from versity/ben/scoutfs-mp-cleanup 
fix: scoutfs multipart cleanup in complete/abort mp
 2025-03-10 14:46:54 -07:00
Ben McClelland
f77058b817 fix: scoutfs multipart cleanup in complete/abort mp 
This was previously not including the bucket directory for the
mutlipart temp file cleanup. This fixes leftovers in the tmp
directories after uploading multipart uploads.
 2025-03-10 13:44:24 -07:00
Luke McCrone
485f2abbaa test: crc32c checksum 2025-03-08 18:56:42 -03:00
Luke McCrone
4c5cd918d8 test: crc64nvme checksum 2025-03-08 17:56:14 -03:00
Ben McClelland
293e7faf31 Merge pull request #1124 from versity/sis/getobject-exceeding-range 
fix: Fixes GetObject and UploadPartCopy actions data range parsing.
 2025-03-07 15:45:15 -08:00
Ben McClelland
d6d4f304e1 Merge pull request #1096 from versity/test/rest_checksum_crc64nvme 
Test/rest checksum crc64nvme
 2025-03-07 15:43:52 -08:00
niksis02
96af2b6471 fix: Fixes GetObject and UploadPartCopy actions data range parsing. 
Fixes #1004
Fixes #1122
Fixes #1120

Separates `GetObject` and `UploadPartCopy` range parsing/validation.

`GetObject` returns a successful response if acceptRange is invalid.
Adjusts the range upper limit, if it exceeds the actual objects size for `GetObject`.
Corrects the `ContentRange` in the `GetObject` response.

Fixes the `UploadPartCopy` action copy source range parsing/validation.
`UploadPartCopy` returns `InvalidArgument` if the copy source range is not valid.
 2025-03-08 01:39:21 +04:00
Ben McClelland
ca1697f1f5 Merge pull request #1119 from versity/fix/getobjectattributes-etag-quotes 
fix: removes ETag quotes for in GetObjectAttributes response for azur…
 2025-03-05 13:16:01 -08:00
niksis02
4b0dd64836 fix: removes ETag quotes for in GetObjectAttributes response for azure and posix backends 2025-03-06 00:28:55 +04:00
Ben McClelland
12bef32d70 Merge pull request #1118 from versity/fix/checksum-headers-validation 
fix: Adds validation for x-amz-checksum- headers. Makes x-amz-sdk-che…
 2025-03-05 11:53:15 -08:00
Luke McCrone
11f646b051 test: crc64nvme checksum 2025-03-05 16:22:34 -03:00
niksis02
7d6505ec06 fix: Adds validation for x-amz-checksum- headers. Makes x-amz-sdk-checksum-algorithm header case insensitive 2025-03-05 22:06:20 +04:00
Ben McClelland
59c392028f Merge pull request #1117 from versity/ben/readme 
feat: add Azure/S3 to supported backends in README
 2025-03-05 10:00:09 -08:00
Ben McClelland
6361d7780b Merge pull request #1103 from versity/ben/read-only-fs 
feat: return method not allowed when uploading to read only fs
 2025-03-05 09:59:51 -08:00
Luke McCrone
dea316bb26 test: initial checksum tests, REST script additions 2025-03-05 14:53:44 -03:00
Ben McClelland
c6fcb8ea53 feat: add Azure/S3 to supported backends in README 2025-03-05 09:27:33 -08:00
Ben McClelland
7b784d257f Merge pull request #1113 from versity/fix/listobjects-dir-objs 
fix: Fixes ListObjects direcotry objects listing issue. Fixes DeleteO…
 2025-03-04 11:28:06 -08:00