mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-01-09 13:03:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,555 Commits 23 Branches 44 Tags
d3585e6c1c2d1698c79746f82e2ca334a2cd8f75
Commit Graph

1555 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
niksis02
d3585e6c1c feat: optimizes backend.Walk and backend.WalkVersions to avoid sorting the common prefixes. 
Common prefixes were originally stored in a `map[string]struct{}`, which was then converted to a slice and sorted. The new implementation stores the common prefixes in a `map[string]int`, where the map value represents the index of the common prefix. There's no need to sort the common prefixes array, as `fs.WalkDir` comes with sorted directories and files.
 2025-05-10 01:59:39 +04:00
Ben McClelland
42b03b866c Merge pull request #1278 from versity/sis/etag-quotes 
fix: adds the surrounding quotes on ETag in PutObject for dir objects and in UploadPartCopy.
 2025-05-08 14:40:39 -07:00
niksis02
3740d79173 fix: adds the surrounding quotes on ETag in PutObject for dir objects and in UploadPartCopy. 
Fixes #1277
Fixes #1235

Adds surrounding quotes on `ETag` when creating a directory object. Adds the quotes in `UploadPartCopy` as well.
 2025-05-09 00:29:23 +04:00
Ben McClelland
f4577d4af5 Merge pull request #1274 from versity/sis/versioning-getobject-success-test-fix 
fix: fixes the context cancelation issue in Versioning_GetObject_success integration test.
 2025-05-08 10:05:15 -07:00
niksis02
809d969afb fix: fixes the context cancelation issue in Versioning_GetObject_success integration test. 
Fixes #1271

In the `Versioning_GetObject_success` integration test the contexts are canceled before reading the full request body after `GetObject`.
Changes the behaviour to defer the context cancelation, to be sure it's canceled after the full request body is read.
 2025-05-08 20:34:18 +04:00
Ben McClelland
3a9f8c6525 Merge pull request #1272 from versity/sis/debug-logging-chunk-readers 
feat: adds debug logging for chunk readers.
 2025-05-07 13:55:01 -07:00
niksis02
23b5e60854 feat: adds debug logging for chunk readers. 
Closes #1221

Adds debug logging for `signed`/`unsigned` chunk readers.
Adds the `debuglogger.Infof` log method, which prints out green info logs with `[INFO]:` prefix.
The debug logging inclues some chunk details: size, signature, trailers. It also prints out stash/release stash operations.
The error cases are logged with standart yellow `[DEBUG]:` prefix.
The `String to sign` block in signed chunk reader is logged in purple horizontal borders with title.
 2025-05-08 00:22:01 +04:00
Ben McClelland
2d5d641824 Merge pull request #1270 from versity/ben/event-log-panic 
fix: panic with malformed request in event/log handlers
 2025-05-07 11:13:44 -07:00
Ben McClelland
4478ed1143 fix: panic with malformed request in event/log handlers 
Sending the following malformed request with eevnt notifcations
or access logs enabled will cause a panic related to parsing the
bucket and object from the invalid request path:

printf "GET GET  HTTP/1.1\r\nHost: $HOST\r\n\r\n" | nc 127.0.0.1 7070

The fix is to add bounds checks on the slice returned from
splitting the request path to set the bucket/object.

Fixes #1269
 2025-05-06 17:42:05 -07:00
Ben McClelland
22703de0c8 Merge pull request #1267 from versity/ben/controller-bounds-check 
fix: add bounds check for ContentLength type conversion
 2025-05-06 08:27:38 -07:00
Ben McClelland
5122b8c6ed Merge pull request #1268 from sebastian-heinz/use-path-style 
use path style
 2025-05-06 08:19:49 -07:00
sebastian-heinz
42013d365b use path style 2025-05-06 10:28:16 +08:00
Ben McClelland
a77c24f61f Merge pull request #1266 from versity/dependabot/go_modules/dev-dependencies-de083807b3 
chore(deps): bump the dev-dependencies group with 7 updates
 2025-05-05 16:44:48 -07:00
Ben McClelland
e7294c631f fix: add bounds check for ContentLength type conversion 
On 32-bit systems, this value could overflow. Add a check for the
overflow and return ErrInvalidRange if it does overflow.

The type in GetObjectOutput for ContentLength is *int64, but the
fasthttp.RequestCtx.SetBodyStream() takes type int. So there is
no way to set the bodysize to the correct limit if the value
overflows.
 2025-05-05 16:36:29 -07:00
dependabot[bot]
c3334008f5 chore(deps): bump the dev-dependencies group with 7 updates 
Bumps the dev-dependencies group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) | `1.41.2` | `1.42.0` |
| [golang.org/x/sync](https://github.com/golang/sync) | `0.13.0` | `0.14.0` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.32.0` | `0.33.0` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.37.0` | `0.38.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.39.0` | `0.40.0` |
| [golang.org/x/text](https://github.com/golang/text) | `0.24.0` | `0.25.0` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.17.74` | `1.17.75` |


Updates `github.com/nats-io/nats.go` from 1.41.2 to 1.42.0
- [Release notes](https://github.com/nats-io/nats.go/releases)
- [Commits](https://github.com/nats-io/nats.go/compare/v1.41.2...v1.42.0)

Updates `golang.org/x/sync` from 0.13.0 to 0.14.0
- [Commits](https://github.com/golang/sync/compare/v0.13.0...v0.14.0)

Updates `golang.org/x/sys` from 0.32.0 to 0.33.0
- [Commits](https://github.com/golang/sys/compare/v0.32.0...v0.33.0)

Updates `golang.org/x/crypto` from 0.37.0 to 0.38.0
- [Commits](https://github.com/golang/crypto/compare/v0.37.0...v0.38.0)

Updates `golang.org/x/net` from 0.39.0 to 0.40.0
- [Commits](https://github.com/golang/net/compare/v0.39.0...v0.40.0)

Updates `golang.org/x/text` from 0.24.0 to 0.25.0
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.24.0...v0.25.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.74 to 1.17.75
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.17.74...feature/s3/manager/v1.17.75)

---
updated-dependencies:
- dependency-name: github.com/nats-io/nats.go
  dependency-version: 1.42.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sys
  dependency-version: 0.33.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/crypto
  dependency-version: 0.38.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.40.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/text
  dependency-version: 0.25.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-version: 1.17.75
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-05-05 22:51:13 +00:00
Ben McClelland
b12b0d242e Merge pull request #1264 from versity/test/copy_object_param 2025-05-05 15:09:24 -07:00
Luke McCrone
384bb463d3 test: copy object tests (copy source, payload) 2025-05-05 17:25:56 -03:00
Ben McClelland
4b34ef1a5f Merge pull request #1263 from versity/sis/headobject-range 
fix: fixes the range parsing for GetObject. Adds range query support for HeadObject.
 2025-05-05 12:23:15 -07:00
Ben McClelland
e0999ce5a3 Merge pull request #1261 from versity/ben/scoutfs-fixes 
Ben/scoutfs fixes
 2025-05-05 11:42:36 -07:00
niksis02
dfa1ed2358 fix: fixes the range parsing for GetObject. Adds range query support for HeadObject. 
Fixes #1258
Fixes #1257
Closes #1244

Adds range queries support for `HeadObject`.
Fixes the range parsing logic for `GetObject`, which is used for `HeadObject` as well. Both actions follow the same rules for range parsing.

Fixes the error message returned by `GetObject`.
 2025-05-05 22:41:12 +04:00
Ben McClelland
98867bc731 Merge pull request #1262 from versity/ben/iam-internal 
fix: use createtemp()/rename() for iam internal files
 2025-05-05 08:12:27 -07:00
Ben McClelland
e98f7763d0 fix: use createtemp()/rename() for iam internal files 
This cleans up a previous fix to #630 to use a better temp/rename
scheme thats less likely to have bad side effects.

The test for the previous issue still passes these cases, and we
will be less liekly to find a case where the file doesnt exist
or corrpted backup files.
 2025-05-03 12:39:06 -07:00
Ben McClelland
e9286f7a23 feat: add scoutfs group tests to integration 2025-05-03 12:04:47 -07:00
Ben McClelland
a60d6a7faa fix: scoutfs racing mutlipart uploads internal error 
When multiple uploads with the same object key are racing, we can
end up with an EEXIST when trying to link the final object into
the namespace. When this happens, we should just remove the
existing file and try again since the semantics are that the
last upload should win.
 2025-05-03 09:30:45 -07:00
Ben McClelland
a29f7b1839 fix: scoutfs missing ListObjectsV2() start after 
This brings ListObjectsV2 for scoutfs in sync with posix to handle
the start after and continuation token ases.
 2025-05-03 09:15:01 -07:00
Ben McClelland
6321406008 fix: scoutfs missing ListObjects() response fields 
This fixes some tests that were fialing due to missing response
fields in ListObjects().
 2025-05-03 09:07:56 -07:00
Ben McClelland
cd9cb108a3 Merge pull request #1260 from versity/ben/debug-log 
feat: cleanup calling of debuglogger with managed debug setting
 2025-05-02 18:26:01 -07:00
Ben McClelland
78910fb556 Merge pull request #1259 from versity/ben/test-fixes 
fix: cleanup test cases that could lead to panic with invalid response
 2025-05-02 18:25:48 -07:00
Ben McClelland
a9fcf63063 feat: cleanup calling of debuglogger with managed debug setting 2025-05-02 17:05:59 -07:00
Ben McClelland
1ef81d985e fix: cleanup test cases that could lead to panic with invalid response 2025-05-02 16:44:10 -07:00
Ben McClelland
d19c446f72 Merge pull request #1256 from versity/ben/goreleaser-config-updates 
chore: update goreleaser configs
 2025-05-02 12:03:22 -07:00
Ben McClelland
2e7a7fcbe9 Merge pull request #1255 from versity/ben/fix-scoutfs-mp-etag-check 
fix: scoutfs etag check for multipart uploads
 2025-05-02 12:03:08 -07:00
Ben McClelland
c45b32066f chore: update goreleaser configs 
This cleans up deprecated config options, and sets the github
job to use the newer goreleaser v2.

Fixes #682
 2025-05-02 10:37:26 -07:00
Ben McClelland
9f13b544f7 fix: scoutfs etag check for multipart uploads 
The Etag can be quoted or not, so the check to verify the part
Etag must remove the quotes before checking for equality. This
check is the same now as posix.
 2025-05-02 10:07:47 -07:00
Ben McClelland
1f96af5c66 Merge pull request #1254 from versity/sis/duplicate-xmlns-responses 
fix: removes the xml pretty printing from debug logger.
 2025-05-01 12:37:25 -07:00
Ben McClelland
ddceb28f98 Merge pull request #1252 from versity/ben/mp-complete-xml-response 
fix: xml response field names for complete multipart upload
 2025-05-01 12:36:00 -07:00
niksis02
c497baa733 fix: removes the xml pretty printing from debug logger. 
Fixes #1253

Removes the xml pretty printing from debug logger. Instead it prints out the raw request/response body. This way we avoid to miss/add something to raw xml, which could lead to misconfusion.
 2025-05-01 22:56:21 +04:00
Ben McClelland
9244e9100d fix: xml response field names for complete multipart upload 
The xml encoding for the s3.CompleteMultipartUploadOutput response
type was not producing exactly the right field names for the
expected complete multipart upload result.

This change follows the pattern we have had to do for other xml
responses to create our own type that will encode better to the
expected response.

This will change the backend.Backend interface, so plugins and
other backends will have to make the corresponding changes.
 2025-04-30 14:36:48 -07:00
Ben McClelland
4eba4e031c Merge pull request #1251 from versity/sis/uploadpart-etag-quotes 
fix: adds quotes to part Etag in UploadPart
 2025-04-30 14:35:34 -07:00
niksis02
32faf9a4c3 fix: adds quotes to part Etag in UploadPart 
Fixes #1233

Add double quotes to the `ETag` in `UploadPart`.
 2025-04-30 23:26:18 +04:00
Ben McClelland
a4d2f5c180 Merge pull request #1247 from ttschampel/feature/s3proxy_with_client 
Add support for supplying s3.Client instance to S3 Proxy
 2025-04-30 11:28:37 -07:00
Ben McClelland
24fbbdbd63 Merge pull request #1250 from versity/sis/obj-upload-max-limit 
fix: Adds validation for Content-Length in upload operations.
 2025-04-30 09:27:48 -07:00
niksis02
2b1e1af89b fix: Adds validation for Content-Length in upload operations. 
Fixes #961
Fixes #1248

The gateway should return a `MissingContentLength` error if the `Content-Length` HTTP header is missing for upload operations (`PutObject`, `UploadPart`).

The second fix involves enforcing a maximum object size limit of `5 * 1024 * 1024 * 1024` bytes (5 GB) by validating the value of the `Content-Length` header. If the value exceeds this limit, the gateway should return an `EntityTooLarge` error.
 2025-04-30 14:20:28 +04:00
Timothy Tschampel
dea4b6382f add additional constructor with s3.Client instance 2025-04-29 09:10:54 -07:00
Ben McClelland
8c101b3901 Merge pull request #1246 from versity/dependabot/go_modules/dev-dependencies-e40766069b 
chore(deps): bump the dev-dependencies group with 6 updates
 2025-04-28 16:07:27 -07:00
dependabot[bot]
7f9b9dfd97 chore(deps): bump the dev-dependencies group with 6 updates 
Bumps the dev-dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) | `1.6.0` | `1.6.1` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.79.2` | `1.79.3` |
| [github.com/valyala/fasthttp](https://github.com/valyala/fasthttp) | `1.60.0` | `1.61.0` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.17.72` | `1.17.74` |
| [github.com/aws/aws-sdk-go-v2/service/internal/checksum](https://github.com/aws/aws-sdk-go-v2) | `1.7.0` | `1.7.1` |
| [github.com/cpuguy83/go-md2man/v2](https://github.com/cpuguy83/go-md2man) | `2.0.6` | `2.0.7` |


Updates `github.com/Azure/azure-sdk-for-go/sdk/storage/azblob` from 1.6.0 to 1.6.1
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.0...sdk/azcore/v1.6.1)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.79.2 to 1.79.3
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.79.2...service/s3/v1.79.3)

Updates `github.com/valyala/fasthttp` from 1.60.0 to 1.61.0
- [Release notes](https://github.com/valyala/fasthttp/releases)
- [Commits](https://github.com/valyala/fasthttp/compare/v1.60.0...v1.61.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.72 to 1.17.74
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.17.72...feature/s3/manager/v1.17.74)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.7.0 to 1.7.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.7.0...v1.7.1)

Updates `github.com/cpuguy83/go-md2man/v2` from 2.0.6 to 2.0.7
- [Release notes](https://github.com/cpuguy83/go-md2man/releases)
- [Commits](https://github.com/cpuguy83/go-md2man/compare/v2.0.6...v2.0.7)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.6.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.79.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/valyala/fasthttp
  dependency-version: 1.61.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-version: 1.17.74
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-version: 1.7.1
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/cpuguy83/go-md2man/v2
  dependency-version: 2.0.7
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-04-28 22:24:27 +00:00
Ben McClelland
224ab5111f Merge pull request #1245 from versity/ben/actions_permissions 
chore: add token permissions to githubb actions
 2025-04-28 13:55:04 -07:00
Ben McClelland
b69352bdd6 chore: add token permissions to githubb actions 2025-04-28 13:26:17 -07:00
Ben McClelland
aecea5f068 Merge pull request #1243 from versity/sis/tagging-url-encoding 
fix: fixes tagging string parsing for PutObject, CopyObject and CreateMultipartUpload
 2025-04-28 12:34:46 -07:00
niksis02
5e6056467e fix: fixes tagging string parsing for PutObject, CopyObject and CreateMultipartUpload 
Fixes #1215
Fixes #1216

`PutObject`, `CopyObject` and `CreateMultipartUpload` accept tag string as an http request header which should be url-encoded. The tag string should be a valid url-encoded string and each key/value pair should be valid, otherwise they should fail with `APIError`.

If the provided tag set contains duplicate `keys` the calls should fail with the same `InvalidURLEncodedTagging` error.

Not all url-encoded characters are supported by `S3`. The tagging string should contain only `letters`, `digits` and the following special chars:
- `-`
- `.`
- `/`
- `_`
- `+`
- ` `(space)

And their url-encoded versions: e.g. `%2F`(/), `%2E`(.) ... .

If the provided tagging string contains invalid `key`/`value`, the calls should fail with the following errors respectively:
`invalid key` - `(InvalidTag) The TagKey you have provided is invalid`
`invalid value` - `(InvalidTag) The TagValue you have provided is invalid`
 2025-04-28 20:28:20 +04:00