mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-01-08 20:43:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,087 Commits 21 Branches 44 Tags
ben/plugfest
Commit Graph

1087 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ben McClelland
7086579590 fix: list objects trim common prefixes that match marker prefix 
This checks to see if the common prefix is before the marker and
thus would have been returned in earlier list objects request.

The error case was aws cli listing multiple entries for the same
common prefix when the listing required multiple pagination
requests.

Fixes #778
 2024-09-16 14:55:55 -07:00
jonaustin09
dea5e0c0b2 feat: Implemented object versioning for multipart uploads. Implemented integration tests for the versioning implementation for multipart uploads 2024-09-16 14:51:22 -07:00
jonaustin09
16995acc17 feat: Added integration tests for bucket object versioning. Made a couple of bug fixes in the versioning implementation 2024-09-16 14:51:20 -07:00
jonaustin09
1f41f91f2d feat: basic logic implementation of bucket object versioning in posix backend 
New posix backend option --versioning-dir will enable storing object versions
in specified directory.
 2024-09-16 14:42:24 -07:00
Ben McClelland
21a51380f9 Merge pull request #805 from versity/ben/direct_tests 
fix: separate out direct tests dockerfile for easier use
 2024-09-13 16:27:09 -07:00
Ben McClelland
590295c3d1 test: separate out direct tests dockerfile for easier use 
The direct tests don't need the gateway or any go compilers.
This separates out the dockerfile to minimize it a bit more
from the other tests.
 2024-09-13 15:47:51 -07:00
Ben McClelland
f43040d1da Merge pull request #804 from versity/static_setup_fix 
test: static bucket init fix
 2024-09-13 15:47:32 -07:00
Luke McCrone
ae1c566656 test: static bucket init fix, github-actions enhancements 2024-09-13 19:11:00 -03:00
Ben McClelland
53a7abf82f Merge pull request #802 from versity/ben/utc_time 
fix: time locale utc in responses
 2024-09-13 08:34:06 -07:00
Ben McClelland
9dbfaeed0c fix: time locale utc in responses 
Change all response times to UTC. Fix bucket create date time
formatting. This was using the incorrct xml field name before.
 2024-09-12 19:17:46 -07:00
Ben McClelland
080fd0136c Merge pull request #801 from versity/ben/docker_cleanup 
chore: move test docker files to tests dir
 2024-09-12 16:50:02 -07:00
Ben McClelland
220819444f chore: move test docker files to tests dir 2024-09-12 16:15:33 -07:00
Ben McClelland
cf6c1b97d1 Merge pull request #800 from versity/ben/tests_direct 
fix: get direct tests working against a non-versitygw endpoint
 2024-09-12 16:08:23 -07:00
Ben McClelland
d50027419e fix: get direct tests working against a non-versitygw endpoint 2024-09-12 11:39:31 -07:00
Ben McClelland
2d9a7cc019 Merge pull request #794 from versity/test_cmdline_xml_retrieval 
Test cmdline xml retrieval
 2024-09-11 15:25:58 -07:00
Ben McClelland
de67b1d718 Merge pull request #799 from versity/ben/dashboard 
chore: move dashboard to extra
 2024-09-11 15:25:37 -07:00
Ben McClelland
22a958bcc4 chore: move dashboard to extra 2024-09-11 13:50:27 -07:00
Ben McClelland
cecf563d92 Merge pull request #798 from versity/fix/azure-obj-content-enc 
fix: Fixed the mechanism to put/get object content-type, content-enco…
 2024-09-11 13:30:39 -07:00
jonaustin09
47d1a799f6 fix: Fixed the mechanism to put/get object content-type, content-encoding in azure 2024-09-11 15:11:37 -04:00
Luke McCrone
4ed54d9bd9 test: initial rests tests, run/assert, docker cleanup 2024-09-11 15:13:13 -03:00
Ben McClelland
7127cdeee5 Merge pull request #797 from versity/fix/getobjectattributes-mp 
fix: Removed multipart upload part from GetObjectAttributes action
 2024-09-11 10:34:46 -07:00
jonaustin09
a6fd1322f7 fix: Removed multipart upload part from GetObjectAttributes action 2024-09-11 11:57:32 -04:00
Ben McClelland
5b6f806829 Merge pull request #795 from versity/ben/update_actions 
chore: update docker-bats github actions
 2024-09-10 18:54:43 -07:00
Ben McClelland
90fb90d9a5 chore: update docker-bats github actions 2024-09-10 15:59:30 -07:00
Ben McClelland
f742a40ac2 Merge pull request #792 from versity/fix/azure-integration-tests-coverage 
Azure integration test issues
 2024-09-10 13:32:16 -07:00
jonaustin09
8a46de8e3b fix: change multipart upload implementation with .sgwtmp namespace in azure 2024-09-10 13:07:15 -07:00
Ben McClelland
448765ba04 Merge pull request #786 from versity/ben/content_type 
fix: set content type on put object
 2024-09-10 11:42:01 -07:00
Ben McClelland
dc71365bab fix: set content type/encoding on put/mutlipart object 
This fixes put object with setting a content type. If no content
type is set, then we will return a default content type for
following requests. Mutli-part upload appears to be ok.

Also fixed content eincoding and multipart uploads.

Fixes #783
 2024-09-09 16:04:54 -07:00
Ben McClelland
6ad1e25c2b Merge pull request #791 from versity/dependabot/go_modules/dev-dependencies-d5f69bb6e6 
chore(deps): bump the dev-dependencies group with 19 updates
 2024-09-09 15:36:32 -07:00
dependabot[bot]
cae5535556 chore(deps): bump the dev-dependencies group with 19 updates 
Bumps the dev-dependencies group with 19 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.30.4` | `1.30.5` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.61.0` | `1.61.2` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.24.0` | `0.25.0` |
| [github.com/aws/aws-sdk-go-v2/feature/ec2/imds](https://github.com/aws/aws-sdk-go-v2) | `1.16.12` | `1.16.13` |
| [github.com/aws/aws-sdk-go-v2/service/sso](https://github.com/aws/aws-sdk-go-v2) | `1.22.5` | `1.22.7` |
| [github.com/aws/aws-sdk-go-v2/service/ssooidc](https://github.com/aws/aws-sdk-go-v2) | `1.26.5` | `1.26.7` |
| [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) | `1.30.5` | `1.30.7` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.26.0` | `0.27.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.28.0` | `0.29.0` |
| [golang.org/x/text](https://github.com/golang/text) | `0.17.0` | `0.18.0` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.27.31` | `1.27.33` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.17.30` | `1.17.32` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.17.16` | `1.17.18` |
| [github.com/aws/aws-sdk-go-v2/internal/configsources](https://github.com/aws/aws-sdk-go-v2) | `1.3.16` | `1.3.17` |
| [github.com/aws/aws-sdk-go-v2/internal/endpoints/v2](https://github.com/aws/aws-sdk-go-v2) | `2.6.16` | `2.6.17` |
| [github.com/aws/aws-sdk-go-v2/internal/v4a](https://github.com/aws/aws-sdk-go-v2) | `1.3.16` | `1.3.17` |
| [github.com/aws/aws-sdk-go-v2/service/internal/checksum](https://github.com/aws/aws-sdk-go-v2) | `1.3.18` | `1.3.19` |
| [github.com/aws/aws-sdk-go-v2/service/internal/presigned-url](https://github.com/aws/aws-sdk-go-v2) | `1.11.18` | `1.11.19` |
| [github.com/aws/aws-sdk-go-v2/service/internal/s3shared](https://github.com/aws/aws-sdk-go-v2) | `1.17.16` | `1.17.17` |


Updates `github.com/aws/aws-sdk-go-v2` from 1.30.4 to 1.30.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.30.4...v1.30.5)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.61.0 to 1.61.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.61.0...service/s3/v1.61.2)

Updates `golang.org/x/sys` from 0.24.0 to 0.25.0
- [Commits](https://github.com/golang/sys/compare/v0.24.0...v0.25.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.16.12 to 1.16.13
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.16.12...v1.16.13)

Updates `github.com/aws/aws-sdk-go-v2/service/sso` from 1.22.5 to 1.22.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pi/v1.22.5...service/mq/v1.22.7)

Updates `github.com/aws/aws-sdk-go-v2/service/ssooidc` from 1.26.5 to 1.26.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.26.5...service/s3/v1.26.7)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.30.5 to 1.30.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.30.5...service/sts/v1.30.7)

Updates `golang.org/x/crypto` from 0.26.0 to 0.27.0
- [Commits](https://github.com/golang/crypto/compare/v0.26.0...v0.27.0)

Updates `golang.org/x/net` from 0.28.0 to 0.29.0
- [Commits](https://github.com/golang/net/compare/v0.28.0...v0.29.0)

Updates `golang.org/x/text` from 0.17.0 to 0.18.0
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.17.0...v0.18.0)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.27.31 to 1.27.33
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.31...config/v1.27.33)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.17.30 to 1.17.32
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.30...credentials/v1.17.32)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.16 to 1.17.18
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.16...credentials/v1.17.18)

Updates `github.com/aws/aws-sdk-go-v2/internal/configsources` from 1.3.16 to 1.3.17
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/ini/v1.3.16...internal/ini/v1.3.17)

Updates `github.com/aws/aws-sdk-go-v2/internal/endpoints/v2` from 2.6.16 to 2.6.17
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/endpoints/v2.6.16...internal/endpoints/v2.6.17)

Updates `github.com/aws/aws-sdk-go-v2/internal/v4a` from 1.3.16 to 1.3.17
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/ini/v1.3.16...internal/ini/v1.3.17)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.3.18 to 1.3.19
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/ini/v1.3.18...internal/ini/v1.3.19)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/presigned-url` from 1.11.18 to 1.11.19
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/service/dax/v1.11.19/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/dax/v1.11.18...service/dax/v1.11.19)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/s3shared` from 1.17.16 to 1.17.17
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.16...credentials/v1.17.17)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sso
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ssooidc
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/net
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/text
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/configsources
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/v4a
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/presigned-url
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/s3shared
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-09 22:07:38 +00:00
Ben McClelland
bdc8324242 Merge pull request #790 from versity/ben/response_time_format 
fix: format timestamp responses with RFC3339TimeFormat
 2024-09-09 08:16:37 -07:00
Ben McClelland
e4bc3d51e5 Merge pull request #789 from versity/ben/copy_obj_meta 
fix: copy object missing prefix on metadata delete
 2024-09-09 08:16:24 -07:00
Ben McClelland
ccd4166b2e fix: copy object missing prefix on metadata delete 
When using the REPLACE directive, we were incorrectly removing the
old metadata on the object due to missing the metadata prefix on
the key.  Fix this to remove the correct metadata before setting
new metadata.

Fixes #787
 2024-09-07 17:06:45 -07:00
Ben McClelland
3bf8b296d8 fix: format timestamp responses with RFC3339TimeFormat 
This fixes the rest of the s3reponse types to marshall the time
field in RFC3339TimeFormat when MarshalXML called.

Fixes #782
 2024-09-07 12:29:09 -07:00
Ben McClelland
66a7879b0a Merge pull request #785 from digitalglue-software/fix/delete-object-remove-parents 
fix: DeleteObject not working as expected when using `--bucketlinks` (#784)
 2024-09-06 15:12:30 -07:00
Kyle Upton
5321095de5 fix bucket dir detection 2024-09-05 16:19:07 +00:00
Ben McClelland
1adf3d9565 Merge pull request #777 from versity/test_cmdline_file_compares 
test: add better file comparisons
 2024-09-04 14:53:07 -07:00
Luke McCrone
2823676aa2 test: improve file data comparisons for testing 2024-09-04 14:24:01 -03:00
Ben McClelland
ddcc62ae0a Merge pull request #780 from versity/dependabot/go_modules/dev-dependencies-355b76d293 
chore(deps): bump the dev-dependencies group with 2 updates
 2024-09-02 15:43:39 -07:00
dependabot[bot]
151326b5d7 chore(deps): bump the dev-dependencies group with 2 updates 
Bumps the dev-dependencies group with 2 updates: [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2).


Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.60.1 to 1.61.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.60.1...service/s3/v1.61.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.15 to 1.17.16
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.15...credentials/v1.17.16)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-09-02 21:33:24 +00:00
Ben McClelland
9cfc2c7b08 Merge pull request #776 from versity/copy-object-with-starting-slash 
feat: Added an integration test to cover the case to CopyObject with …
v1.0.6 		2024-08-29 12:58:57 -07:00
jonaustin09
9be4f27550 feat: Added an integration test to cover the case to CopyObject with the copysource starting with / 2024-08-29 15:32:20 -04:00
Ben McClelland
cdb5187ca2 Merge pull request #774 from cvubrugier/fix-773-copy-source-slash 
fix: handle "x-amz-copy-source" header starting with '/' in s3api
 2024-08-29 09:10:12 -07:00
Ben McClelland
c2f9e801ef Merge pull request #775 from versity/fix/list-objects-next-marker 
fix: Added an integration test case for ListObjectsV2 to specify max-…
 2024-08-29 08:56:45 -07:00
jonaustin09
201777c819 fix: Added an integration test case for ListObjectsV2 to specify max-keys as the exact number of objects in the bucket 2024-08-29 11:14:29 -04:00
Christophe Vu-Brugier
20940f0b46 fix: handle "x-amz-copy-source" header starting with '/' in s3api 
The "x-amz-copy-source" header may start with '/' as observed with
WinSCP. However, '/' is also the separator between the bucket and the
object path in "x-amz-copy-source".

Consider the following code in VerifyObjectCopyAccess():

    srcBucket, srcObject, found := strings.Cut(copySource, "/")

If `copySource` starts with '/', then `srcBucket` is set to an empty
string. Later, an error is returned because bucket "" does not exist.

This issue was fixed in the Posix and Azure backends by the following
commit:

 * 5e484f2 fix: Fixed CopySource parsing to handle the values starting with '/' in CopyObject action in posix and azure backends.

But the issue was not fixed in `VerifyObjectCopyAccess`.

This commit sanitizes "x-amz-copy-source" right after the header is
extracted in `s3api/controllers/base.go`. This ensures that the
`CopySource` argument passed to the backend functions UploadPartCopy()
and CopyObject() does not start with '/'. Since the backends no longer
need to strip away any leading '/' in `CopySource`, the parts of
commit 5e484f2 modifying the Posix and Azure backends are reverted.

Fixes issue #773.

Signed-off-by: Christophe Vu-Brugier <christophe.vu-brugier@seagate.com>
 2024-08-29 15:08:17 +02:00
Ben McClelland
26ef99c593 Merge pull request #772 from versity/ben/docs 
fix: update help description to reference repo
 2024-08-28 19:41:26 -07:00
Ben McClelland
923ee5f0db fix: update help description to reference repo 2024-08-28 19:00:46 -07:00
Izzy-B37
282213a9de Demonstration dashboard for Versity gateway metrics (#771) 
* feat: demo grafana dashboard with docker compose
 2024-08-28 18:31:38 -07:00
Ben McClelland
366993d9d1 Merge pull request #748 from versity/ben/banner 
feat: change startup banner to versitygw version
 2024-08-28 11:14:30 -07:00