mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-01-07 12:15:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
645 Commits 20 Branches 44 Tags
b801a700d52debbf2a485d0f809b61e5e69395b0
Commit Graph

645 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ben McClelland
6e37096b35 fix: s3 backend performance increase with client reuse 
The previous way of initializing the s3 client in each call was
adding a lot of overhead and would tank performance beyond about
20 simultaneous requests.

Since the backend access is through a single account, we can init
and store this client for use from each api call.
 2023-12-22 15:09:44 -08:00
Ben McClelland
6f6af8ec07 Merge pull request #344 from versity/ben/s3proxy 
Ben/s3proxy
v0.10 		2023-12-19 09:58:30 -08:00
Ben McClelland
f27162b36d Merge pull request #347 from versity/dependabot/go_modules/golang.org/x/crypto-0.17.0 
chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0
 2023-12-18 17:09:10 -08:00
dependabot[bot]
22fcabe085 chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0 
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-12-18 21:51:46 +00:00
Ben McClelland
89783a69f4 Merge pull request #346 from versity/dependabot/go_modules/dev-dependencies-f554204c74 
chore(deps): bump the dev-dependencies group with 4 updates
 2023-12-18 13:51:20 -08:00
dependabot[bot]
9afca13329 chore(deps): bump the dev-dependencies group with 4 updates 
Bumps the dev-dependencies group with 4 updates: [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2), [github.com/google/uuid](https://github.com/google/uuid), [github.com/segmentio/kafka-go](https://github.com/segmentio/kafka-go) and [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2).


Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.47.5 to 1.47.6
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.47.5...service/s3/v1.47.6)

Updates `github.com/google/uuid` from 1.4.0 to 1.5.0
- [Release notes](https://github.com/google/uuid/releases)
- [Changelog](https://github.com/google/uuid/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/uuid/compare/v1.4.0...v1.5.0)

Updates `github.com/segmentio/kafka-go` from 0.4.46 to 0.4.47
- [Release notes](https://github.com/segmentio/kafka-go/releases)
- [Commits](https://github.com/segmentio/kafka-go/compare/v0.4.46...v0.4.47)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.15.7 to 1.15.8
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.15.7...config/v1.15.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/google/uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/segmentio/kafka-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-12-18 21:47:29 +00:00
Ben McClelland
3d6e37bbb1 fix: make iam s3 object work similar to internal files 2023-12-18 10:39:20 -08:00
Ben McClelland
ab43240b4e fix: add cli options to enable s3 iam service 2023-12-18 08:29:04 -08:00
Ben McClelland
b833e94c4b Merge pull request #343 from versity/ben/s3proxy 
chore: clarify s3 backend struct with better name
 2023-12-18 08:20:11 -08:00
Ben McClelland
2fb5ecfbc4 chore: clarify s3 backend struct with better name 2023-12-16 09:50:26 -08:00
Ben McClelland
0c7537e3b5 Merge pull request #331 from versity/ben/req-body-streaming 
Ben/req body streaming
 2023-12-15 08:23:58 -08:00
Ben McClelland
ba501e482d feat: steaming requests for put object and put part 
This builds on the previous work that sets up the body streaming
for the put object and put part requests. This adds the auth and
checksum readers to postpone the v4auth checks and the content
checksum until the end of the body stream.

This means that the backend with start reading the data from the
body stream before the request is fully validated and signatures
checked. So the backend must check the error returned from the
body reader for the final auth and content checks. The backend
is expected to discard the data upon error.

This should increase performance and reduce memory utilization
to no longer require caching the entire request body in memory
for put object and put part.
 2023-12-14 19:19:46 -08:00
jonaustin09
27eb43d089 feat: Closes #290, implemented request body stream reading for PutObject and UploadPart actions. 2023-12-14 18:27:51 -08:00
Jon Austin
90bb43f7c9 S3 Proxy PutBucketAcl reference bug (#341) 
* feat: implemented the logic to run integration tests separately

* fix: Fixes #336, fixed s3 proxy PutBucketAcl action's referance bug
 2023-12-14 09:43:02 -08:00
Jon Austin
206231f27b S3 Proxy DeleteObjects empty input (#340) 
* feat: implemented the logic to run integration tests separately

* fix: Fixes #335, fixed s3 proxy DeleteObject empty input bug
 2023-12-14 09:42:22 -08:00
Ben McClelland
a6e9fc5b00 Merge pull request #339 from versity/fix/s3proxy-copyobject-invarg 
S3 Proxy CopyObject invalid argument bug
 2023-12-13 15:39:30 -08:00
Ben McClelland
c6cbd82f8b fix: Fixes 330, Fixes 334, Fixed copySrcModifSince bug in CopyObject action 
commit 48d951edfe
Author: jonaustin09 <jonaustin460@gmail.com>
Date:   Wed Dec 13 09:06:44 2023 -0500
 2023-12-13 15:35:46 -08:00
Jon Austin
e9f01c8cce S3 proxy ListParts max parts issue (#338) 
* feat: implemented the logic to run integration tests separately

* fix: Resolved tests closer bug

* fix: Fixes #329, Fixed ListParts max-parts property issue

* fix: removed max-parts max int value check
 2023-12-13 10:22:34 -08:00
Jon Austin
68073b9b73 Docker proxy fix (#337) 
* feat: implemented the logic to run integration tests separately

* fix: Fixed docker proxy command due to latest changes
 2023-12-12 16:06:55 -08:00
Ben McClelland
c97c814c33 Merge pull request #333 from versity/dependabot/go_modules/dev-dependencies-716cc4f3d3 
chore(deps): bump the dev-dependencies group with 4 updates
 2023-12-12 09:48:00 -08:00
dependabot[bot]
0d370a8bb7 chore(deps): bump the dev-dependencies group with 4 updates 
Bumps the dev-dependencies group with 4 updates: [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2).


Updates `github.com/aws/aws-sdk-go-v2` from 1.23.5 to 1.24.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.23.5...v1.24.0)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.47.2 to 1.47.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.47.2...service/s3/v1.47.5)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.25.11 to 1.26.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.25.11...config/v1.26.1)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.15.4 to 1.15.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.15.7/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.15.4...config/v1.15.7)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-12-11 21:30:59 +00:00
Ben McClelland
3c14c46738 Merge pull request #328 from versity/s3-proxy-admin-actions 
S3 proxy admin actions
 2023-12-11 11:08:33 -08:00
jonaustin09
056c905a65 fix: Closes #323, fixed s3 proxy single user PutBucketAcl issue 2023-12-11 11:05:08 -08:00
jonaustin09
ef5a94420c feat: Created admin CLI actions in s3 proxy, Created iam proxy for proxy server 2023-12-11 11:05:08 -08:00
Ben McClelland
ac66ad01e5 Merge pull request #301 from versity/dockerize-app 
Dockerize the application
 2023-12-08 08:20:39 -08:00
Ben McClelland
cb394fd000 Merge pull request #326 from versity/ben/select_object_content 
Ben/select object content
 2023-12-08 08:19:00 -08:00
Ben McClelland
bed1691a93 feat: implement logic for s3 select object content stream 2023-12-07 15:01:24 -08:00
jonaustin09
48818927bb feat: Fixes #286, Created a struct which handles s3 select event streaming and event message construction 2023-12-06 14:02:36 -08:00
jonaustin09
32d7ada232 feat: Made gateway iam and setup directories configurable from environment variables in docker images 2023-12-06 10:08:06 -05:00
Ben McClelland
cd4821baa6 Merge pull request #327 from versity/dependabot/go_modules/dev-dependencies-be0b77cb9a 
chore(deps): bump the dev-dependencies group with 5 updates
 2023-12-04 15:27:11 -08:00
dependabot[bot]
e4922eb2e5 chore(deps): bump the dev-dependencies group with 5 updates 
Bumps the dev-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.23.2` | `1.23.5` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.46.0` | `1.47.2` |
| [github.com/urfave/cli/v2](https://github.com/urfave/cli) | `2.25.7` | `2.26.0` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.25.8` | `1.25.11` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.15.1` | `1.15.4` |


Updates `github.com/aws/aws-sdk-go-v2` from 1.23.2 to 1.23.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.23.2...v1.23.5)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.46.0 to 1.47.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.46.0...service/s3/v1.47.2)

Updates `github.com/urfave/cli/v2` from 2.25.7 to 2.26.0
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.25.7...v2.26.0)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.25.8 to 1.25.11
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.25.8...config/v1.25.11)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.15.1 to 1.15.4
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.15.4/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.15.1...config/v1.15.4)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-12-04 21:52:58 +00:00
Ben McClelland
f380613870 Merge pull request #325 from versity/proxy-iam-creds 
S3 proxy iam single user
 2023-12-04 09:47:49 -08:00
jonaustin09
c5007a68aa eat: Closes #317, Changed s3 proxy behavior to run in single user mode with CLI provided credentials 2023-12-04 10:06:59 -05:00
Ben McClelland
4a81f7a7a5 Merge pull request #321 from versity/ben/release_builds 
fix: fixes #311 - disable cgo in release builds
 2023-11-30 17:28:43 -08:00
Ben McClelland
ea55a488df fix: fixes #311 - disable cgo in release builds 
We were getting the following error on el7:
$ ./versitygw -h
./versitygw: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./versitygw)
./versitygw: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./versitygw)

The temporary fix is to disable cgo in the builds per this issue:
https://github.com/golang/go/issues/58550

We will need to visit per distro builds once we need to re-enable
cgo.
 2023-11-30 17:09:21 -08:00
Ben McClelland
29d3bfe184 Merge pull request #319 from versity/ben/integration_unit 
feat: allow integrations tests to be called as unit tests
 2023-11-29 11:58:58 -08:00
Ben McClelland
26a7d567b0 feat: allow integrations tests to be called as unit tests 
This reconfigures the integration tests a bit to be called as
unit tests. This makes debugging failing tests a little easier.
 2023-11-29 09:53:17 -08:00
Ben McClelland
c48f126557 Merge pull request #318 from versity/fix/issue-300-proxy-listbuckets-panic 
S3 proxy ListBucket panic
 2023-11-29 09:51:17 -08:00
Jon Austin
315cb0ea81 fix: Fixes #307, Fixed ListMultipartUploads and ListParts parts timestamp in s3 proxy (#316) 2023-11-29 08:54:12 -08:00
jonaustin09
c4b019f6ee fix: Fixes #300, Removed Displayname field from ListBuckets output as it's not supported in the gateway 2023-11-29 11:49:27 -05:00
Ben McClelland
ff787dc8f7 Merge pull request #315 from versity/fix/issue-308/proxy-error-handling 
S3 proxy SDK error handling
 2023-11-29 08:42:46 -08:00
jonaustin09
2b22509a90 fix: Fixes #302, #308. Added error handling function in s3 proxy implementation, which converts sdk error to API Errors 2023-11-29 08:40:06 -08:00
Ben McClelland
953d05ca56 Merge pull request #314 from versity/dependabot/go_modules/dev-dependencies-06ec20d170 
chore(deps): bump the dev-dependencies group with 5 updates
 2023-11-29 08:39:15 -08:00
dependabot[bot]
6102ef08a3 chore(deps): bump the dev-dependencies group with 5 updates 
Bumps the dev-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.43.1` | `1.46.0` |
| [github.com/segmentio/kafka-go](https://github.com/segmentio/kafka-go) | `0.4.45` | `0.4.46` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.14.0` | `0.15.0` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.25.4` | `1.25.8` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.14.1` | `1.15.1` |


Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.43.1 to 1.46.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.43.1...service/s3/v1.46.0)

Updates `github.com/segmentio/kafka-go` from 0.4.45 to 0.4.46
- [Release notes](https://github.com/segmentio/kafka-go/releases)
- [Commits](https://github.com/segmentio/kafka-go/compare/v0.4.45...v0.4.46)

Updates `golang.org/x/sys` from 0.14.0 to 0.15.0
- [Commits](https://github.com/golang/sys/compare/v0.14.0...v0.15.0)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.25.4 to 1.25.8
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.25.4...config/v1.25.8)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.14.1 to 1.15.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.15.1/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.14.1...config/v1.15.1)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/segmentio/kafka-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-11-29 16:31:56 +00:00
Ben McClelland
b210bf81f4 Merge pull request #312 from versity/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/credentials-1.16.6 
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.16.3 to 1.16.6
 2023-11-29 08:28:54 -08:00
dependabot[bot]
346f0c6d72 chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials 
Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.16.3 to 1.16.6.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/v1.16.6/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.16.3...v1.16.6)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-11-29 16:19:35 +00:00
Ben McClelland
7c9386d077 Merge pull request #310 from versity/dependabot/go_modules/dev-dependencies-37849a5052 
Dependabot/go modules/dev dependencies 37849a5052
 2023-11-29 08:18:12 -08:00
Ben McClelland
cd8ad7d482 fix: breaking changes with aws sdk updates 2023-11-28 13:51:32 -08:00
jonaustin09
452152ad11 feat: Added multistage build, removed unnecessary stuff from final image by only leaving the built binary. Added env variables reading instructions in Makefile 2023-11-27 14:23:52 -05:00
jonaustin09
3feddbd698 feat: Closes #185, Dockerized the application. Created Dockerfiles for dev and prod environments, created a docker compose config file to run the s3 and proxy servers in dev environments with live code update and container recreation. Added commands in Makefile to run s3, proxy and both servers as docker containers 2023-11-27 11:00:09 -05:00