mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-07-24 09:02:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
519 Commits 37 Branches 53 Tags
8fc16392d1bf16699e2af983fd9935e24acdccfd
Commit Graph
74 Commits
Author SHA1 Message Date
jonaustin09andBen McClelland 8fc16392d1 feat: Dockerized azure backend to run 2 images: one for azurite, one for azure backend 2024-01-08 10:07:50 -08:00
jonaustin09andBen McClelland 8cc89fa713 feat: Azure backend implementation 2023-12-29 21:55:32 -08:00
Ben McClelland 3b945f72fc feat: azure blob backend initial pass 2023-12-29 21:54:56 -08:00
Ben McClellandandGitHub 6146dcff4a Merge pull request #358 from versity/ben/gw_perf 
fix: disable keepalive for better request rates under heavy load
 2023-12-26 10:28:46 -08:00
Ben McClellandandGitHub 3ba218bd9a Merge pull request #357 from versity/ben/quiet_logging 
feat: add quiet option to silence request log output
 2023-12-26 10:28:30 -08:00
Ben McClellandandGitHub 60bc9a3fc5 Merge pull request #354 from versity/ben/req_body_limit 
fix: body limit no longer needed for streaming request body
 2023-12-26 10:27:44 -08:00
Ben McClelland 216e50b9fd fix: disable keepalive for better request rates under heavy load 2023-12-23 22:56:49 -08:00
Ben McClelland d47cbcb39f feat: add quiet option to silence request log output 2023-12-23 16:52:56 -08:00
Ben McClelland 43bfe8a869 fix: body limit no longer needed for streaming request body 2023-12-22 15:24:23 -08:00
Ben McClelland 6e37096b35 fix: s3 backend performance increase with client reuse 
The previous way of initializing the s3 client in each call was
adding a lot of overhead and would tank performance beyond about
20 simultaneous requests.

Since the backend access is through a single account, we can init
and store this client for use from each api call.
 2023-12-22 15:09:44 -08:00
Ben McClelland 3d6e37bbb1 fix: make iam s3 object work similar to internal files 2023-12-18 10:39:20 -08:00
Ben McClelland ab43240b4e fix: add cli options to enable s3 iam service 2023-12-18 08:29:04 -08:00
jonaustin09andBen McClelland 27eb43d089 feat: Closes #290, implemented request body stream reading for PutObject and UploadPart actions. 2023-12-14 18:27:51 -08:00
Jon AustinandGitHub e9f01c8cce S3 proxy ListParts max parts issue (#338) 
* feat: implemented the logic to run integration tests separately

* fix: Resolved tests closer bug

* fix: Fixes #329, Fixed ListParts max-parts property issue

* fix: removed max-parts max int value check
 2023-12-13 10:22:34 -08:00
Jon AustinandGitHub 68073b9b73 Docker proxy fix (#337) 
* feat: implemented the logic to run integration tests separately

* fix: Fixed docker proxy command due to latest changes
 2023-12-12 16:06:55 -08:00
jonaustin09andBen McClelland 056c905a65 fix: Closes #323, fixed s3 proxy single user PutBucketAcl issue 2023-12-11 11:05:08 -08:00
jonaustin09andBen McClelland ef5a94420c feat: Created admin CLI actions in s3 proxy, Created iam proxy for proxy server 2023-12-11 11:05:08 -08:00
jonaustin09 c5007a68aa eat: Closes #317, Changed s3 proxy behavior to run in single user mode with CLI provided credentials 2023-12-04 10:06:59 -05:00
Ben McClelland 26a7d567b0 feat: allow integrations tests to be called as unit tests 
This reconfigures the integration tests a bit to be called as
unit tests. This makes debugging failing tests a little easier.
 2023-11-29 09:53:17 -08:00
jonaustin09 f2c02c6362 feat: Added gateway throughput performance test, got upload and download tests separated 2023-11-16 15:56:38 -05:00
jonaustin09 920b4945cd feat: Closes #236, Added 3 optional fields in iam user account, UserID, GroupID, ProjectID 2023-10-12 13:24:34 -04:00
Ben McClelland f58646b58d feat: s3proxy backend 
This backend redirects incoming requests to another s3 service.
This will use the incoming credentials to setup the client
requests to the external s3 service. So the IAM accounts (or
root account) must match what the external s3 service expects.
 2023-10-11 09:00:02 -07:00
jonaustin09andBen McClelland 52674ab0c5 feat: Closes #282, created a new integration test group and test cases for posix specific behaviours testing 2023-10-11 08:17:10 -07:00
Ben McClelland ae0b270c2c feat: move local iam cache to a more generic cache mechanism 
The local IAM accounts were being cached in memory for improved
performance, but this can be moved up a layer so that the cache
can benefit any configured IAM service.

This adds options to disable and tune TTL for cache. The balance
for the TTL is that a longer life will send requests to the IAM
service less frequently, but could be out of date with the service
accounts for that duration.
 2023-10-09 08:15:56 -07:00
jonaustin09 8f7a1bfc86 feat: Integrated a new option for IAM servcie: store IAM data in LDAP server 2023-10-03 14:02:21 -04:00
Ben McClelland ae4e382e61 feat: refactor internal iam service 
This moves the internal iam service from the posix backend so
that we can start implementing new iam services right in the auth
module.

The internal iam service has same behavior as before, but now
must be enabled with the --iam-dir cli option.

New single user service is the default when no other iam service
is selected. This just runs the gateway in single user mode with
just the root account.
 2023-10-02 11:12:18 -07:00
Ben McClelland c9475adb04 fix: remove body limit for admin app 
The BodyLimit is needed for large PUTs in the s3 api server, but
the admin server requests are not expected to exceed the default
4MB limit.
 2023-09-12 16:16:47 -07:00
jonaustin09andBen McClelland 5ab38e3dab feat: Closes #232, Added an option to run admin server in a different network, by specifying admin server address/ip 2023-09-12 16:04:31 -07:00
Ben McClellandandGitHub c04f6d7f00 Merge pull request #233 from versity/admin/list-buckets 
Issue 217, Admin API and CLI action to list all the buckets and its owners.
 2023-09-12 16:02:29 -07:00
jonaustin09 6ac69b3198 feat: Closes #217, Created an admin API and CLI action to list all the buckets and its owners as a table 2023-09-12 08:29:34 -04:00
Ben McClelland 145c2dd4e3 fix: builds for non 64 bit linux arch 
The scoutfs backend is only supported on 64bit linux.  This corrects
the build constraints to only supported linux arch, and prevents the
incompatible import for unspported arch.

We also need to adjust the body limit setting on 32 bit since this
is an int, and our default limit will overfow on 32 bit.
 2023-09-10 20:49:56 -07:00
Ben McClelland 6ab4090216 fix: add ADMIN_ENDPOINT_URL env var to admin cli 2023-09-07 13:17:54 -07:00
Ben McClelland ef92f57e7d feat: format admin cli list-users output in table 
Format list-users output in an easier to read table:
% versitygw admin list-users
Account  Role
-------  ----
myadmin  admin
myuser   user
 2023-09-07 08:48:18 -07:00
jonaustin09 7620651a49 fix: Merge conflicts resolved with main 2023-09-06 17:45:23 -04:00
jonaustin09 4c7584c99f feat: Closes #206, Added an admin api endpoint and a CLI action to change buckets owner 2023-09-06 17:41:47 -04:00
Ben McClelland df81ead6bc fix: move auth internal UserAcc to auth.Account 2023-09-05 16:21:21 -07:00
jonaustin09 d80580380d feat: Closes #205, Add admin api endpoint and CLI action to list users. Added unit tests for the api endpoint 2023-09-05 18:12:11 -04:00
jonaustin09 2cc0c7203c feat: Closes #189, added utility functions for testing, restructured tests for CreateBucket, DeleteBucket... and 5 more actions 2023-08-11 02:13:01 +04:00
Ben McClelland 5a9b744dd1 fix: allow logging to user specified log files 
This also cleans up some the of the error output to send to stderr.

This adds the Shutdown() to the logging interface, so we can keep the
log file open and just append entries.

This add HangUp() to the logging interface for log rotations.
 2023-07-25 23:39:45 -07:00
Ben McClelland 51e818b3e3 fix: cleanup unused adminRegion 2023-07-22 18:53:58 -07:00
Ben McClelland 8765a6c67f fix: signal.go spelling 2023-07-20 13:59:51 -07:00
jonaustin09andBen McClelland 7bceaaca39 feat: Set up bucket event notifications with nats 2023-07-20 13:36:16 -07:00
jonaustin09andBen McClelland fe547a19e9 feat: bucket event notifications 
Set up Bucket event notifications interface to send aws compatible format event messages to a configured event service.
First integrated service is kafka message broker as an option for bucket event notifications.
 2023-07-20 11:37:14 -07:00
jonaustin09 87d61a1eb3 feat: Setup audit loggin with webhook url and root level access.log file. CLI enables either webhook or server access logs by providing the flags 2023-07-14 23:40:05 +04:00
jonaustin09 fbafc6b34c feat: Changed admin api http methods, some cleanup in admin cli commands, bug fix in delete user IAM service 2023-07-06 21:21:20 +04:00
Jon AustinandGitHub 3c69c6922a Integration test cases for HeadBucket, CopyObject, DeleteObject actions (#117) 
* feat: Added integration test cases for HeadBucket, CopyObject, DeleteObjects
* feat: Added logger for debugging
 2023-06-29 10:40:54 -07:00
Jon AustinandGitHub 4bfb3d84d3 Acl integration test (#115) 
* feat: Added test an integration test case for acl actions(get, put), fixed PutBucketAcl actions bugs, fixed iam bugs on getting and creating user accounts

* fix: Fixed acl unit tests

* fix: Fixed cli path in exec command in acl integration test

* fix: fixed account creation bug
 2023-06-28 19:38:35 -07:00
Jon AustinandGitHub 30dbd02a83 Tag actions integrations tests (#114) 
* feat: Added an integration test case for for tag actions(get, put, delete)
 2023-06-26 14:25:24 -07:00
Ben McClelland 1d476c6d4d add signal handler for clean shutdown 2023-06-25 10:29:14 -07:00
Ben McClelland c4f5f958eb add functional tests to github actions 2023-06-23 18:38:19 -07:00