Commit Graph

63 Commits

Author SHA1 Message Date
Ben McClelland
ab43240b4e fix: add cli options to enable s3 iam service 2023-12-18 08:29:04 -08:00
jonaustin09
27eb43d089 feat: Closes #290, implemented request body stream reading for PutObject and UploadPart actions. 2023-12-14 18:27:51 -08:00
Jon Austin
e9f01c8cce S3 proxy ListParts max parts issue (#338)
* feat: implemented the logic to run integration tests separately

* fix: Resolved tests closer bug

* fix: Fixes #329, Fixed ListParts max-parts property issue

* fix: removed max-parts max int value check
2023-12-13 10:22:34 -08:00
Jon Austin
68073b9b73 Docker proxy fix (#337)
* feat: implemented the logic to run integration tests separately

* fix: Fixed docker proxy command due to latest changes
2023-12-12 16:06:55 -08:00
jonaustin09
056c905a65 fix: Closes #323, fixed s3 proxy single user PutBucketAcl issue 2023-12-11 11:05:08 -08:00
jonaustin09
ef5a94420c feat: Created admin CLI actions in s3 proxy, Created iam proxy for proxy server 2023-12-11 11:05:08 -08:00
jonaustin09
c5007a68aa eat: Closes #317, Changed s3 proxy behavior to run in single user mode with CLI provided credentials 2023-12-04 10:06:59 -05:00
Ben McClelland
26a7d567b0 feat: allow integrations tests to be called as unit tests
This reconfigures the integration tests a bit to be called as
unit tests. This makes debugging failing tests a little easier.
2023-11-29 09:53:17 -08:00
jonaustin09
f2c02c6362 feat: Added gateway throughput performance test, got upload and download tests separated 2023-11-16 15:56:38 -05:00
jonaustin09
920b4945cd feat: Closes #236, Added 3 optional fields in iam user account, UserID, GroupID, ProjectID 2023-10-12 13:24:34 -04:00
Ben McClelland
f58646b58d feat: s3proxy backend
This backend redirects incoming requests to another s3 service.
This will use the incoming credentials to setup the client
requests to the external s3 service. So the IAM accounts (or
root account) must match what the external s3 service expects.
2023-10-11 09:00:02 -07:00
jonaustin09
52674ab0c5 feat: Closes #282, created a new integration test group and test cases for posix specific behaviours testing 2023-10-11 08:17:10 -07:00
Ben McClelland
ae0b270c2c feat: move local iam cache to a more generic cache mechanism
The local IAM accounts were being cached in memory for improved
performance, but this can be moved up a layer so that the cache
can benefit any configured IAM service.

This adds options to disable and tune TTL for cache. The balance
for the TTL is that a longer life will send requests to the IAM
service less frequently, but could be out of date with the service
accounts for that duration.
2023-10-09 08:15:56 -07:00
jonaustin09
8f7a1bfc86 feat: Integrated a new option for IAM servcie: store IAM data in LDAP server 2023-10-03 14:02:21 -04:00
Ben McClelland
ae4e382e61 feat: refactor internal iam service
This moves the internal iam service from the posix backend so
that we can start implementing new iam services right in the auth
module.

The internal iam service has same behavior as before, but now
must be enabled with the --iam-dir cli option.

New single user service is the default when no other iam service
is selected. This just runs the gateway in single user mode with
just the root account.
2023-10-02 11:12:18 -07:00
Ben McClelland
c9475adb04 fix: remove body limit for admin app
The BodyLimit is needed for large PUTs in the s3 api server, but
the admin server requests are not expected to exceed the default
4MB limit.
2023-09-12 16:16:47 -07:00
jonaustin09
5ab38e3dab feat: Closes #232, Added an option to run admin server in a different network, by specifying admin server address/ip 2023-09-12 16:04:31 -07:00
Ben McClelland
c04f6d7f00 Merge pull request #233 from versity/admin/list-buckets
Issue 217, Admin API and CLI action to list all the buckets and its owners.
2023-09-12 16:02:29 -07:00
jonaustin09
6ac69b3198 feat: Closes #217, Created an admin API and CLI action to list all the buckets and its owners as a table 2023-09-12 08:29:34 -04:00
Ben McClelland
145c2dd4e3 fix: builds for non 64 bit linux arch
The scoutfs backend is only supported on 64bit linux.  This corrects
the build constraints to only supported linux arch, and prevents the
incompatible import for unspported arch.

We also need to adjust the body limit setting on 32 bit since this
is an int, and our default limit will overfow on 32 bit.
2023-09-10 20:49:56 -07:00
Ben McClelland
6ab4090216 fix: add ADMIN_ENDPOINT_URL env var to admin cli 2023-09-07 13:17:54 -07:00
Ben McClelland
ef92f57e7d feat: format admin cli list-users output in table
Format list-users output in an easier to read table:
% versitygw admin list-users
Account  Role
-------  ----
myadmin  admin
myuser   user
2023-09-07 08:48:18 -07:00
jonaustin09
7620651a49 fix: Merge conflicts resolved with main 2023-09-06 17:45:23 -04:00
jonaustin09
4c7584c99f feat: Closes #206, Added an admin api endpoint and a CLI action to change buckets owner 2023-09-06 17:41:47 -04:00
Ben McClelland
df81ead6bc fix: move auth internal UserAcc to auth.Account 2023-09-05 16:21:21 -07:00
jonaustin09
d80580380d feat: Closes #205, Add admin api endpoint and CLI action to list users. Added unit tests for the api endpoint 2023-09-05 18:12:11 -04:00
jonaustin09
2cc0c7203c feat: Closes #189, added utility functions for testing, restructured tests for CreateBucket, DeleteBucket... and 5 more actions 2023-08-11 02:13:01 +04:00
Ben McClelland
5a9b744dd1 fix: allow logging to user specified log files
This also cleans up some the of the error output to send to stderr.

This adds the Shutdown() to the logging interface, so we can keep the
log file open and just append entries.

This add HangUp() to the logging interface for log rotations.
2023-07-25 23:39:45 -07:00
Ben McClelland
51e818b3e3 fix: cleanup unused adminRegion 2023-07-22 18:53:58 -07:00
Ben McClelland
8765a6c67f fix: signal.go spelling 2023-07-20 13:59:51 -07:00
jonaustin09
7bceaaca39 feat: Set up bucket event notifications with nats 2023-07-20 13:36:16 -07:00
jonaustin09
fe547a19e9 feat: bucket event notifications
Set up Bucket event notifications interface to send aws compatible format event messages to a configured event service.
First integrated service is kafka message broker as an option for bucket event notifications.
2023-07-20 11:37:14 -07:00
jonaustin09
87d61a1eb3 feat: Setup audit loggin with webhook url and root level access.log file. CLI enables either webhook or server access logs by providing the flags 2023-07-14 23:40:05 +04:00
jonaustin09
fbafc6b34c feat: Changed admin api http methods, some cleanup in admin cli commands, bug fix in delete user IAM service 2023-07-06 21:21:20 +04:00
Jon Austin
3c69c6922a Integration test cases for HeadBucket, CopyObject, DeleteObject actions (#117)
* feat: Added integration test cases for HeadBucket, CopyObject, DeleteObjects
* feat: Added logger for debugging
2023-06-29 10:40:54 -07:00
Jon Austin
4bfb3d84d3 Acl integration test (#115)
* feat: Added test an integration test case for acl actions(get, put), fixed PutBucketAcl actions bugs, fixed iam bugs on getting and creating user accounts

* fix: Fixed acl unit tests

* fix: Fixed cli path in exec command in acl integration test

* fix: fixed account creation bug
2023-06-28 19:38:35 -07:00
Jon Austin
30dbd02a83 Tag actions integrations tests (#114)
* feat: Added an integration test case for for tag actions(get, put, delete)
2023-06-26 14:25:24 -07:00
Ben McClelland
1d476c6d4d add signal handler for clean shutdown 2023-06-25 10:29:14 -07:00
Ben McClelland
c4f5f958eb add functional tests to github actions 2023-06-23 18:38:19 -07:00
Jon Austin
f84cfe58e7 Bench test (#110)
* feat: test CLI command set up for client side testing, test cases are corresponded with subcommands, added full-flow test case

* fix: TLS configuration removed

* feat: Added benchmark test for client side testing in the CLI

* fix: Removed unused variables

* fix: fixed staticcheck error
2023-06-23 09:55:04 -07:00
jonaustin09
24ae7a2e86 feat: test CLI command set up for client side testing, test cases are corresponded with subcommands, added full-flow test case 2023-06-22 10:08:11 -07:00
Ben McClelland
c4e0aa69a8 scoutfs: add support for glacier emulation mode 2023-06-21 10:27:14 -07:00
Ben McClelland
0f733ae0c8 refactor move auth to top level 2023-06-19 11:15:19 -07:00
Ben McClelland
d2eab5bce3 posix: move iam data store to file
Storing to a file will allow more than 64k of storage that the xattr
would be limited to. This attempts to resolve racing updates between
multiple gateways without an explicit coordination between gateways.

This wil also setup a default IAM file on init.
2023-06-19 10:21:53 -07:00
Ben McClelland
94808bb4a9 refactor iam service for blind backend store 2023-06-19 09:53:19 -07:00
Ben McClelland
09e8889e75 feat: scoutfs backend with move blocks multipart optimized 2023-06-16 10:25:52 -07:00
Jon Austin
1bcdf948ba feat: Move IAM configuration file creation on backend running, set up… (#89)
* feat: Move IAM configuration file creation on backend running
2023-06-13 11:13:18 -07:00
jonaustin09
def500d464 fix: Merged main branch into admin-delete-api 2023-06-12 20:00:34 +04:00
jonaustin09
b98f48ce2c feat: Added admin api and admin CLI aciton to delete a user 2023-06-12 19:58:28 +04:00
Ben McClelland
ef06d11d7c fix: get simple multipart upload tests passing 2023-06-12 07:37:21 -07:00