This adds the webui-s3-prefix option to specify a prefix and host
the webui on the same port as the s3 service. Like the health
endpoint, this will mask any bucket with the same name as the
webui prefix.
The benefit of hosting this on the same interface as the s3
service is no longer needing the CORS headers for the browser
access if the webui and s3 access are on the same IP:PORT.
When access/secret are not provided, let AWS SDK v2 resolve credentials
from the default provider chain (env vars, IRSA, ECS/EC2 roles, etc.)
instead of forcing anonymous credentials.
Add an explicit anonymous credentials option for s3 proxy to force
backend anonymous access.
Fixes#1955
This adds the ability to specify unix domain socket paths for the
service listener with the --port <path> option. Where <path> can
be either a path to a file in a filesystem or prefixed with @ for
an abstract socket name.
Anything not matching the <host>:<port> pattern in the --port
option will be considered a socket filename.
Fix warnings from newer systemd:
Standard output type syslog is obsolete,
automatically updating to journal.
This updates the stdout/stderr to journal output type which is
what is getting set anyways after the syslog type has been
deprecated.
No expected behavior change with this other than quieting
warnings.
Closes#1847
This PR introduces a global optional gateway CLI flag `--disable-acl` (`VGW_DISABLE_ACL`) to disable ACL handling. When this flag is enabled, the gateway ignores all ACL-related headers, particularly in `CreateBucket`, `PutObject`, `CopyObject`, and `CreateMultipartUpload`.
`GetBucketAcl` behavior is unchanged simply returning the bucket ACL config.
There's no change in object ACL actions(`PutObjectACL`, `GetObjectACL`). They return a`NotImplemented` error as before.
A new custom error is added for PutBucketAcl calls when ACLs are disabled at the gateway level. Its HTTP status code and error code match AWS S3’s behavior, with only a slightly different error message.
In the access-control checker, ACL evaluation is fully bypassed. If ACLs are disabled only the bucket owner gets access to the bucket and all grantee checks are ignored.
The PR also includes minor refactoring of the S3 API server and router. The growing list of parameters passed to the router’s Init method has been consolidated into fields within the router struct, initialized during router construction. Parameters not needed by the S3 server are no longer stored in the server configuration and are instead forwarded directly to the router.
AWS introduced a relatively newer option for data integrity checks
that not all non-AWS server support yet. See this for mmore info:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
This change adds a new option: disable-data-integrity-check
to disable the data integrity checks in the client sdk for the
servers that may not yet support this. Use this only when the s3
service for the proxy does not support the data integrity features.
Fixes#1867
New cli options added:
webui-gateways - override auto-detected S3 gateway URLs for WebUI
webui-admin-gateways - override auto-detected admin gateway URLs
for WebUI
These also accept env vars VGW_WEBUI_GATEWAYS and
VGW_WEBUI_ADMIN_GATEWAYS for the options.
When setting these, this will override the url auto-detection for
the webui service urls dropdown options. By default, the gateway
auto-detects URLs based on the configured port settings. Use these
options to specify custom URLs when the auto-detected values are
incorrect (e.g., when running behind a reverse proxy or load
balancer). Multiple URLs can be specified with repeated options
or a comma-separated list with the environment variables.
for example:
--webui-gateways https://s3.example.com \
--webui-gateways http://192.168.1.100:7070
or
VGW_WEBUI_GATEWAYS=https://s3.example.com,http://192.168.1.100:7070
The gateway will validate the provided URLs with warnings for any
invalid URL specified. The gateway will terminate if these options
are set but contain no valid URLs.
Also added sorting to the auto-detected URLs so that localhost
URLs will be last in the list, since these will not likely work
on remote systems. The specified lists when provided are left
in the order they are specified to allow admins to determine
dropdown list ordering.
Fixes#1851
This allows specifying the following options more than once:
port, admin-port, webui
or using a comma-separated list for the env vars:
e.g., VGW_PORT=:7070,:8080,localhost:9090
This will also expand multiple interfaces from hostnames, for example
"localhost" in this case would resolve to both IPv4 and IPv6 interfaces:
localhost has address 127.0.0.1
localhost has IPv6 address ::1
This updates the banner to reflect all of the listening interfaces/ports,
and starts the service listener on all requested interfaces/ports.
Fixes#1761
* Update client.go to support anonymous S3 access
* Update s3.go to make access and secret parameters optional
* Update example.conf for more clear S3 access and secret usage
Fixes#1836
Implements a web interface for VersityGW with role-based access:
- Object explorer for all users to browse, upload, and download S3 objects
- Admin dashboard showing system overview and gateway status
- Admin-only user management for IAM user administration
- Admin-only bucket management for creating and configuring S3 buckets
- User authentication with automatic role-based page access
The web UI is disabled by default and only enabled with the --webui or
VGW_WEBUI_PORT env options that specify the listening address/port for
the web UI server. This preserves previous version behavior to not enable
any new ports/services unless opted in.
Login to the web UI login page with accesskey/secretkey credentials as
either user or admin account. UI functionality will auto detect login
role.
Regular users have access to the object explorer for managing files within
their accessible buckets. Admins additionally have access to user and bucket
management interfaces. The web UI is served on a separate port from the S3
server and integrates with existing S3 and Admin API endpoints.
All requests to the S3 and Admin services are signed by the browser and sent
directly to the S3/Admin service handlers. The login credentials are never
sent over the network for security purposes. This requires the S3/Admin
service to configure CORS Access-Control-Allow-Origin headers for these
requests.
The scoutfs filesystem allows setting project IDs on files and
directories for project level accounting tracking. This adds the
option to set the project id for the following:
create bucket
put object
put part
complete multipart upload
The project id will only be set if all of the following is true:
- set project id option enabled
- filesystem format version supports projects (version >1)
- account project id > 0
Some systems may choose to allow non-aws compliant bucket names
and/or handle the bucket naem validation in the backend instead.
This adds the option to turn off the strict bucket name validation
checks in the frontend API handlers.
When frontend bucket name validation is disabled, we need to do
sanity checks for posix compliant names in the posix/scoutfs
backends. This is automatically enabled when strict bucket
name validation is disabled.
Fixes#1564
Using Docker ENTRYPOINT should allow for configuration of running
versitygw within Docker container similar to how the systemd
service is setup with environment variables.
This also adds the backends azure and plugin to the acceptable
backend options for both docker and systemd.
Fixes#1335
This adds support for rabbitmq publisher for s3 events. The
mechanics are similar to kafka and nats, but will use the amqp
protocol to send bucket events.
O_TMPFILE can fail if the location we need to link the final
file is not within the same filesystem. This can happen if
there are different filesystem mounts within a bucket or if
using zfs nested datasets within a bucket.
Fixes#1194Fixes#1035
This was missed when we added an option for setting directory permissions
different than the default 0755. This adds the VGW_DIR_PERMS option and
description to the example.conf file.
This updates the systemd config to add VGW_IAM_LDAP_USER_ID_ATR
amd VGW_IAM_LDAP_GROUP_ID_ATR env var options that were already
in the cli config.
Fixes#733
This adds the ability to treat symlinks to directories at the top
level gateway directory as buckets the same as normal directories.
This could be a potential security issue allowing traversal into
other filesystems within the system, so is defaulted to off. This
can be enabled when specifically needed for both posix and scoutfs
backend systems.
Fixes#644
