Use net.JoinHostPort instead of manual string formatting in
buildServiceURLs so IPv6 addresses are properly wrapped in brackets.
This fixes malformed URLs like "http://::1:7070" being presented as
the default server URL in the WebUI login form when listening on an
IPv6 address; they now correctly render as "http://[::1]:7070".
Fixes#1926
Closes#1847
This PR introduces a global optional gateway CLI flag `--disable-acl` (`VGW_DISABLE_ACL`) to disable ACL handling. When this flag is enabled, the gateway ignores all ACL-related headers, particularly in `CreateBucket`, `PutObject`, `CopyObject`, and `CreateMultipartUpload`.
`GetBucketAcl` behavior is unchanged simply returning the bucket ACL config.
There's no change in object ACL actions(`PutObjectACL`, `GetObjectACL`). They return a`NotImplemented` error as before.
A new custom error is added for PutBucketAcl calls when ACLs are disabled at the gateway level. Its HTTP status code and error code match AWS S3’s behavior, with only a slightly different error message.
In the access-control checker, ACL evaluation is fully bypassed. If ACLs are disabled only the bucket owner gets access to the bucket and all grantee checks are ignored.
The PR also includes minor refactoring of the S3 API server and router. The growing list of parameters passed to the router’s Init method has been consolidated into fields within the router struct, initialized during router construction. Parameters not needed by the S3 server are no longer stored in the server configuration and are instead forwarded directly to the router.
This brings scoutfs in-line with the posix concurrency limiter.
This fixes a hang with scoutfs due to not correctly initializing
the concurrency in posix leading to a concurrency of 0 allowed.
This also adds a sane default to the posix concurrency when not
initialized.
AWS introduced a relatively newer option for data integrity checks
that not all non-AWS server support yet. See this for mmore info:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
This change adds a new option: disable-data-integrity-check
to disable the data integrity checks in the client sdk for the
servers that may not yet support this. Use this only when the s3
service for the proxy does not support the data integrity features.
Fixes#1867
New cli options added:
webui-gateways - override auto-detected S3 gateway URLs for WebUI
webui-admin-gateways - override auto-detected admin gateway URLs
for WebUI
These also accept env vars VGW_WEBUI_GATEWAYS and
VGW_WEBUI_ADMIN_GATEWAYS for the options.
When setting these, this will override the url auto-detection for
the webui service urls dropdown options. By default, the gateway
auto-detects URLs based on the configured port settings. Use these
options to specify custom URLs when the auto-detected values are
incorrect (e.g., when running behind a reverse proxy or load
balancer). Multiple URLs can be specified with repeated options
or a comma-separated list with the environment variables.
for example:
--webui-gateways https://s3.example.com \
--webui-gateways http://192.168.1.100:7070
or
VGW_WEBUI_GATEWAYS=https://s3.example.com,http://192.168.1.100:7070
The gateway will validate the provided URLs with warnings for any
invalid URL specified. The gateway will terminate if these options
are set but contain no valid URLs.
Also added sorting to the auto-detected URLs so that localhost
URLs will be last in the list, since these will not likely work
on remote systems. The specified lists when provided are left
in the order they are specified to allow admins to determine
dropdown list ordering.
Fixes#1851
This allows specifying the following options more than once:
port, admin-port, webui
or using a comma-separated list for the env vars:
e.g., VGW_PORT=:7070,:8080,localhost:9090
This will also expand multiple interfaces from hostnames, for example
"localhost" in this case would resolve to both IPv4 and IPv6 interfaces:
localhost has address 127.0.0.1
localhost has IPv6 address ::1
This updates the banner to reflect all of the listening interfaces/ports,
and starts the service listener on all requested interfaces/ports.
Fixes#1761
Closes#1815
Implements posix actions concurrency limiter. Since posix actions perform filesystem-heavy syscalls, a semaphore-based limiter is introduced to cap the maximum number of concurrent posix actions. When the limit is reached, additional action calls block until a slot becomes available.
For internal posix calls, the `no_acquire_slot` context key is used to prevent acquiring the limiter multiple times within a single action (e.g., PutObject internally calling PutObjectLegalHold).
The posix concurrency limit can be configured via the gateway posix subcommand flag (--concurrency) or the environment variable `VGW_POSIX_CONCURRENCY`. The default value is `5000`.
This is part of the thread exhaustion issue (#1815).
This PR introduces:
* A **maximum Fiber HTTP connections limit**
* A middleware that enforces a **hard limit on in-flight HTTP requests**
When the in-flight request limit is reached, the middleware returns an **S3-compatible `503 SlowDown`** error.
The same mechanism is implemented for the **admin server** (both max connections and max in-flight requests).
All limits are configurable via **CLI flags** and **environment variables**, for both the `s3api` server and the `admin` server.
---
| Setting | CLI Flag | Alias | Environment Variable | Default |
| --------------- | ------------------- | ----- | --------------------- | ------- |
| Max Connections | `--max-connections` | `-mc` | `VGW_MAX_CONNECTIONS` | 250000 |
| Max Requests | `--max-requests` | `-mr` | `VGW_MAX_REQUESTS` | 100000 |
---
| Setting | CLI Flag | Alias | Environment Variable | Default |
| --------------- | ------------------------- | ------ | --------------------------- | ------- |
| Max Connections | `--admin-max-connections` | `-amc` | `VGW_ADMIN_MAX_CONNECTIONS` | 250000 |
| Max Requests | `--admin-max-requests` | `-amr` | `VGW_ADMIN_MAX_REQUESTS` | 100000 |
* Update client.go to support anonymous S3 access
* Update s3.go to make access and secret parameters optional
* Update example.conf for more clear S3 access and secret usage
Fixes#1836
Fixes#1792Fixes#1747Fixes#1797Fixes#1799
This PR primarily introduces delimiter support and several bug fixes for the `ListMultipartUploads` action in the POSIX and Azure backends. Delimiter handling is now implemented — when a delimiter is present in multipart-upload object key names, the backend collects and returns the appropriate common prefixes.
This functionality is achieved by introducing a common multipart-upload lister in the backend package. All backends (Azure, POSIX) now use this lister. The lister accepts a list that is already sorted and filtered by `KeyMarker` and `Prefix`.
Previously, the `KeyMarker` was required to exactly match an existing multipart-upload object key. This restriction is removed. The listing now relies on a lexicographical comparison between the provided `KeyMarker` and existing multipart-upload object keys.
Validation for `UploadIdMarker` is also added to correctly return an `InvalidArgument` error for invalid upload IDs. If `KeyMarker` is missing, the `UploadIdMarker` is ignored entirely. If `KeyMarker` is provided, a valid upload ID is one that matches an upload belonging to *the first object key after the KeyMarker*. For example, if the `KeyMarker` is `foo`, but the provided `UploadIdMarker` corresponds to an upload under `quxx`, it is invalid. It must match one of the uploads for the next object key equal to `foo`.
Finally, this PR fixes multipart-upload sorting. Multipart uploads must be sorted primarily lexicographically by their object key, and secondarily—when multiple uploads share the same object key—by their initiation time in ascending order.
Closes#1760
The admin subcommand accepts the `--access` and `--secret` flags as admin user credentials. This change makes these flags optional and allows the root user credentials to be used as admin credentials with lower precedence when the admin flags are omitted.
Both invocation styles are now valid:
```bash
versitygw admin --access user --secret pass -er http://127.0.0.1:7070 list-buckets
```
and
```bash
versitygw --access user --secret pass admin -er http://127.0.0.1:7070 list-buckets
```
Additionally, if `ROOT_ACCESS_KEY` and `ROOT_SECRET_KEY` are already set, the following also works:
```bash
versitygw admin -er http://127.0.0.1:7070 list-buckets
```
* Add utils.CertStorage for holding cert data that can be updated
at runtime.
* Add utils.NewTLSListener() to have a central place to control e.g. TLS
MinVersion across different servers.
* Add WithTLS() to webserver code so it looks more like the other
servers.
Fixes#1299
Implements a web interface for VersityGW with role-based access:
- Object explorer for all users to browse, upload, and download S3 objects
- Admin dashboard showing system overview and gateway status
- Admin-only user management for IAM user administration
- Admin-only bucket management for creating and configuring S3 buckets
- User authentication with automatic role-based page access
The web UI is disabled by default and only enabled with the --webui or
VGW_WEBUI_PORT env options that specify the listening address/port for
the web UI server. This preserves previous version behavior to not enable
any new ports/services unless opted in.
Login to the web UI login page with accesskey/secretkey credentials as
either user or admin account. UI functionality will auto detect login
role.
Regular users have access to the object explorer for managing files within
their accessible buckets. Admins additionally have access to user and bucket
management interfaces. The web UI is served on a separate port from the S3
server and integrates with existing S3 and Admin API endpoints.
All requests to the S3 and Admin services are signed by the browser and sent
directly to the S3/Admin service handlers. The login credentials are never
sent over the network for security purposes. This requires the S3/Admin
service to configure CORS Access-Control-Allow-Origin headers for these
requests.
Closes#1731
Implements the admin `CreateBucket` (`PATCH /:bucket/create`) endpoint and CLI command, which create a new bucket with the provided owner access key ID. The endpoint internally calls the S3 `CreateBucket` API, storing the new owner information in the request context under the `bucket-owner` key. This value is then retrieved by the S3 API layer and the backends.
The endpoint uses the custom `x-vgw-owner` HTTP header to pass the bucket owner access key ID.
The admin CLI command mirrors `aws s3api create-bucket` and supports all flags implemented by the gateway (for example, `--create-bucket-configuration`, `--acl`, `--object-ownership`, etc.).
There is some desire to have a web dashboard for the gateway. So
that we dont have to proxy all requests through the webserver
and expose credentials over the wire, the better approach would
be to enable CORS headers to allow browser requests directly to
the s3/admin service.
The default for these headers is off, so that they are only
enabled for instances that specfically want to support this
workload.
The scoutfs filesystem allows setting project IDs on files and
directories for project level accounting tracking. This adds the
option to set the project id for the following:
create bucket
put object
put part
complete multipart upload
The project id will only be set if all of the following is true:
- set project id option enabled
- filesystem format version supports projects (version >1)
- account project id > 0
Closes#1621
These changes introduce the `projectID` field in IAM user accounts. The field has been added across all IAM systems: internal, IPA, LDAP, Vault, and S3 object. Support has also been added to the admin CLI commands to create, update, and list users with the `projectID` included.
Implements graceful shutdown for the admin and s3api servers. They are shut down before other components (IAM, s3logger, etc.) to allow the servers to properly handle any pending requests while dependencies are still active. The shutdown process is controlled by a context with a 10-second timeout. If it exceeds this duration, all remaining requests are forcefully terminated and the servers are closed.
Some systems may choose to allow non-aws compliant bucket names
and/or handle the bucket naem validation in the backend instead.
This adds the option to turn off the strict bucket name validation
checks in the frontend API handlers.
When frontend bucket name validation is disabled, we need to do
sanity checks for posix compliant names in the posix/scoutfs
backends. This is automatically enabled when strict bucket
name validation is disabled.
Fixes#1564
This change introduces concurrent execution for integration tests. It adds a mechanism to run tests either synchronously or in parallel, controlled by a new flag. By default, tests continue to run in synchronous mode to maintain predictable behavior during local development. In GitHub Actions, the tests are now executed in parallel mode to significantly reduce overall runtime.
The implementation uses a semaphore-based concurrency control to limit the number of parallel test executions and ensures graceful shutdown through context cancellation. This approach improves test performance while keeping the system stable and backward compatible.
New CLI flags:
- --iam-vault-namespace
- --iam-vault-auth-namespace
- --iam-vault-secret-storage-namespace
Behavior:
- Auth requests use the auth namespace
- KV operations use the secret storage namespace
- If a specific namespace is not set, the shared namespace is used
- With AppRole, different auth and secret namespaces are rejected
Using Docker ENTRYPOINT should allow for configuration of running
versitygw within Docker container similar to how the systemd
service is setup with environment variables.
This also adds the backends azure and plugin to the acceptable
backend options for both docker and systemd.
Fixes#1335
Fiber includes a built-in panic recovery middleware that catches panics in route handlers and middlewares, preventing the server from crashing and allowing it to recover. Alongside this, a stack trace handler has been implemented to store system panics in the context locals (stack).
Both the S3 API server and the Admin server use a global error handler to catch unexpected exceptions and recovered panics. The middleware’s logic is to log the panic or internal error and return an S3-style internal server error response.
Additionally, dedicated **Panic** and **InternalError** loggers have been added to the `s3api` debug logger to record system panics and internal errors in the console.
The buildSearchFilter function doesn't need to be exported, and
can use strings.Builder. Add a unit test to make sure this didn't
change any logic.
This will also use the debuglogger to enable debugging.
The debuglogger should be a top level module since we expect
all modules within the project to make use of this. If its
hidden in s3api, then contributors are less likely to make
use of this outside of s3api.
This fix enables Versity Gateway to serve clients that use the AWS C++ SDK - without enabling keepalive in the fiber connection, clients that use the AWS C++ SDK like Run:ai's model streamer [will wig out from all of the closed connections and fail to function](https://github.com/run-ai/runai-model-streamer/issues/55) when connecting to a Versity GW back end.
This fix is intentionally side-effect free in that it retains the current default behavior, with the ability to override it via an env var or CLI arg
This adds support for rabbitmq publisher for s3 events. The
mechanics are similar to kafka and nats, but will use the amqp
protocol to send bucket events.
Adjusts the admin apis to the new advanced routing changes.
Enables debug logging for the separate admin server(when a separate server is run for the admin apis).
Adds the quiet mode for the separate admin server.
Closes#803
Implements host-style bucket addressing in the gateway. This feature can be enabled by running the gateway with the `--virtual-domain` flag and specifying a virtual domain name.
Example:
```bash
./versitygw -a user -s secret --virtual-domain localhost:7070 posix /tmp/vgw
```
The implementation follows this approach: it introduces a middleware (`HostStyleParser`) that parses the bucket name from the `Host` header and appends it to the URL path. This effectively transforms the request into a path-style bucket addressing format, which the gateway already supports. With this design, the gateway can handle both path-style and host-style requests when running in host-style mode.
For local testing, one can either set up a local DNS server to wildcard-match all subdomains of a specified domain and resolve them to the local IP address, or manually add entries to `/etc/hosts` to resolve bucket-prefixed hosts to the server IP (e.g., `127.0.0.1`).
Closes#1295
Makes the user `role` mutable in /update-user admin endpoint.
Integrates the changes in the `admin update-user` cli command, by adding the `role` flag for a user role modification.
Closes#1111
Bucket ACLs and policies are now stored in the meta bucket as objects with the following prefixes:
- `vgw-meta-acl-<bucket-name>`
- `vgw-meta-policy-<bucket-name>`
The name of the meta bucket is provided during S3 proxy initialization. The gateway verifies whether the specified bucket exists; if it does not, an error is returned.
If no meta bucket is provided, the S3 proxy returns default values for ACL and policy actions.
O_TMPFILE can fail if the location we need to link the final
file is not within the same filesystem. This can happen if
there are different filesystem mounts within a bucket or if
using zfs nested datasets within a bucket.
Fixes#1194Fixes#1035
The part files for multipart uploads are considered temporary
files and should not be archived by default. This adds the
noarchive attribute to the part files to prevent scoutam from
trying to archive these.
There is a new parameter, disablenoarchive, that will prevent
adding the noarchive attribute to these files for the case
where there is a desire to archive these temp files.
