* Add utils.CertStorage for holding cert data that can be updated
at runtime.
* Add utils.NewTLSListener() to have a central place to control e.g. TLS
MinVersion across different servers.
* Add WithTLS() to webserver code so it looks more like the other
servers.
Fixes#1299
Implements a web interface for VersityGW with role-based access:
- Object explorer for all users to browse, upload, and download S3 objects
- Admin dashboard showing system overview and gateway status
- Admin-only user management for IAM user administration
- Admin-only bucket management for creating and configuring S3 buckets
- User authentication with automatic role-based page access
The web UI is disabled by default and only enabled with the --webui or
VGW_WEBUI_PORT env options that specify the listening address/port for
the web UI server. This preserves previous version behavior to not enable
any new ports/services unless opted in.
Login to the web UI login page with accesskey/secretkey credentials as
either user or admin account. UI functionality will auto detect login
role.
Regular users have access to the object explorer for managing files within
their accessible buckets. Admins additionally have access to user and bucket
management interfaces. The web UI is served on a separate port from the S3
server and integrates with existing S3 and Admin API endpoints.
All requests to the S3 and Admin services are signed by the browser and sent
directly to the S3/Admin service handlers. The login credentials are never
sent over the network for security purposes. This requires the S3/Admin
service to configure CORS Access-Control-Allow-Origin headers for these
requests.
Closes#1731
Implements the admin `CreateBucket` (`PATCH /:bucket/create`) endpoint and CLI command, which create a new bucket with the provided owner access key ID. The endpoint internally calls the S3 `CreateBucket` API, storing the new owner information in the request context under the `bucket-owner` key. This value is then retrieved by the S3 API layer and the backends.
The endpoint uses the custom `x-vgw-owner` HTTP header to pass the bucket owner access key ID.
The admin CLI command mirrors `aws s3api create-bucket` and supports all flags implemented by the gateway (for example, `--create-bucket-configuration`, `--acl`, `--object-ownership`, etc.).
There is some desire to have a web dashboard for the gateway. So
that we dont have to proxy all requests through the webserver
and expose credentials over the wire, the better approach would
be to enable CORS headers to allow browser requests directly to
the s3/admin service.
The default for these headers is off, so that they are only
enabled for instances that specfically want to support this
workload.
Closes#1714
There is a `Location` field in the `CompleteMultipartUpload` result that represents the newly created object URL. This PR adds this property to the `CompleteMultipartUpload` response, generating it dynamically in either host-style or path-style format, depending on the gateway configuration.
Fixes#1364
When a completely malformed request is sent to the gateway, Fiber/Fasthttp fails to parse the request, and the code execution never reaches the routers or handlers. Instead, the error is caught by the global error handler. These kinds of errors (malformed requests that fail during request parsing) are prefixed with **"error when reading request headers"** in Fiber. The implementation adds a check in the global error handler for this specific error message and returns an S3-like XML **BadRequest** error instead.
Implements graceful shutdown for the admin and s3api servers. They are shut down before other components (IAM, s3logger, etc.) to allow the servers to properly handle any pending requests while dependencies are still active. The shutdown process is controlled by a context with a 10-second timeout. If it exceeds this duration, all remaining requests are forcefully terminated and the servers are closed.
Fixes#1426
Fiber returns a custom error, if it fails to parse the `Content-Length` header. This implementation adds a check in the fiber global error handler to return an empty `400` Bad Request error, if fiber fails to parse the `Content-Length` header.
Fiber includes a built-in panic recovery middleware that catches panics in route handlers and middlewares, preventing the server from crashing and allowing it to recover. Alongside this, a stack trace handler has been implemented to store system panics in the context locals (stack).
Both the S3 API server and the Admin server use a global error handler to catch unexpected exceptions and recovered panics. The middleware’s logic is to log the panic or internal error and return an S3-style internal server error response.
Additionally, dedicated **Panic** and **InternalError** loggers have been added to the `s3api` debug logger to record system panics and internal errors in the console.
The debuglogger should be a top level module since we expect
all modules within the project to make use of this. If its
hidden in s3api, then contributors are less likely to make
use of this outside of s3api.
Implements a middleware that validates incoming bucket and object names before authentication. This helps prevent malicious attacks that attempt to access restricted or unreachable data in `POSIX`.
Adds test cases to cover such attack scenarios, including false negatives where encoded paths are used to try accessing resources outside the intended bucket.
Removes bucket validation from all other layers—including `controllers` and both `POSIX` and `ScoutFS` backends — by moving the logic entirely into the middleware layer.
This implementation introduces **public buckets**, which are accessible without signature-based authentication.
There are two ways to grant public access to a bucket:
* **Bucket ACLs**
* **Bucket Policies**
Only `Get` and `List` operations are permitted on public buckets. All **write operations** require authentication, regardless of whether public access is granted through an ACL or a policy.
The implementation includes an `AuthorizePublicBucketAccess` middleware, which checks if public access has been granted to the bucket. If so, authentication middlewares are skipped. For unauthenticated requests, appropriate errors are returned based on the specific S3 action.
---
**1. Bucket-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
}
]
}
```
**2. Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
**3. Both Bucket and Object-Level Operations:**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test"
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::test/*"
}
]
}
```
---
```sh
aws s3api create-bucket --bucket test --object-ownership BucketOwnerPreferred
aws s3api put-bucket-acl --bucket test --acl public-read
```
Closes#803
Implements host-style bucket addressing in the gateway. This feature can be enabled by running the gateway with the `--virtual-domain` flag and specifying a virtual domain name.
Example:
```bash
./versitygw -a user -s secret --virtual-domain localhost:7070 posix /tmp/vgw
```
The implementation follows this approach: it introduces a middleware (`HostStyleParser`) that parses the bucket name from the `Host` header and appends it to the URL path. This effectively transforms the request into a path-style bucket addressing format, which the gateway already supports. With this design, the gateway can handle both path-style and host-style requests when running in host-style mode.
For local testing, one can either set up a local DNS server to wildcard-match all subdomains of a specified domain and resolve them to the local IP address, or manually add entries to `/etc/hosts` to resolve bucket-prefixed hosts to the server IP (e.g., `127.0.0.1`).
Added missing debug logs in the `front-end` and `utility` functions.
Enhanced debug logging with the following improvements:
- Each debug message is now prefixed with [DEBUG] and appears in color.
- The full request URL is printed at the beginning of each debug log block.
- Request/response details are wrapped in framed sections for better readability.
- Headers are displayed in a colored box.
- XML request/response bodies are pretty-printed with indentation and color.
There are a few cases where parsing, validations checks, etc
error details are getting lost with the more generic error
responses. This add some opt-in debug logging to log more
info for these various error cases.
As described in
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
this adds support for reading from a chunked upload encoded request
body. The chunked reader modifies the data stream to remove the
chunk encoding while validating the chunk signatures in line. This
allows the upper layers to get just the object data stream.
Set up Bucket event notifications interface to send aws compatible format event messages to a configured event service.
First integrated service is kafka message broker as an option for bucket event notifications.
* feat: Added admin CLI, created api endpoint for creating new user, created action for admin CLI to create a new user, changed the authentication middleware to verify the users from db
* feat: Added both single and multi user support, added caching layer for getting IAM users
* fix: Added all the files
