12f4920c8d
Closes #1549 Fixes #1593 Fixes #1521 Fixes #1427 Fixes #1311 Fixes #1301 Fixes #1040 This PR primarily focuses on checksum calculation within the gateway, but it also includes several related fixes and improvements. It introduces a middleware responsible for handling and calculating checksums for the `x-amz-checksum-*` headers and `Content-MD5`. The middleware is applied only to actions that expect a request body or checksum headers. It also enforces validation for actions that require a non-empty request body, returning an error if the body is missing. Similarly, it returns an error for actions where at least one checksum header (`Content-MD5` or `x-amz-checksum-*`) is required but none is provided. The implementation is based on [https://gist.github.com/niksis02/eec3198f03e561a0998d67af75c648d7](the reference table), tested directly against S3: It also fixes the error case where the `x-amz-sdk-checksum-algorithm` header is present but no corresponding `x-amz-checksum-*` or `x-amz-trailer` header is included. Additionally, the PR improves validation for the `x-amz-content-sha256` header. For actions that require this header, an error is now returned when it’s missing. For actions that don’t require it, the middleware no longer enforces its presence. Following the common S3 pattern, the header remains mandatory for admin routes. Finally, the `x-amz-content-sha256` header is now optional for anonymous requests, as it is not required in that case.
The Versity S3 Gateway:
A High-Performance S3 Translation Service
Binary release builds
Download latest release
| Linux/amd64 | Linux/arm64 | MacOS/amd64 | MacOS/arm64 | BSD/amd64 | BSD/arm64 |
|---|---|---|---|---|---|
| ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Use Cases
- Turn your local filesystem into an S3 server with a single command!
- Proxy S3 requests to S3 storage
- Simple to deploy S3 server with a single command
- Protocol compatibility in
posixallows common access to files via posix or S3 - Simplified interface for adding new storage system support
News
Check out latest wiki articles: https://github.com/versity/versitygw/wiki/Articles
Mailing List
Keep up to date with latest gateway announcements by signing up to the versitygw mailing list.
Documentation
See project documentation on the wiki.
Need help?
Ask questions in the community discussions.
Contact Versity Sales to discuss enterprise support.
Overview
Versity Gateway, a simple to use tool for seamless inline translation between AWS S3 object commands and storage systems. The Versity Gateway bridges the gap between S3-reliant applications and other storage systems, enabling enhanced compatibility and integration while offering exceptional scalability.
The server translates incoming S3 API requests and transforms them into equivalent operations to the backend service. By leveraging this gateway server, applications can interact with the S3-compatible API on top of already existing storage systems. This project enables leveraging existing infrastructure investments while seamlessly integrating with S3-compatible systems, offering increased flexibility and compatibility in managing data storage.
The Versity Gateway is focused on performance, simplicity, and expandability. The Versity Gateway is designed with modularity in mind, enabling future extensions to support additional backend storage systems. At present, the Versity Gateway supports any generic POSIX file backend storage, Versity’s open source ScoutFS filesystem, Azure Blob Storage, and other S3 servers.
The gateway is completely stateless. Multiple Versity Gateway instances may be deployed in a cluster to increase aggregate throughput. The Versity Gateway’s stateless architecture allows any request to be serviced by any gateway thereby distributing workloads and enhancing performance. Load balancers may be used to evenly distribute requests across the cluster of gateways for optimal performance.
The S3 HTTP(S) server and routing is implemented using the Fiber web framework. This framework is actively developed with a focus on performance. S3 API compatibility leverages the official aws-sdk-go-v2 whenever possible for maximum service compatibility with AWS S3.
Getting Started
See the Quickstart documentation.
Run the gateway with posix backend:
mkdir /tmp/vgw
ROOT_ACCESS_KEY="testuser" ROOT_SECRET_KEY="secret" ./versitygw --port :10000 posix /tmp/vgw
This will enable an S3 server on the current host listening on port 10000 and hosting the directory /tmp/vgw.
To get the usage output, run the following:
./versitygw --help
The command format is
versitygw [global options] command [command options] [arguments...]
The global options are specified before the backend type and the backend options are specified after.
Run the gateway in Docker
Use the published image like the native binary by passing CLI arguments:
docker run --rm versity/versitygw:latest --version
When no command arguments are supplied, the container looks for VGW_BACKEND and optional VGW_BACKEND_ARG/VGW_BACKEND_ARGS environment variables to determine which backend to start. Backend-specific configuration continues to come from the existing environment flags (for example ROOT_ACCESS_KEY, VGW_PORT, and others).
docker run --rm \
-e ROOT_ACCESS_KEY=testuser \
-e ROOT_SECRET_KEY=secret \
-e VGW_BACKEND=posix \
-e VGW_BACKEND_ARG=/data \
-p 10000:7070 \
-v $(pwd)/data:/data \
versity/versitygw:latest
If you need to pass additional CLI options, set VGW_ARGS with a space-delimited list, or continue passing arguments directly to docker run.
Versity gives you clarity and control over your archival storage, so you can allocate more resources to your core mission.
Contact
info@versity.com
+1 844 726 8826
@versitysoftware
