mirror of https://github.com/versity/versitygw.git synced 2026-09-23 08:24:17 +00:00

Files

T

DavvyyandGitHub d03ac3c299 fix: reject a PutObject whose body ends before Content-Length

* fix(s3api): reject a PUT whose body ends before Content-Length

With a plain (non-aws-chunked) request body, a PutObject whose body ends
before Content-Length bytes have arrived is committed as a complete but
shorter object. Real S3 rejects this with IncompleteBody and the object
never becomes visible.

Nothing on that path compares bytes received against Content-Length:
fasthttp reports a connection closed mid-body as a plain io.EOF (the
conversion to io.ErrUnexpectedEOF exists only on the chunked
transfer-encoding branch), the authentication middleware leaves an
UNSIGNED-PAYLOAD body unwrapped, and io.Copy treats io.EOF as a normal
end of stream.

ErrIncompleteBody already exists and is enforced by the chunk readers.
This adds the equivalent check for plain bodies, in the controller so
that every backend is covered, and only when the body is not already an
aws-chunked reader.

POST-Object is deliberately untouched: it is a separate handler whose
ContentLength is an upper bound, so a byte-count check there would
break browser form uploads.

* test(s3api): cover ContentLengthReader

Complete and empty bodies, truncated bodies (including an EOF delivered
together with the final bytes, and one byte per Read), a body longer
than announced, and pass-through of a non-EOF error.

* fix(s3api): check plain bodies against Content-Length, not the decoded length

Review catch: the controller replaces contentLength with
X-Amz-Decoded-Content-Length whenever that header is present, regardless
of payload type, and the previous commit fed that value to
ContentLengthReader. A complete plain upload whose decoded header is
larger than Content-Length was then rejected with IncompleteBody, where
it had succeeded before.

AWS S3 ignores X-Amz-Decoded-Content-Length on a plain body and stores
Content-Length bytes. The decoded length only describes aws-chunked
payloads, and those skip this wrapper anyway, so the check now reads the
raw Content-Length header.

If the header is missing or unparseable the body is left unwrapped: an
aws-chunked request without Content-Length is already rejected earlier
with ErrMissingContentLength, so there is nothing to check here.

* test(integration): cover aborted uploads and the decoded-length case

Three tests, as requested in review.

PutObject_aborted_plain_body       plain PUT, body ends early -> no object
PutObject_aborted_streaming_body   same for an aws-chunked upload
PutObject_plain_body_with_decoded_length
                                   complete plain PUT carrying
                                   X-Amz-Decoded-Content-Length still
                                   succeeds and stores Content-Length bytes

The abort is driven by a reader that fails partway, so the transport
tears the connection down mid-body - that is what a client that dies or
cancels looks like on the wire. Sending a short body with a normal
reader would not reproduce it: net/http would simply report the
mismatch itself.

The third test is the regression the review found. It fails without the
accompanying fix.

* test(integration): replace the bogus streaming abort test with a real one

The reviewer is right: putObjectAborted always sends
x-amz-content-sha256: UNSIGNED-PAYLOAD, and the gateway decides aws-chunked
from that header alone, so PutObject_aborted_streaming_body took the same
plain path as the test above it. It duplicated the plain test instead of
covering the streaming one.

Dropped it and added UnsignedStreamingPayloadTrailer_aborted_connection in
tests/integration/unsigned_streaming_payload_trailer.go, where it belongs.

UnsignedStreamingPayloadTrailer_incomplete_body already covers malformed and
truncated framing, but every case there is a COMPLETE request. The new test is
the other shape: valid framing whose bytes simply stop arriving.

2026-09-20 11:18:15 -07:00

certs

fix: honor explicit public bucket policy deny

2026-05-22 00:01:24 +04:00

checker

chore: run go modernize tool

2026-03-10 09:47:37 -07:00

commands

Merge pull request #2389 from dualfroz/fix-illegal-location-constraint

2026-09-14 15:03:16 -07:00

conditionalrace

fix(posix): make conditional PUT evaluation and publication atomic per key

2026-08-10 17:10:21 -05:00

drivers

test: website redirect test, multipart upload util cleanup, install updates

2026-09-13 11:36:46 -03:00

host-style-tests

feat: automates the host-style tests in the pipeline

2025-05-23 02:22:45 +04:00

integration

fix: reject a PutObject whose body ends before Content-Length

2026-09-20 11:18:15 -07:00

rest_scripts

Merge pull request #2389 from dualfroz/fix-illegal-location-constraint

2026-09-14 15:03:16 -07:00

README.md

Command-Line Tests

Instructions - Running Locally
* Posix Backend
* Static Bucket Mode
* S3 Backend
* Direct Mode
Instructions - Running With Docker
Instructions - Running With Docker-Compose
Environment Parameters
* Secret
* Non-Secret
REST Scripts

Instructions - Running Locally

Posix Backend

Note

: many of the required libraries, and a good rundown of the installation procedure, can be found in the Dockerfile_test_bats dockerfile in the root folder, as well as system.yml in the .github/workflows folder.

Automatic

Run tests/install_test.sh to automatically run steps 1 through 5 below. Then proceed with manual setup.

Manual

Build the versitygw binary.
Install the command-line interface(s) you want to test if unavailable on your machine.
- aws cli: Instructions are here.
- s3cmd: Instructions are here.
- mc: Instructions are here.
Install BATS. Instructions are here.
Install bats-support and bats-assert. This can be done by saving the root folder of each repo (https://github.com/bats-core/bats-support and https://github.com/ztombol/bats-assert) in the tests folder.
If running on Mac OS, install jq with the command brew install jq.
Create a .secrets file in the tests folder, and add the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, and AWS_PROFILE values to the file.
Create a local AWS profile for connection to S3, and add the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION values for your account to the profile. Example:

    export AWS_PROFILE=versity-test
    export AWS_ACCESS_KEY_ID=<your account ID>
    export AWS_SECRET_ACCESS_KEY=<your account key>
    export AWS_REGION=<your account region>
    aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID --profile $AWS_PROFILE
    aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY --profile $AWS_PROFILE
    aws configure set aws_region $AWS_REGION --profile $AWS_PROFILE

Create an environment file (.env) similar to the ones in this folder, setting the AWS_PROFILE parameter to the name of the profile you created.
If using SSL, create a local private key and certificate, such as with the commands below. Afterwards, set the KEY and CERT fields in the .env file to these, respectively.

    openssl genpkey -algorithm RSA -out versitygw.pem -pkeyopt rsa_keygen_bits:2048
    openssl req -new -x509 -key versitygw.pem -out cert.pem -days 365

The program uses two environment values for the bucket names (or prefixes): BUCKET_ONE_NAME and BUCKET_TWO_NAME. Originally, and for static and some older tests, these are the full bucket names. In later, non-static tests, these are prefixes, and bucket suffixes are auto-generated. Set BUCKET_ONE_NAME and BUCKET_TWO_NAME to the desired names of your buckets, for older and static bucket tests, or prefixes, for newer and non-static bucket tests. If you want static buckets, i.e. you don't want them to be re-created each test, set RECREATE_BUCKETS to false.
In the root repo folder, run single test group with VERSITYGW_TEST_ENV=<env file> tests/run.sh <options>. To print options, run tests/run.sh -h. To run all tests, technically, the user can run VERSITYGW_TEST_ENV=<env file> tests/run_all.sh, but this is not recommended due to long running time. Instead, see the docker section below to run tests in parallel.
BATS tests can also be run directly with the format VERSIYTGW_TEST_ENV=<env file> tests/<test file name>, or for single tests, VERSIYTGW_TEST_ENV=<env file> tests/<file name> -f <test name>. Example: VERSITYGW_TEST_ENV=tests/.env tests/test_rest_bucket.sh -f "REST - HeadBucket".

Static Bucket Mode

To preserve buckets while running tests, set RECREATE_BUCKETS to false. Two utility functions are included, if needed, to create, and delete buckets for this: tests/setup_static.sh and tests/remove_static.sh. Note that this creates a bucket with object lock enabled, and some tests may fail if the bucket being tested doesn't have object lock enabled.

S3 Backend (Not Working)

Instructions are mostly the same; however, testing with the S3 backend requires two S3 accounts. Ideally, these are two real accounts, but one can also be a dummy account that versity uses internally.

To set up the latter:

Create a new AWS profile with ID and key values set to dummy 20-char allcaps and 40-char alphabetical values respectively.
In the .secrets file being used, create the fields AWS_ACCESS_KEY_ID_TWO and AWS_SECRET_ACCESS_KEY_TWO. Set these values to the actual AWS ID and key.
Set the values for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY the same dummy values set in the AWS profile, and set AWS_PROFILE to the profile you just created.
Create a new AWS profile with these dummy values. In the .env file being used, set the AWS_PROFILE parameter to the name of this new profile, and the ID and key fields to the dummy values.
Set BACKEND to s3. Also, change the MC_ALIAS value if testing mc in this configuration.

Direct Mode

To communicate directly with s3, in order to compare the gateway results to direct results:

Create an AWS profile with the direct connection info. Set AWS_PROFILE to this.
Set RUN_VERSITYGW to false.
Set AWS_ENDPOINT_URL to the typical endpoint location (usually https://s3.amazonaws.com).
If testing s3cmd, create a new s3cfg.local file with host_base and host_bucket set to s3.amazonaws.com.
If testing mc, change the MC_ALIAS value to a new value such as versity-direct.

Instructions - Running With Docker

Copy .secrets.default to .secrets in the tests folder and change the parameters and add the additional s3 fields explained in the S3 Backend section above if running with the s3 backend.
By default, the dockerfile uses the arm architecture (usually modern Mac). If using amd (usually earlier Mac or Linux), you can either replace the corresponding ARG values directly, or with arg="<param>=<amd library or folder>" Also, you can determine which is used by your OS with uname -a.
Build and run the Dockerfile_test_bats file. Change the SECRETS_FILE and CONFIG_FILE parameters to point to your secrets and config file, respectively, if not using the defaults. Example: docker build -t <tag> --build-arg="SECRETS_FILE=<file>" --build-arg="CONFIG_FILE=<file>" -f tests/Dockerfile_test_bats .. Also, default docker parameters, such as TARGETARCH, at the top of the Dockerfile can be changed by the user if needed.
To run the entire suite, run docker run -it <image name>. This is not recommended due to the sheer amount of tests. To run an individual suite, pass in the name of the suite as defined in tests/run.sh (e.g. REST bucket tests -> docker run -it <image tag or ID> rest-bucket). Also, multiple specific suites can be run, if separated by comma.
To list all suites available, the -h tag can be passed. Example: docker run -t <image tag or ID> -h.
By default, the config is placed in the /home/tester/config folder inside the container. Logs are printed to /home/tester/log. To overwrite the config, an .env folder can be placed in a mounted host folder, and to view the logs, a mounted folder can also be used. Example: docker run -v $PWD/runtime/config:/home/tester/config -v $PWD/runtime/log:/home/tester/log -t bats_test s3
To use tag functionality, the --tags parameter can be passed to the container.
To troubleshoot the Docker container, use docker run -it --entrypoint /bin/bash <image tag or ID> to use the shell and examine the container.

In Parallel

The script run_parallel.sh is provided to allow users to more quickly run tests in parallel on the local machine. To use this script, first build and tag a docker container with the desired configuration. Next, run the script with this tag, the desired tests (can find with ./tests/run.sh -l) separated by comma, the maximum number of parallel jobs (default 4), and the folder to place the logs.

Instructions - Running with docker-compose

A file named docker-compose-bats.yml is provided in the root folder. A few configurations are provided, and you can also create your own provided you have a secrets and config file:

insecure (without certificates), with creation/removal of buckets
secure, posix backend, with static buckets
secure, posix backend, with creation/removal of buckets
secure, s3 backend, with creation/removal of buckets
direct mode

To use each of these, creating a separate .env file for each is suggested. How to do so is explained below.

To run in insecure mode, comment out the CERT and KEY parameters in the .env file, and change the prefix for the AWS_ENDPOINT_URL parameter to http://. Also, set S3CMD_CONFIG to point to a copy of the default s3cmd config file that has use_https set to false. Finally, change MC_ALIAS to something new to avoid overwriting the secure MC_ALIAS values.

To use static buckets set the RECREATE_BUCKETS value to false.

For the s3 backend, see the S3 Backend instructions above.

If using AMD rather than ARM architecture, add the corresponding args values matching those in the Dockerfile for amd libraries.

A single instance can be run with docker-compose -f docker-compose-bats.yml up <service name>

Environment Parameters

Secret

AWS_PROFILE, AWS_ENDPOINT_URL, AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY: identical to the same parameters in s3.

AWS_CANONICAL_ID: for direct mode, the canonical ID for the main user (owner)

ACL_AWS_CANONICAL_ID: for direct mode, the canonical ID for the user to test ACL changes and access by non-owners

ACL_AWS_ACCESS_KEY_ID, ACL_AWS_SECRET_ACCESS_KEY: for direct mode, the ID and key for the S3 user in the ACL_AWS_CANONICAL_ID account.

ACL_AWS_ACCESS_KEY_ID_TWO, ACL_AWS_SECRET_ACCESS_KEY_TWO: if running a second versitygw application, the user ID and secret key for this application.

USER_ID_{role}_{id}, USERNAME_{role}_{id}, PASSWORD_{role}_{id}: for setup_user_v2 non-autocreated users, the format for the user.

example: USER_ID_USER_1={name}: user ID corresponding to the first user with user permissions in the test.

Non-Secret

VERSITY_EXE: location of the versity executable relative to test folder.

RUN_VERSITYGW: whether to run the versitygw executable, should be set to false when running tests directly against s3, or testing against another S3-compatible endpoint.

PORT: port to run the versity app on, if not specified, defaults to 7070.

PORT_TWO: port to run the second versity app on, if running two versity applications simultaneously. If not specified, defaults to 7071.

BACKEND: the storage backend type for the gateway, e.g. posix or s3.

LOCAL_FOLDER: if running with a posix backend, the backend storage folder.

BUCKET_ONE_NAME, BUCKET_TWO_NAME: test bucket names. In newer tests and when RECREATE_BUCKETS is set to true, these are prefixes and the suffixes are autogenerated.

RECREATE_BUCKETS: whether to delete buckets between tests. If set to false, the bucket will be restored to an original state for the purpose of ensuring consistent tests, but not deleted.

CERT, KEY: certificate and key locations if using SSL.

S3CMD_CONFIG: location of s3cmd config file if running s3cmd tests.

SECRETS_FILE: file where sensitive values, such as AWS_SECRET_ACCESS_KEY, should be stored.

MC_ALIAS: Minio MC alias if running MC tests.

LOG_LEVEL: level for test logger (1 - only critical, 2 - errors, 3 - warnings, 4 - info, 5 - debug info, 6 - tracing)

GOCOVERDIR: folder to put golang coverage info in, if checking coverage info.

USERS_FOLDER: folder to use if storing IAM data in a folder.

USERS_BUCKET: bucket to use if storing IAM data in an S3 bucket

IAM_TYPE: how to store IAM data (s3 or folder).

TEST_LOG_FILE: log file location for these bats tests.

VERSITY_LOG_FILE: log file for versity application as it is tested by bats tests.

DIRECT: if true, bypass versitygw and run directly against s3, for comparison and validity-checking purposes. This parameter disables the AWS_ENDPOINT_URL parameter.

DIRECT_DISPLAY_NAME: AWS ACL main user display name if DIRECT is set to true.

DIRECT_AWS_USER_ID: AWS policy 12-digit user ID if DIRECT is set to true.

USERNAME_ONE, PASSWORD_ONE, USERNAME_TWO, PASSWORD_TWO: setup_user (v1), credentials for users created and tested for non-root user versitygw operations (non-setup_user_v2).

TEST_FILE_FOLDER: where to put temporary test files.

REMOVE_TEST_FILE_FOLDER: whether to delete the test file folder between tests, should be set to true unless checking the files after a single test, or not yet sure that the test folder is in a safe location to avoid deleting other files.

VERSIONING_DIR: where to put gateway file versioning info.

COMMAND_LOG: where to store list of client commands, which if using will be reported during test failures.

TIME_LOG: optional log to show duration of individual tests

DIRECT_S3_ROOT_ACCOUNT_NAME: for direct mode, S3 username for user with root permissions

DELETE_BUCKETS_AFTER_TEST: whether or not to delete buckets after individual tests, useful for debugging if the post-test bucket state needs to be checked

AUTOGENERATE_USERS: setup_user_v2, whether or not to autocreate users for tests. If set to false, users must be pre-created (see Secret section above).

USER_AUTOGENERATION_PREFIX: setup_user_v2, if AUTOGENERATE_USERS is set to true, the prefix for the autocreated username.

CREATE_STATIC_USERS_IF_NONEXISTENT: setup_user_v2, if AUTOGENERATE_USERS is set to false, generate non-existing users if they don't exist, but don't delete them, as with user autogeneration

DIRECT_POST_COMMAND_DELAY: in v1 direct mode, time to wait before sending new commands to try to prevent propagation delay issues

SKIP_ACL_TESTING: avoid ACL tests for systems which do not use ACLs

MAX_FILE_DOWNLOAD_CHUNK_SIZE: when set, will divide the download of large files with GetObject into chunks of the given size. Useful for direct testing with slower connections.

SKIP_USERS_TESTS: skip versitygw-specific users tests, set to false to test against other S3 gateways

MAX_OPENSSL_COMMAND_LOG_BYTES: number of OpenSSL command bytes to display in command log, can prevent the display of too many chars in the case of large payload commands, -1 means display whole command

COVERAGE_LOG: if set, where to write test or test suite coverage data

PYTHON_ENV_FOLDER: where to place or use the python environment to calculate certain AWS checksums. The default is env in the TEST_FILE_FOLDER.

TEMPLATE_MATRIX_FILE: YAML file location used to retrieve the templates for expected responses, in some cases

SKIP_POLICY: set to true to skip tests involving policies

SKIP_BUCKET_OWNERSHIP_CONTROLS: set to true to avoid bucket ownership operations. This is needed to properly set up and clean the buckets if these operations are not supported by the server.

BYPASS_ENV_FILE: skip loading .env file on startup, default is false

DIRECT_CLOUDFRONT_TAG: if communicating directly with S3, name of tag used for HTTPS cloudfront distributions for bucket websites

GO_COMMAND_GENERATOR_EXECUTABLE: location to set executable for go REST command generation. If not used, program will run go run ... each time rather than compiling. However, if using, user must ensure that executable is removed and recompiled on go code updates.

QUICK_COMPARE_SIZE: for some comparisons between local and endpoint files to verify correct transfer, use checksum rather than full download to compare if file or part is above this size. If not used, then just download and compare.

WEBSITE: website port, if using versitygw website functionality

WEBSITE_ENDPOINT: website endpoint, if using versitygw website functionality

REST Scripts

REST scripts are included for calls to S3's REST API in the ./tests/rest_scripts/ folder. To call a script, the following parameters are needed:

AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.
AWS_ENDPOINT_URL (default: https://localhost:7070)
OUTPUT_FILE: file where the command's response data is written
Any other parameters specified at the top of the script file, such as payloads and variables. Sometimes, defaults are included.

Upon success, the script will return a response code, and write the data to the OUTPUT_FILE location.

Example: AWS_ACCESS_KEY_ID={id} AWS_SECRET_ACCESS_KEY={key} AWS_ENDPOINT_URL=https://s3.amazonaws.com OUTPUT_FILE=./output_file.xml ./tests/rest_scripts/list_buckets.sh