mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-08-17 12:46:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
eeff64c25576b71d7da7d992816dbf4e3dce23e6
versitygw/runtests.sh
T
niksis02 3227a629dc feat: add AWS-compatible standalone IAM service 
Closes #1640

Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI.

Server usage

Start the IAM server with internal file-backed storage:

    mkdir -p /tmp/versitygw-iam
    ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam

Start the IAM server with Vault KV v2 storage using AppRole:

    VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam

Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates.

Configure the AWS CLI credentials used by the IAM server:

    export AWS_ACCESS_KEY_ID=user
    export AWS_SECRET_ACCESS_KEY=pass
    export AWS_DEFAULT_REGION=us-east-1

Implemented IAM actions

CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users.

    aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob

    aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage

GetUser returns a stored user or the root identity when requested without a username through the IAM Query API.

    aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob

ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits.

    aws --endpoint-url http://127.0.0.1:7070 iam list-users

    aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100

UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users.

    aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/

DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users.

    aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert

IAM protocol and authentication

- Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests.
- Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields.
- Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1.
- Support both Authorization-header and query-string SigV4 authentication.
- Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes.
- Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures.

Storage implementations

- Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts.
- Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration.
- Introduce a common Storer interface and require exactly one storage backend to be configured.

Server and embedding support

- Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends.
- Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications.

Gateway-level internal packages

- Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it.
- Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors.
- Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses.
- Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling.
- Add `internal/routekit` for shared query, form, and header route matchers.
- Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling.
- Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures.

Testing and CI

- Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination.
- Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`.
- Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior.
- Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole.
- Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting.
- Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
2026-08-15 17:38:14 +04:00

279 lines
7.9 KiB
Bash
Executable File
Raw Blame History

#!/bin/bash
# parse options
USE_SIDECAR=false
RUN_DATA_INTEGRITY_ETAG_TESTS=true
for arg in "$@"; do
case "$arg" in
--sidecar) USE_SIDECAR=true ;;
--skip-data-integrity-etag-tests) RUN_DATA_INTEGRITY_ETAG_TESTS=false ;;
esac
done
# build sidecar flag for versitygw invocations
SIDECAR_FLAG=""
SIDECAR_TEST_FLAG=""
if $USE_SIDECAR; then
rm -rf /tmp/sidecar
mkdir /tmp/sidecar
SIDECAR_FLAG="--sidecar /tmp/sidecar"
SIDECAR_TEST_FLAG="--sidecar"
fi
# make temp dirs
rm -rf /tmp/gw
mkdir /tmp/gw
rm -rf /tmp/covdata
mkdir /tmp/covdata
rm -rf /tmp/https.covdata
mkdir /tmp/https.covdata
rm -rf /tmp/versioning.covdata
mkdir /tmp/versioning.covdata
rm -rf /tmp/versioning.https.covdata
mkdir /tmp/versioning.https.covdata
rm -rf /tmp/noacl.covdata
mkdir /tmp/noacl.covdata
rm -rf /tmp/versioningdir
mkdir /tmp/versioningdir
# setup tls certificate and key
ECHO "Generating TLS certificate and key in the cert.pem and key.pem files"
openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048
openssl req -new -x509 -key key.pem -out cert.pem -days 365 -subj "/C=US/ST=California/L=San Francisco/O=Versity/OU=Software/CN=versity.com"
ECHO "Running the sdk test over http"
# run server in background not versioning-enabled
# port: 7070(default)
GOCOVERDIR=/tmp/covdata ./versitygw -a user -s pass --iam-dir /tmp/gw posix $SIDECAR_FLAG /tmp/gw &
GW_PID=$!
sleep 1
# check if gateway process is still running
if ! kill -0 $GW_PID; then
echo "server no longer running"
exit 1
fi
# run tests
# full flow tests
if ! ./versitygw test -a user -s pass -e http://127.0.0.1:7070 full-flow --parallel $SIDECAR_TEST_FLAG; then
echo "full flow tests failed"
kill $GW_PID
exit 1
fi
# posix tests
if ! ./versitygw test -a user -s pass -e http://127.0.0.1:7070 posix; then
echo "posix tests failed"
kill $GW_PID
exit 1
fi
# gateway iam tests
if ! ./versitygw test -a user -s pass -e http://127.0.0.1:7070 gw-iam; then
echo "gateway iam tests failed"
kill $GW_PID
exit 1
fi
kill $GW_PID
ECHO "Running the sdk test over https"
# run server in background with TLS certificate
# port: 7071(default)
GOCOVERDIR=/tmp/https.covdata ./versitygw --cert "$PWD/cert.pem" --key "$PWD/key.pem" -p :7071 -a user -s pass --iam-dir /tmp/gw posix $SIDECAR_FLAG /tmp/gw &
GW_HTTPS_PID=$!
sleep 1
# check if https gateway process is still running
if ! kill -0 $GW_HTTPS_PID; then
echo "server no longer running"
exit 1
fi
# run tests
# full flow tests
if ! ./versitygw test --allow-insecure -a user -s pass -e https://127.0.0.1:7071 full-flow --parallel $SIDECAR_TEST_FLAG; then
echo "full flow tests failed"
kill $GW_HTTPS_PID
exit 1
fi
# posix tests
if ! ./versitygw test --allow-insecure -a user -s pass -e https://127.0.0.1:7071 posix; then
echo "posix tests failed"
kill $GW_HTTPS_PID
exit 1
fi
# gateway iam tests
if ! ./versitygw test --allow-insecure -a user -s pass -e https://127.0.0.1:7071 gw-iam; then
echo "gateway iam tests failed"
kill $GW_HTTPS_PID
exit 1
fi
kill $GW_HTTPS_PID
ECHO "Running the sdk test over http against the versioning-enabled gateway"
# run server in background versioning-enabled
# port: 7072
GOCOVERDIR=/tmp/versioning.covdata ./versitygw -p :7072 -a user -s pass --iam-dir /tmp/gw posix $SIDECAR_FLAG --versioning-dir /tmp/versioningdir /tmp/gw &
GW_VS_PID=$!
# wait a second for server to start up
sleep 1
# check if versioning-enabled gateway process is still running
if ! kill -0 $GW_VS_PID; then
echo "versioning-enabled server no longer running"
exit 1
fi
# run tests
# full flow tests
if ! ./versitygw test -a user -s pass -e http://127.0.0.1:7072 full-flow -vs --parallel $SIDECAR_TEST_FLAG; then
echo "versioning-enabled full-flow tests failed"
kill $GW_VS_PID
exit 1
fi
# posix tests
if ! ./versitygw test -a user -s pass -e http://127.0.0.1:7072 posix -vs; then
echo "versiongin-enabled posix tests failed"
kill $GW_VS_PID
exit 1
fi
# kill off server
kill $GW_VS_PID
ECHO "Running the sdk test over https against the versioning-enabled gateway"
# run server in background versioning-enabled
# port: 7073
GOCOVERDIR=/tmp/versioning.https.covdata ./versitygw --cert "$PWD/cert.pem" --key "$PWD/key.pem" -p :7073 -a user -s pass --iam-dir /tmp/gw posix $SIDECAR_FLAG --versioning-dir /tmp/versioningdir /tmp/gw &
GW_VS_HTTPS_PID=$!
# wait a second for server to start up
sleep 1
# check if versioning-enabled gateway process is still running
if ! kill -0 $GW_VS_HTTPS_PID; then
echo "versioning-enabled server no longer running"
exit 1
fi
# run tests
# full flow tests
if ! ./versitygw test --allow-insecure -a user -s pass -e https://127.0.0.1:7073 full-flow -vs --parallel $SIDECAR_TEST_FLAG; then
echo "versioning-enabled full-flow tests failed"
kill $GW_VS_HTTPS_PID
exit 1
fi
# posix tests
if ! ./versitygw test --allow-insecure -a user -s pass -e https://127.0.0.1:7073 posix -vs; then
echo "versiongin-enabled posix tests failed"
kill $GW_VS_HTTPS_PID
exit 1
fi
# kill off server
kill $GW_VS_HTTPS_PID
if $RUN_DATA_INTEGRITY_ETAG_TESTS; then
ECHO "Running data-integrity-etag integration tests"
# run server in background with data-integrity-etag enabled
# port: 7075
GOCOVERDIR=/tmp/covdata ./versitygw -p :7075 -a user -s pass --iam-dir /tmp/gw posix $SIDECAR_FLAG --data-integrity-etag /tmp/gw &
GW_DI_ETAG_PID=$!
# wait a second for server to start up
sleep 1
# check if data-integrity-etag gateway process is still running
if ! kill -0 $GW_DI_ETAG_PID; then
echo "data-integrity-etag server no longer running"
exit 1
fi
if ! ./versitygw test -a user -s pass -e http://127.0.0.1:7075 data-integrity-etag; then
echo "data-integrity-etag tests failed"
kill $GW_DI_ETAG_PID
exit 1
fi
kill $GW_DI_ETAG_PID
fi
ECHO "Running No ACL integration tests"
# run server in background versioning-enabled
# port: 7073
GOCOVERDIR=/tmp/noacl.covdata ./versitygw -p :7074 -a user -s pass -noacl --iam-dir /tmp/gw posix $SIDECAR_FLAG /tmp/gw &
GW_NO_ACL_PID=$!
# wait a second for server to start up
sleep 1
# check if noacl gateway process is still running
if ! kill -0 $GW_NO_ACL_PID; then
echo "noacl server no longer running"
exit 1
fi
if ! ./versitygw test --allow-insecure -a user -s pass -e http://127.0.0.1:7074 noacl; then
echo "No ACL integration tests failed"
kill $GW_NO_ACL_PID
exit 1
fi
# kill off server
kill $GW_NO_ACL_PID
exit 0
# -----------------------------------------------------------------------------
# Coverage Reports (Go 1.20+ Runtime Coverage)
#
# The servers above were started with GOCOVERDIR=<dir>, which causes Go to write
# raw coverage artifacts into those directories (covmeta + covcounters.* files).
# These raw files must be processed with "go tool covdata" to generate
# human-readable coverage reports.
#
# You may generate *per-environment* coverage or a *merged full-suite* report.
#
# -----------------------------------------------------------------------------
# 1) INDIVIDUAL COVERAGE REPORTS
#
# Example for a single environment (e.g. /tmp/covdata):
#
# go tool covdata percent -i=/tmp/covdata
# go tool covdata textfmt -i=/tmp/covdata -o /tmp/profile.txt
# go tool cover -html=/tmp/profile.txt
#
# Repeat using:
# /tmp/covdata
# /tmp/https.covdata
# /tmp/versioning.covdata
# /tmp/versioning.https.covdata
# /tmp/noacl.covdata
#
# This gives you coverage metrics isolated per test suite / server mode.
#
# -----------------------------------------------------------------------------
# 2) MERGED COVERAGE REPORT (RECOMMENDED)
#
# If you want a unified report combining all environments:
#
# go tool covdata merge \
# -i=/tmp/covdata,/tmp/https.covdata,/tmp/versioning.covdata,/tmp/versioning.https.covdata,/tmp/noacl.covdata \
# -o /tmp/allcovdata
#
# go tool covdata percent -i=/tmp/allcovdata
# go tool covdata textfmt -i=/tmp/allcovdata -o /tmp/all_profile.txt
# go tool cover -html=/tmp/all_profile.txt
#
# This produces the full aggregate coverage across all HTTP/HTTPS,
# versioning-enabled, non-versioning, and no-ACL test runs.
#
# -----------------------------------------------------------------------------
Reference in New Issue View Git Blame Copy Permalink