mirror of
https://github.com/versity/versitygw.git
synced 2026-08-17 20:56:21 +00:00
Closes #1640 Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI. Server usage Start the IAM server with internal file-backed storage: mkdir -p /tmp/versitygw-iam ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam Start the IAM server with Vault KV v2 storage using AppRole: VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates. Configure the AWS CLI credentials used by the IAM server: export AWS_ACCESS_KEY_ID=user export AWS_SECRET_ACCESS_KEY=pass export AWS_DEFAULT_REGION=us-east-1 Implemented IAM actions CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users. aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage GetUser returns a stored user or the root identity when requested without a username through the IAM Query API. aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits. aws --endpoint-url http://127.0.0.1:7070 iam list-users aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100 UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users. aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/ DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users. aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert IAM protocol and authentication - Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests. - Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields. - Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1. - Support both Authorization-header and query-string SigV4 authentication. - Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes. - Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures. Storage implementations - Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts. - Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration. - Introduce a common Storer interface and require exactly one storage backend to be configured. Server and embedding support - Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends. - Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications. Gateway-level internal packages - Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it. - Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors. - Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses. - Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling. - Add `internal/routekit` for shared query, form, and header route matchers. - Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling. - Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures. Testing and CI - Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination. - Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`. - Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior. - Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole. - Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting. - Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
211 lines
6.8 KiB
Bash
Executable File
211 lines
6.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
set -Eeuo pipefail
|
|
|
|
IAM_PID=""
|
|
IAM_HTTPS_PID=""
|
|
IAM_VAULT_PID=""
|
|
CERT_DIR=""
|
|
|
|
stop_process() {
|
|
local pid="${1:-}"
|
|
if [[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null; then
|
|
kill "$pid" 2>/dev/null || true
|
|
fi
|
|
if [[ -n "$pid" ]]; then
|
|
wait "$pid" 2>/dev/null || true
|
|
fi
|
|
}
|
|
|
|
cleanup() {
|
|
local status=$?
|
|
trap - EXIT
|
|
stop_process "$IAM_VAULT_PID"
|
|
stop_process "$IAM_HTTPS_PID"
|
|
stop_process "$IAM_PID"
|
|
if [[ -n "$CERT_DIR" ]]; then
|
|
rm -rf "$CERT_DIR"
|
|
fi
|
|
exit "$status"
|
|
}
|
|
|
|
trap cleanup EXIT
|
|
trap 'exit 130' INT
|
|
trap 'exit 143' TERM
|
|
|
|
wait_for_server() {
|
|
local name="$1"
|
|
local url="$2"
|
|
local pid="$3"
|
|
shift 3
|
|
|
|
for _ in {1..50}; do
|
|
if curl --fail --silent --max-time 1 "$@" "$url" >/dev/null 2>&1; then
|
|
return 0
|
|
fi
|
|
if ! kill -0 "$pid" 2>/dev/null; then
|
|
echo "$name stopped before becoming ready" >&2
|
|
wait "$pid" 2>/dev/null || true
|
|
return 1
|
|
fi
|
|
sleep 0.2
|
|
done
|
|
|
|
echo "timed out waiting for $name at $url" >&2
|
|
return 1
|
|
}
|
|
|
|
for tool in curl jq openssl; do
|
|
if ! command -v "$tool" >/dev/null 2>&1; then
|
|
echo "required command not found: $tool" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
# Create fresh data and coverage directories for each run.
|
|
rm -rf /tmp/iam /tmp/iam-https \
|
|
/tmp/iam.covdata /tmp/iam.https.covdata /tmp/iam.vault.covdata
|
|
mkdir -p /tmp/iam /tmp/iam-https \
|
|
/tmp/iam.covdata /tmp/iam.https.covdata /tmp/iam.vault.covdata
|
|
|
|
CERT_DIR=$(mktemp -d)
|
|
echo "Generating a temporary TLS certificate"
|
|
openssl genpkey -algorithm RSA -out "$CERT_DIR/key.pem" -pkeyopt rsa_keygen_bits:2048
|
|
openssl req -new -x509 -key "$CERT_DIR/key.pem" -out "$CERT_DIR/cert.pem" \
|
|
-days 1 -subj "/C=US/ST=California/L=San Francisco/O=Versity/OU=Software/CN=versity.com"
|
|
|
|
echo "Running IAM API integration tests over HTTP"
|
|
GOCOVERDIR=/tmp/iam.covdata ./versitygw --health /healthz -p :7075 -a user -s pass \
|
|
iam --dir /tmp/iam &
|
|
IAM_PID=$!
|
|
wait_for_server "IAM API HTTP server" "http://127.0.0.1:7075/healthz" "$IAM_PID"
|
|
./versitygw test -a user -s pass -e http://127.0.0.1:7075 iam
|
|
stop_process "$IAM_PID"
|
|
IAM_PID=""
|
|
|
|
echo "Running IAM API integration tests over HTTPS"
|
|
GOCOVERDIR=/tmp/iam.https.covdata ./versitygw --health /healthz \
|
|
--cert "$CERT_DIR/cert.pem" --key "$CERT_DIR/key.pem" \
|
|
-p :7076 -a user -s pass iam --dir /tmp/iam-https &
|
|
IAM_HTTPS_PID=$!
|
|
wait_for_server "IAM API HTTPS server" "https://127.0.0.1:7076/healthz" "$IAM_HTTPS_PID" --insecure
|
|
./versitygw test --allow-insecure -a user -s pass -e https://127.0.0.1:7076 iam
|
|
stop_process "$IAM_HTTPS_PID"
|
|
IAM_HTTPS_PID=""
|
|
|
|
# Vault is provided by the GitHub Actions service container. The root token is
|
|
# used only to provision a least-privilege AppRole for the IAM API under test.
|
|
readonly VAULT_ADDR="${VAULT_ADDR:-http://127.0.0.1:8200}"
|
|
: "${VAULT_TOKEN:?VAULT_TOKEN must contain a Vault provisioning token}"
|
|
readonly VAULT_PROVISION_TOKEN="$VAULT_TOKEN"
|
|
unset VAULT_TOKEN
|
|
readonly VAULT_MOUNT_PATH="kv"
|
|
readonly VAULT_SECRET_PATH="iam"
|
|
readonly VAULT_POLICY_NAME="iam-api-tests"
|
|
readonly VAULT_ROLE_NAME="iam-api-tests"
|
|
|
|
vault_request() {
|
|
local method="$1"
|
|
local path="$2"
|
|
local data="${3:-}"
|
|
local args=(
|
|
--fail
|
|
--silent
|
|
--show-error
|
|
--request "$method"
|
|
--header "X-Vault-Token: $VAULT_PROVISION_TOKEN"
|
|
)
|
|
|
|
if [[ -n "$data" ]]; then
|
|
args+=(--header "Content-Type: application/json" --data "$data")
|
|
fi
|
|
|
|
curl "${args[@]}" "${VAULT_ADDR%/}/v1/$path"
|
|
}
|
|
|
|
echo "Waiting for Vault"
|
|
for _ in {1..30}; do
|
|
if curl --fail --silent --max-time 1 "${VAULT_ADDR%/}/v1/sys/health" >/dev/null 2>&1; then
|
|
break
|
|
fi
|
|
sleep 0.5
|
|
done
|
|
curl --fail --silent --show-error "${VAULT_ADDR%/}/v1/sys/health" >/dev/null
|
|
|
|
echo "Provisioning Vault KV v2 and AppRole"
|
|
vault_mounts=$(vault_request GET sys/mounts)
|
|
if jq -e --arg mount "$VAULT_MOUNT_PATH/" '.data[$mount] == null' <<<"$vault_mounts" >/dev/null; then
|
|
vault_request POST "sys/mounts/$VAULT_MOUNT_PATH" \
|
|
'{"type":"kv","options":{"version":"2"}}' >/dev/null
|
|
elif ! jq -e --arg mount "$VAULT_MOUNT_PATH/" \
|
|
'.data[$mount].type == "kv" and .data[$mount].options.version == "2"' \
|
|
<<<"$vault_mounts" >/dev/null; then
|
|
echo "Vault mount $VAULT_MOUNT_PATH exists but is not KV v2" >&2
|
|
exit 1
|
|
fi
|
|
|
|
vault_auth_methods=$(vault_request GET sys/auth)
|
|
if jq -e '.data["approle/"] == null' <<<"$vault_auth_methods" >/dev/null; then
|
|
vault_request POST sys/auth/approle '{"type":"approle"}' >/dev/null
|
|
fi
|
|
|
|
vault_policy=$(printf '%s\n' \
|
|
"path \"$VAULT_MOUNT_PATH/data/$VAULT_SECRET_PATH/*\" { capabilities = [\"create\", \"update\", \"read\"] }" \
|
|
"path \"$VAULT_MOUNT_PATH/metadata/$VAULT_SECRET_PATH/\" { capabilities = [\"list\"] }" \
|
|
"path \"$VAULT_MOUNT_PATH/metadata/$VAULT_SECRET_PATH/*\" { capabilities = [\"delete\"] }")
|
|
vault_policy_payload=$(jq -nc --arg policy "$vault_policy" '{policy: $policy}')
|
|
vault_request PUT "sys/policies/acl/$VAULT_POLICY_NAME" "$vault_policy_payload" >/dev/null
|
|
|
|
vault_role_payload=$(jq -nc --arg policy "$VAULT_POLICY_NAME" '{
|
|
token_policies: [$policy],
|
|
token_no_default_policy: true,
|
|
token_ttl: "5m",
|
|
token_max_ttl: "15m",
|
|
secret_id_ttl: "15m"
|
|
}')
|
|
vault_request POST "auth/approle/role/$VAULT_ROLE_NAME" "$vault_role_payload" >/dev/null
|
|
|
|
vault_role_id=$(vault_request GET "auth/approle/role/$VAULT_ROLE_NAME/role-id" | jq -er '.data.role_id')
|
|
vault_role_secret=$(vault_request POST "auth/approle/role/$VAULT_ROLE_NAME/secret-id" | jq -er '.data.secret_id')
|
|
|
|
echo "Running IAM API integration tests with the Vault backend"
|
|
VGW_IAM_VAULT_ROLE_SECRET="$vault_role_secret" \
|
|
GOCOVERDIR=/tmp/iam.vault.covdata ./versitygw --health /healthz -p :7077 -a user -s pass iam \
|
|
--vault-endpoint-url "$VAULT_ADDR" \
|
|
--vault-auth-method approle \
|
|
--vault-role-id "$vault_role_id" \
|
|
--vault-mount-path "$VAULT_MOUNT_PATH" \
|
|
--vault-secret-storage-path "$VAULT_SECRET_PATH" &
|
|
IAM_VAULT_PID=$!
|
|
wait_for_server "IAM API Vault server" "http://127.0.0.1:7077/healthz" "$IAM_VAULT_PID"
|
|
./versitygw test -a user -s pass -e http://127.0.0.1:7077 iam
|
|
stop_process "$IAM_VAULT_PID"
|
|
IAM_VAULT_PID=""
|
|
|
|
# -----------------------------------------------------------------------------
|
|
# Coverage Reports (Go 1.20+ Runtime Coverage)
|
|
#
|
|
# The IAM servers above were started with GOCOVERDIR=<dir>, which causes Go to
|
|
# write raw coverage artifacts into these directories:
|
|
#
|
|
# /tmp/iam.covdata
|
|
# /tmp/iam.https.covdata
|
|
# /tmp/iam.vault.covdata
|
|
#
|
|
# Generate individual HTTP, HTTPS, and Vault coverage reports with:
|
|
#
|
|
# go tool covdata percent -i=/tmp/iam.covdata
|
|
# go tool covdata percent -i=/tmp/iam.https.covdata
|
|
# go tool covdata percent -i=/tmp/iam.vault.covdata
|
|
#
|
|
# Generate a merged IAM coverage report with:
|
|
#
|
|
# go tool covdata merge \
|
|
# -i=/tmp/iam.covdata,/tmp/iam.https.covdata,/tmp/iam.vault.covdata \
|
|
# -o /tmp/iam.all.covdata
|
|
#
|
|
# go tool covdata percent -i=/tmp/iam.all.covdata
|
|
# go tool covdata textfmt -i=/tmp/iam.all.covdata -o /tmp/iam_profile.txt
|
|
# go tool cover -html=/tmp/iam_profile.txt
|
|
# -----------------------------------------------------------------------------
|