feat: add AWS-compatible standalone IAM service

Closes #1640 Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI. Server usage Start the IAM server with internal file-backed storage: mkdir -p /tmp/versitygw-iam ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam Start the IAM server with Vault KV v2 storage using AppRole: VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates. Configure the AWS CLI credentials used by the IAM server: export AWS_ACCESS_KEY_ID=user export AWS_SECRET_ACCESS_KEY=pass export AWS_DEFAULT_REGION=us-east-1 Implemented IAM actions CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users. aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage GetUser returns a stored user or the root identity when requested without a username through the IAM Query API. aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits. aws --endpoint-url http://127.0.0.1:7070 iam list-users aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100 UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users. aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/ DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users. aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert IAM protocol and authentication - Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests. - Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields. - Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1. - Support both Authorization-header and query-string SigV4 authentication. - Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes. - Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures. Storage implementations - Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts. - Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration. - Introduce a common Storer interface and require exactly one storage backend to be configured. Server and embedding support - Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends. - Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications. Gateway-level internal packages - Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it. - Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors. - Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses. - Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling. - Add `internal/routekit` for shared query, form, and header route matchers. - Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling. - Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures. Testing and CI - Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination. - Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`. - Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior. - Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole. - Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting. - Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
2026-09-24 08:54:47 +00:00 · 2026-08-25 01:03:12 +04:00
parent 762e244b43
commit e012a6fd01
60 changed files with 9108 additions and 629 deletions
@@ -49,12 +49,15 @@ import (

 	"github.com/aws/aws-sdk-go-v2/aws"
 	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
+	awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http"
+	"github.com/aws/aws-sdk-go-v2/service/iam"
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/aws/smithy-go"
 	"github.com/aws/smithy-go/middleware"
 	smithyhttp "github.com/aws/smithy-go/transport/http"
 	"github.com/cespare/xxhash/v2"
+	"github.com/versity/versitygw/iamapi/iamerr"
 	"github.com/versity/versitygw/s3err"
 	"github.com/zeebo/xxh3"
 	"golang.org/x/sync/errgroup"
@@ -297,6 +300,19 @@ func actionHandlerNoSetup(s *S3Conf, testName string, handler func(s3client *s3.
 	return handlerErr
 }

+func iamActionHandler(s *S3Conf, testName string, handler func(client *iam.Client) error) error {
+	runF(testName)
+
+	err := handler(s.GetIAMClient())
+	if err != nil {
+		failF("%v: %v", testName, err)
+		return fmt.Errorf("%v: %w", testName, err)
+	}
+
+	passF(testName)
+	return nil
+}
+
 type authConfig struct {
 	testName       string
 	path           string
@@ -304,13 +320,26 @@ type authConfig struct {
 	overrideSha256 string
 	body           []byte
 	service        string
+	access         string
+	secret         string
+	region         string
 	date           time.Time
 	headers        map[string]string
 }

 func authHandler(s *S3Conf, cfg *authConfig, handler func(req *http.Request) error) error {
 	runF(cfg.testName)
-	req, err := createSignedReq(cfg.method, s.endpoint, cfg.path, s.awsID, s.awsSecret, cfg.service, s.awsRegion, cfg.overrideSha256, cfg.body, cfg.date, cfg.headers)
+	access, secret, region := s.awsID, s.awsSecret, s.awsRegion
+	if cfg.access != "" {
+		access = cfg.access
+	}
+	if cfg.secret != "" {
+		secret = cfg.secret
+	}
+	if cfg.region != "" {
+		region = cfg.region
+	}
+	req, err := createSignedReq(cfg.method, s.endpoint, cfg.path, access, secret, cfg.service, region, cfg.overrideSha256, cfg.body, cfg.date, cfg.headers)
 	if err != nil {
 		failF("%v: %v", cfg.testName, err)
 		return fmt.Errorf("%v: %w", cfg.testName, err)
@@ -365,7 +394,12 @@ func createSignedReq(method, endpoint, path, access, secret, service, region, ov
 		hexPayload = hex.EncodeToString(hashedPayload[:])
 	}

-	req.Header.Set("X-Amz-Content-Sha256", hexPayload)
+	// x-amz-content-sha256 is an S3 signing header. Other services still use
+	// hexPayload in the canonical request, but should not send or sign this
+	// header unless the caller explicitly supplies it.
+	if service == "s3" {
+		req.Header.Set("X-Amz-Content-Sha256", hexPayload)
+	}
 	for key, val := range headers {
 		req.Header.Add(key, val)
 	}
@@ -431,6 +465,16 @@ type APIErrorResponse struct {
 	HostID                      string                     `xml:"HostId,omitempty"`
 }

+type IAMErrorResponse struct {
+	XMLName xml.Name `xml:"ErrorResponse"`
+	Error   struct {
+		Type    string
+		Code    string
+		Message string
+	}
+	RequestID string `xml:"RequestId"`
+}
+
 func checkHTTPResponseApiErr(resp *http.Response, expected s3err.S3Error) error {
 	apiErr := expected.BaseError()
 	body, err := io.ReadAll(resp.Body)
@@ -452,6 +496,62 @@ func checkHTTPResponseApiErr(resp *http.Response, expected s3err.S3Error) error
 	return compareS3ApiError(expected, &errResp)
 }

+func checkIAMAuthRequest(s *S3Conf, req *http.Request, expected iamerr.APIError) error {
+	resp, err := s.httpClient.Do(req)
+	if err != nil {
+		return err
+	}
+
+	return checkHTTPResponseIAMErr(resp, expected)
+}
+
+func checkHTTPResponseIAMErr(resp *http.Response, expected iamerr.APIError) error {
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+
+	resp.Body.Close()
+
+	var errResp IAMErrorResponse
+	err = xml.Unmarshal(body, &errResp)
+	if err != nil {
+		return err
+	}
+
+	if resp.StatusCode != expected.StatusCode() {
+		return fmt.Errorf("expected response status code to be %v, instead got %v", expected.StatusCode(), resp.StatusCode)
+	}
+	if errResp.XMLName.Space != iamerr.Namespace {
+		return fmt.Errorf("expected IAM error namespace, instead got %q", errResp.XMLName.Space)
+	}
+	if errResp.RequestID == "" {
+		return fmt.Errorf("expected IAM error response request id")
+	}
+
+	expectedBody := expected.XMLBody(errResp.RequestID)
+	if string(body) != string(expectedBody) {
+		return fmt.Errorf("expected IAM error response body to be %q, instead got %q", expectedBody, body)
+	}
+
+	return nil
+}
+
+// isSuccessStatus returns true for 2xx HTTP status codes.
+func isSuccessStatus(statusCode int) bool {
+	return statusCode >= http.StatusOK && statusCode < http.StatusMultipleChoices
+}
+
+func checkIAMSuccess(resp *http.Response) error {
+	defer resp.Body.Close()
+	if !isSuccessStatus(resp.StatusCode) {
+		body, _ := io.ReadAll(resp.Body)
+		return fmt.Errorf("expected response status code to be %v, instead got %v: %s", http.StatusOK, resp.StatusCode, body)
+	}
+
+	return nil
+}
+
 // websiteGet issues a plain HTTP GET to the dedicated website endpoint.
 // The bucket is resolved from the request URL host. No S3 signing is applied.
 func websiteGet(s *S3Conf, bucket, path string, headers map[string]string) (*http.Response, error) {
@@ -799,6 +899,41 @@ func checkSdkApiErr(err error, code string) error {
 	return err
 }

+func checkIAMApiErr(err error, expected iamerr.APIError) error {
+	if err == nil {
+		return fmt.Errorf("expected IAM API error, instead got nil")
+	}
+
+	var apiErr smithy.APIError
+	if !errors.As(err, &apiErr) {
+		return fmt.Errorf("expected IAM API error, instead got: %w", err)
+	}
+
+	expectedErr, ok := expected.(iamerr.Error)
+	if !ok {
+		return fmt.Errorf("expected concrete IAM error, got %T", expected)
+	}
+	if apiErr.ErrorCode() != expectedErr.Code {
+		return fmt.Errorf("expected IAM error code to be %q, instead got %q", expectedErr.Code, apiErr.ErrorCode())
+	}
+	if apiErr.ErrorMessage() != expectedErr.Message {
+		return fmt.Errorf("expected IAM error message to be %q, instead got %q", expectedErr.Message, apiErr.ErrorMessage())
+	}
+
+	var responseErr *awshttp.ResponseError
+	if !errors.As(err, &responseErr) {
+		return fmt.Errorf("expected IAM HTTP response error, instead got: %w", err)
+	}
+	if responseErr.HTTPStatusCode() != expected.StatusCode() {
+		return fmt.Errorf("expected IAM response status code to be %v, instead got %v", expected.StatusCode(), responseErr.HTTPStatusCode())
+	}
+	if responseErr.ServiceRequestID() == "" {
+		return fmt.Errorf("expected IAM error response request id")
+	}
+
+	return nil
+}
+
 func putObjects(client *s3.Client, objs []string, bucket string) ([]types.Object, error) {
 	var contents []types.Object
 	var size int64