feat: add AWS-compatible standalone IAM service

Closes #1640 Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI. Server usage Start the IAM server with internal file-backed storage: mkdir -p /tmp/versitygw-iam ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam Start the IAM server with Vault KV v2 storage using AppRole: VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates. Configure the AWS CLI credentials used by the IAM server: export AWS_ACCESS_KEY_ID=user export AWS_SECRET_ACCESS_KEY=pass export AWS_DEFAULT_REGION=us-east-1 Implemented IAM actions CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users. aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage GetUser returns a stored user or the root identity when requested without a username through the IAM Query API. aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits. aws --endpoint-url http://127.0.0.1:7070 iam list-users aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100 UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users. aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/ DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users. aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert IAM protocol and authentication - Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests. - Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields. - Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1. - Support both Authorization-header and query-string SigV4 authentication. - Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes. - Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures. Storage implementations - Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts. - Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration. - Introduce a common Storer interface and require exactly one storage backend to be configured. Server and embedding support - Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends. - Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications. Gateway-level internal packages - Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it. - Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors. - Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses. - Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling. - Add `internal/routekit` for shared query, form, and header route matchers. - Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling. - Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures. Testing and CI - Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination. - Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`. - Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior. - Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole. - Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting. - Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
2026-09-25 01:14:22 +00:00 · 2026-08-25 01:03:12 +04:00
parent 762e244b43
commit e012a6fd01
60 changed files with 9108 additions and 629 deletions
@@ -0,0 +1,44 @@
+// Copyright 2026 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package netutil
+
+import (
+	"crypto/tls"
+	"fmt"
+	"sync/atomic"
+)
+
+type CertStorage struct {
+	cert atomic.Pointer[tls.Certificate]
+}
+
+func NewCertStorage() *CertStorage {
+	return &CertStorage{}
+}
+
+func (cs *CertStorage) GetCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error) {
+	return cs.cert.Load(), nil
+}
+
+func (cs *CertStorage) SetCertificate(certFile string, keyFile string) error {
+	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
+	if err != nil {
+		return fmt.Errorf("unable to set certificate: %w", err)
+	}
+
+	cs.cert.Store(&cert)
+
+	return nil
+}
@@ -0,0 +1,316 @@
+// Copyright 2026 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package netutil
+
+import (
+	"crypto/tls"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+)
+
+// MultiListener implements net.Listener and accepts connections from multiple
+// underlying listeners.
+type MultiListener struct {
+	listeners []net.Listener
+	acceptCh  chan acceptResult
+	closeCh   chan struct{}
+	closeOnce sync.Once
+	wg        sync.WaitGroup
+}
+
+type acceptResult struct {
+	conn net.Conn
+	err  error
+}
+
+func NewMultiListener(listeners ...net.Listener) *MultiListener {
+	if len(listeners) == 0 {
+		return nil
+	}
+
+	ml := &MultiListener{
+		listeners: listeners,
+		acceptCh:  make(chan acceptResult, 2*len(listeners)),
+		closeCh:   make(chan struct{}),
+	}
+
+	for _, ln := range listeners {
+		ml.wg.Add(1)
+		go ml.acceptLoop(ln)
+	}
+
+	return ml
+}
+
+func (ml *MultiListener) acceptLoop(ln net.Listener) {
+	defer ml.wg.Done()
+
+	for {
+		conn, err := ln.Accept()
+
+		select {
+		case <-ml.closeCh:
+			if conn != nil {
+				conn.Close()
+			}
+			return
+		case ml.acceptCh <- acceptResult{conn: conn, err: err}:
+			if err != nil {
+				return
+			}
+		}
+	}
+}
+
+func (ml *MultiListener) Accept() (net.Conn, error) {
+	select {
+	case <-ml.closeCh:
+		return nil, errors.New("listener closed")
+	case result, ok := <-ml.acceptCh:
+		if !ok {
+			return nil, errors.New("listener closed")
+		}
+		return result.conn, result.err
+	}
+}
+
+func (ml *MultiListener) Close() error {
+	var errs []error
+
+	ml.closeOnce.Do(func() {
+		close(ml.closeCh)
+
+		for _, ln := range ml.listeners {
+			if err := ln.Close(); err != nil {
+				errs = append(errs, err)
+			}
+		}
+
+		ml.wg.Wait()
+
+		close(ml.acceptCh)
+		for range ml.acceptCh {
+		}
+	})
+
+	if len(errs) > 0 {
+		return fmt.Errorf("errors closing listeners: %v", errs)
+	}
+	return nil
+}
+
+func (ml *MultiListener) Addr() net.Addr {
+	if len(ml.listeners) > 0 {
+		return ml.listeners[0].Addr()
+	}
+	return nil
+}
+
+func IsUnixSocketPath(addr string) bool {
+	_, _, err := net.SplitHostPort(addr)
+	return err != nil
+}
+
+func AbsSocketPaths(addrs []string) ([]string, error) {
+	result := make([]string, len(addrs))
+	for i, addr := range addrs {
+		if strings.HasPrefix(addr, "./") {
+			abs, err := filepath.Abs(addr)
+			if err != nil {
+				return nil, fmt.Errorf("failed to resolve socket path %q: %w", addr, err)
+			}
+			result[i] = abs
+		} else {
+			result[i] = addr
+		}
+	}
+	return result, nil
+}
+
+func isAbstractSocket(addr string) bool {
+	return strings.HasPrefix(addr, "@")
+}
+
+func removeStaleSocket(path string) error {
+	fi, err := os.Stat(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return fmt.Errorf("failed to stat socket path %q: %w", path, err)
+	}
+	if fi.Mode()&os.ModeSocket == 0 {
+		return fmt.Errorf("path %q already exists and is not a socket (mode %s)", path, fi.Mode())
+	}
+	return os.Remove(path)
+}
+
+func ResolveHostnameIPs(address string) ([]string, error) {
+	if IsUnixSocketPath(address) {
+		return []string{address}, nil
+	}
+
+	host, _, err := net.SplitHostPort(address)
+	if err != nil {
+		return nil, fmt.Errorf("invalid address %q: %w", address, err)
+	}
+
+	if host == "" {
+		return []string{""}, nil
+	}
+
+	if net.ParseIP(host) != nil {
+		return []string{host}, nil
+	}
+
+	ips, err := net.LookupIP(host)
+	if err != nil {
+		return nil, fmt.Errorf("failed to resolve hostname %q: %w", host, err)
+	}
+	if len(ips) == 0 {
+		return nil, fmt.Errorf("no addresses found for hostname %q", host)
+	}
+
+	result := make([]string, 0, len(ips))
+	for _, ip := range ips {
+		result = append(result, ip.String())
+	}
+
+	return result, nil
+}
+
+func resolveHostnameAddrs(address string) ([]string, error) {
+	if IsUnixSocketPath(address) {
+		return []string{address}, nil
+	}
+
+	host, port, err := net.SplitHostPort(address)
+	if err != nil {
+		return nil, fmt.Errorf("invalid address %q: %w", address, err)
+	}
+
+	if host == "" || net.ParseIP(host) != nil {
+		return []string{address}, nil
+	}
+
+	ips, err := net.LookupIP(host)
+	if err != nil {
+		return nil, fmt.Errorf("failed to resolve hostname %q: %w", host, err)
+	}
+	if len(ips) == 0 {
+		return nil, fmt.Errorf("no addresses found for hostname %q", host)
+	}
+
+	addrs := make([]string, 0, len(ips))
+	for _, ip := range ips {
+		addrs = append(addrs, net.JoinHostPort(ip.String(), port))
+	}
+
+	return addrs, nil
+}
+
+type ListenerOptions struct {
+	SocketPerm os.FileMode
+}
+
+func NewMultiAddrListener(network, address string, opts ListenerOptions) (net.Listener, error) {
+	if IsUnixSocketPath(address) {
+		if !isAbstractSocket(address) {
+			if err := removeStaleSocket(address); err != nil {
+				return nil, err
+			}
+		}
+		ln, err := net.Listen("unix", address)
+		if err != nil {
+			return nil, fmt.Errorf("failed to bind unix socket listener %s: %w", address, err)
+		}
+		if opts.SocketPerm != 0 && !isAbstractSocket(address) {
+			if err := os.Chmod(address, opts.SocketPerm); err != nil {
+				ln.Close()
+				return nil, fmt.Errorf("failed to set permissions on socket %s: %w", address, err)
+			}
+		}
+		return NewMultiListener(ln), nil
+	}
+
+	addrs, err := resolveHostnameAddrs(address)
+	if err != nil {
+		return nil, err
+	}
+
+	listeners := make([]net.Listener, 0, len(addrs))
+	for _, addr := range addrs {
+		ln, err := net.Listen(network, addr)
+		if err != nil {
+			for _, l := range listeners {
+				l.Close()
+			}
+			return nil, fmt.Errorf("failed to bind listener %s: %w", addr, err)
+		}
+		listeners = append(listeners, ln)
+	}
+
+	return NewMultiListener(listeners...), nil
+}
+
+func NewMultiAddrTLSListener(network, address string, getCertificateFunc func(*tls.ClientHelloInfo) (*tls.Certificate, error), opts ListenerOptions) (net.Listener, error) {
+	config := &tls.Config{
+		MinVersion:     tls.VersionTLS12,
+		GetCertificate: getCertificateFunc,
+	}
+
+	if IsUnixSocketPath(address) {
+		if !isAbstractSocket(address) {
+			if err := removeStaleSocket(address); err != nil {
+				return nil, err
+			}
+		}
+		ln, err := net.Listen("unix", address)
+		if err != nil {
+			return nil, fmt.Errorf("failed to bind unix TLS socket listener %s: %w", address, err)
+		}
+		if opts.SocketPerm != 0 && !isAbstractSocket(address) {
+			if err := os.Chmod(address, opts.SocketPerm); err != nil {
+				ln.Close()
+				return nil, fmt.Errorf("failed to set permissions on socket %s: %w", address, err)
+			}
+		}
+		return NewMultiListener(tls.NewListener(ln, config)), nil
+	}
+
+	addrs, err := resolveHostnameAddrs(address)
+	if err != nil {
+		return nil, err
+	}
+
+	listeners := make([]net.Listener, 0, len(addrs))
+	for _, addr := range addrs {
+		ln, err := net.Listen(network, addr)
+		if err != nil {
+			for _, l := range listeners {
+				l.Close()
+			}
+			return nil, fmt.Errorf("failed to bind TLS listener %s: %w", addr, err)
+		}
+		listeners = append(listeners, tls.NewListener(ln, config))
+	}
+
+	return NewMultiListener(listeners...), nil
+}