mirror of
https://github.com/versity/versitygw.git
synced 2026-09-23 16:34:18 +00:00
feat: add AWS-compatible standalone IAM service
Closes #1640 Add a standalone AWS IAM Query API implementation for managing IAM users through standard AWS SDKs and the AWS CLI. Server usage Start the IAM server with internal file-backed storage: mkdir -p /tmp/versitygw-iam ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --dir /tmp/versitygw-iam Start the IAM server with Vault KV v2 storage using AppRole: VGW_IAM_VAULT_ROLE_SECRET=<role-secret> ./versitygw --port 127.0.0.1:7070 --access user --secret pass iam --vault-endpoint-url http://127.0.0.1:8200 --vault-auth-method approle --vault-role-id <role-id> --vault-mount-path kv --vault-secret-storage-path iam Vault authentication also supports root tokens, separate authentication and secret-storage namespaces, custom mount paths, server certificate validation, and mutual TLS client certificates. Configure the AWS CLI credentials used by the IAM server: export AWS_ACCESS_KEY_ID=user export AWS_SECRET_ACCESS_KEY=pass export AWS_DEFAULT_REGION=us-east-1 Implemented IAM actions CreateUser creates an IAM user with an AWS-compatible ARN, generated AIDA user ID, creation timestamp, optional path, and tags. It validates usernames, paths, tag limits, reserved tag prefixes, duplicate tag keys, and existing users. aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob aws --endpoint-url http://127.0.0.1:7070 iam create-user --user-name bob --path /engineering/ --tags Key=team,Value=storage GetUser returns a stored user or the root identity when requested without a username through the IAM Query API. aws --endpoint-url http://127.0.0.1:7070 iam get-user --user-name bob ListUsers returns users in deterministic username order and supports path filtering, marker-based pagination, and MaxItems limits. aws --endpoint-url http://127.0.0.1:7070 iam list-users aws --endpoint-url http://127.0.0.1:7070 iam list-users --path-prefix /engineering/ --max-items 100 UpdateUser updates the username and/or path, recalculates the user ARN, and rejects conflicts with existing users. aws --endpoint-url http://127.0.0.1:7070 iam update-user --user-name bob --new-user-name robert --new-path /platform/ DeleteUser permanently removes an IAM user and returns AWS-compatible errors for missing users. aws --endpoint-url http://127.0.0.1:7070 iam delete-user --user-name robert IAM protocol and authentication - Support the AWS IAM Query protocol version 2010-05-08 over GET and POST form requests. - Return AWS-compatible XML responses, error documents, status codes, request IDs, user metadata, and pagination fields. - Authenticate root credentials with AWS Signature Version 4 for the IAM service in us-east-1. - Support both Authorization-header and query-string SigV4 authentication. - Validate credential scope, signed headers, timestamps, clock skew, content length, signatures, and unsupported signature or session-token modes. - Add IAM-specific validation and error mapping for malformed requests, invalid actions, duplicate entities, missing users, throttling, and internal failures. Storage implementations - Add an internal JSON-backed store using iam.json and iam.json.backup with atomic temporary-file replacement, concurrent access protection, stable ordering, pagination, and persistence across restarts. - Add a Vault KV v2 store with one secret per user, CAS-based duplicate protection, permanent deletion, AppRole reauthentication, namespace support, configurable authentication and KV mounts, root-token authentication, and TLS/mTLS configuration. - Introduce a common Storer interface and require exactly one storage backend to be configured. Server and embedding support - Register the new `versitygw iam` command with environment-variable and CLI configuration for both storage backends. - Add `embedgw.RunIAMAPI` and `IAMConfig` for embedding the IAM service in Go applications. Gateway-level internal packages - Add `internal/iamstore` as a reusable generic file-backed IAM persistence engine and migrate the existing gateway internal IAM service to it. - Add `internal/sigv4auth` for shared SigV4 header and presigned-query parsing, canonical request generation, signature verification, and structured authentication errors. - Refactor the S3 authentication paths to use the shared SigV4 implementation while preserving S3-specific error responses. - Add `internal/httpctx` for shared Fiber context keys and AWS-style request ID handling. - Add `internal/routekit` for shared query, form, and header route matchers. - Add `internal/netutil` for reusable certificate storage, hostname-aware listeners, multi-address serving, TLS listeners, and UNIX socket handling. - Update the custom SigV4 signer to honor an explicitly supplied signed-header list so unrelated headers do not alter IAM signatures. Testing and CI - Add AWS IAM SDK-based integration coverage for all supported user actions, header authentication, query authentication, validation, errors, filtering, and pagination. - Split standalone IAM tests into `versitygw test iam` and retain existing gateway IAM tests under `versitygw test gw-iam`. - Add unit coverage for controllers, authentication, routing, storage, embedding, listeners, request matching, persistence, and signing behavior. - Add `runiamtests.sh` to exercise internal storage over HTTP and HTTPS plus Vault storage through AppRole. - Add a dedicated IAM functional-test workflow with a Vault service and merged runtime coverage reporting. - Include the IAM test runner in shellcheck and add the AWS IAM SDK dependency.
This commit is contained in:
niksis02
+21
-156
@@ -16,14 +16,11 @@ package auth
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/versity/versitygw/internal/iamstore"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -40,13 +37,10 @@ type IAMServiceInternal struct {
|
||||
// IAM service. All account updates should be sent to a single
|
||||
// gateway instance if possible.
|
||||
sync.RWMutex
|
||||
dir string
|
||||
engine *iamstore.Engine[iAMConfig]
|
||||
rootAcc Account
|
||||
}
|
||||
|
||||
// UpdateAcctFunc accepts the current data and returns the new data to be stored
|
||||
type UpdateAcctFunc func([]byte) ([]byte, error)
|
||||
|
||||
// iAMConfig stores all internal IAM accounts
|
||||
type iAMConfig struct {
|
||||
AccessAccounts map[string]Account `json:"accessAccounts"`
|
||||
@@ -56,16 +50,16 @@ var _ IAMService = &IAMServiceInternal{}
|
||||
|
||||
// NewInternal creates a new instance for the Internal IAM service
|
||||
func NewInternal(rootAcc Account, dir string) (*IAMServiceInternal, error) {
|
||||
i := &IAMServiceInternal{
|
||||
dir: dir,
|
||||
rootAcc: rootAcc,
|
||||
}
|
||||
|
||||
err := i.initIAM()
|
||||
engine, err := iamstore.New(dir, iamFile, iamBackupFile, defaultIAMConfig(), normalizeIAMConfig)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("init iam: %w", err)
|
||||
}
|
||||
|
||||
i := &IAMServiceInternal{
|
||||
engine: engine,
|
||||
rootAcc: rootAcc,
|
||||
}
|
||||
|
||||
return i, nil
|
||||
}
|
||||
|
||||
@@ -79,7 +73,7 @@ func (s *IAMServiceInternal) CreateAccount(account Account) error {
|
||||
s.Lock()
|
||||
defer s.Unlock()
|
||||
|
||||
return s.storeIAM(func(data []byte) ([]byte, error) {
|
||||
return s.engine.StoreIAM(func(data []byte) ([]byte, error) {
|
||||
conf, err := parseIAM(data)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get iam data: %w", err)
|
||||
@@ -110,7 +104,7 @@ func (s *IAMServiceInternal) GetUserAccount(access string) (Account, error) {
|
||||
s.RLock()
|
||||
defer s.RUnlock()
|
||||
|
||||
conf, err := s.getIAM()
|
||||
conf, err := s.engine.GetIAM()
|
||||
if err != nil {
|
||||
return Account{}, fmt.Errorf("get iam data: %w", err)
|
||||
}
|
||||
@@ -129,7 +123,7 @@ func (s *IAMServiceInternal) UpdateUserAccount(access string, props MutableProps
|
||||
s.Lock()
|
||||
defer s.Unlock()
|
||||
|
||||
return s.storeIAM(func(data []byte) ([]byte, error) {
|
||||
return s.engine.StoreIAM(func(data []byte) ([]byte, error) {
|
||||
conf, err := parseIAM(data)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get iam data: %w", err)
|
||||
@@ -158,7 +152,7 @@ func (s *IAMServiceInternal) DeleteUserAccount(access string) error {
|
||||
s.Lock()
|
||||
defer s.Unlock()
|
||||
|
||||
return s.storeIAM(func(data []byte) ([]byte, error) {
|
||||
return s.engine.StoreIAM(func(data []byte) ([]byte, error) {
|
||||
conf, err := parseIAM(data)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get iam data: %w", err)
|
||||
@@ -180,7 +174,7 @@ func (s *IAMServiceInternal) ListUserAccounts() ([]Account, error) {
|
||||
s.RLock()
|
||||
defer s.RUnlock()
|
||||
|
||||
conf, err := s.getIAM()
|
||||
conf, err := s.engine.GetIAM()
|
||||
if err != nil {
|
||||
return []Account{}, fmt.Errorf("get iam data: %w", err)
|
||||
}
|
||||
@@ -211,145 +205,16 @@ func (s *IAMServiceInternal) Shutdown() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
const (
|
||||
iamMode = 0600
|
||||
)
|
||||
|
||||
func (s *IAMServiceInternal) initIAM() error {
|
||||
fname := filepath.Join(s.dir, iamFile)
|
||||
|
||||
_, err := os.ReadFile(fname)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
b, err := json.Marshal(iAMConfig{AccessAccounts: map[string]Account{}})
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal default iam: %w", err)
|
||||
}
|
||||
err = os.WriteFile(fname, b, iamMode)
|
||||
if err != nil {
|
||||
return fmt.Errorf("write default iam: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *IAMServiceInternal) getIAM() (iAMConfig, error) {
|
||||
b, err := s.readIAMData()
|
||||
if err != nil {
|
||||
return iAMConfig{}, err
|
||||
}
|
||||
|
||||
return parseIAM(b)
|
||||
}
|
||||
|
||||
func parseIAM(b []byte) (iAMConfig, error) {
|
||||
var conf iAMConfig
|
||||
if err := json.Unmarshal(b, &conf); err != nil {
|
||||
return iAMConfig{}, fmt.Errorf("failed to parse the config file: %w", err)
|
||||
}
|
||||
return iamstore.ParseIAM(b, normalizeIAMConfig)
|
||||
}
|
||||
|
||||
func defaultIAMConfig() iAMConfig {
|
||||
return iAMConfig{AccessAccounts: map[string]Account{}}
|
||||
}
|
||||
|
||||
func normalizeIAMConfig(conf *iAMConfig) {
|
||||
if conf.AccessAccounts == nil {
|
||||
conf.AccessAccounts = make(map[string]Account)
|
||||
}
|
||||
|
||||
return conf, nil
|
||||
}
|
||||
|
||||
const (
|
||||
backoff = 100 * time.Millisecond
|
||||
maxretry = 300
|
||||
)
|
||||
|
||||
func (s *IAMServiceInternal) readIAMData() ([]byte, error) {
|
||||
// We are going to be racing with other running gateways without any
|
||||
// coordination. So we might find the file does not exist at times.
|
||||
// For this case we need to retry for a while assuming the other gateway
|
||||
// will eventually write the file. If it doesn't after the max retries,
|
||||
// then we will return the error.
|
||||
|
||||
retries := 0
|
||||
|
||||
for {
|
||||
b, err := os.ReadFile(filepath.Join(s.dir, iamFile))
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
// racing with someone else updating
|
||||
// keep retrying after backoff
|
||||
retries++
|
||||
if retries < maxretry {
|
||||
time.Sleep(backoff)
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("read iam file: %w", err)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return b, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (s *IAMServiceInternal) storeIAM(update UpdateAcctFunc) error {
|
||||
// We are going to be racing with other running gateways without any
|
||||
// coordination. So the strategy here is to read the current file data,
|
||||
// update the data, write back out to a temp file, then rename the
|
||||
// temp file to the original file. This rename will replace the
|
||||
// original file with the new file. This is atomic and should always
|
||||
// allow for a consistent view of the data. There is a small
|
||||
// window where the file could be read and then updated by
|
||||
// another process. In this case any updates the other process did
|
||||
// will be lost. This is a limitation of the internal IAM service.
|
||||
// This should be rare, and even when it does happen should result
|
||||
// in a valid IAM file, just without the other process's updates.
|
||||
|
||||
iamFname := filepath.Join(s.dir, iamFile)
|
||||
backupFname := filepath.Join(s.dir, iamBackupFile)
|
||||
|
||||
b, err := os.ReadFile(iamFname)
|
||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return fmt.Errorf("read iam file: %w", err)
|
||||
}
|
||||
|
||||
// save copy of data
|
||||
datacopy := make([]byte, len(b))
|
||||
copy(datacopy, b)
|
||||
|
||||
// make a backup copy in case something happens
|
||||
err = s.writeUsingTempFile(b, backupFname)
|
||||
if err != nil {
|
||||
return fmt.Errorf("write backup iam file: %w", err)
|
||||
}
|
||||
|
||||
b, err = update(b)
|
||||
if err != nil {
|
||||
return fmt.Errorf("update iam data: %w", err)
|
||||
}
|
||||
|
||||
err = s.writeUsingTempFile(b, iamFname)
|
||||
if err != nil {
|
||||
return fmt.Errorf("write iam file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *IAMServiceInternal) writeUsingTempFile(b []byte, fname string) error {
|
||||
f, err := os.CreateTemp(s.dir, iamFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
defer os.Remove(f.Name())
|
||||
|
||||
_, err = f.Write(b)
|
||||
f.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("write temp file: %w", err)
|
||||
}
|
||||
|
||||
err = os.Rename(f.Name(), fname)
|
||||
if err != nil {
|
||||
return fmt.Errorf("rename temp file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user