Merge pull request #107 from versity/test-cli-setup

Test CLI setup

.github/workflows/go.yml

@@ -8,13 +8,13 @@ jobs:
     steps:
 
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: "1.20"
+        go-version: 'stable'
       id: go
 
     - name: Check out code into the Go module directory
-      uses: actions/checkout@v1
+      uses: actions/checkout@v3
 
     - name: Verify all files pass gofmt formatting
       run: if [ "$(gofmt -s -l . | wc -l)" -gt 0 ]; then gofmt -s -d .; exit 1; fi

CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+versitygw@versity.com.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

README.md

@@ -52,7 +52,8 @@ The global options are specified before the backend type and the backend options
 
 #### Versity gives you clarity and control over your archival storage, so you can allocate more resources to your core mission.
 
-### Contact 
+### Contact
+![versity logo](https://www.versity.com/wp-content/uploads/2022/12/cropped-android-chrome-512x512-1-32x32.png)
 info@versity.com <br />
 +1 844 726 8826
 

auth/acl.go

@@ -0,0 +1,242 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package auth
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/versity/versitygw/s3err"
+)
+
+type ACL struct {
+	ACL      types.BucketCannedACL
+	Owner    string
+	Grantees []Grantee
+}
+
+type Grantee struct {
+	Permission types.Permission
+	Access     string
+}
+
+type GetBucketAclOutput struct {
+	Owner             *types.Owner
+	AccessControlList AccessControlList
+}
+
+type AccessControlList struct {
+	Grants []types.Grant
+}
+
+func ParseACL(data []byte) (ACL, error) {
+	if len(data) == 0 {
+		return ACL{}, nil
+	}
+
+	var acl ACL
+	if err := json.Unmarshal(data, &acl); err != nil {
+		return acl, fmt.Errorf("parse acl: %w", err)
+	}
+	return acl, nil
+}
+
+func ParseACLOutput(data []byte) (GetBucketAclOutput, error) {
+	var acl ACL
+	if err := json.Unmarshal(data, &acl); err != nil {
+		return GetBucketAclOutput{}, fmt.Errorf("parse acl: %w", err)
+	}
+
+	grants := []types.Grant{}
+
+	for _, elem := range acl.Grantees {
+		acs := elem.Access
+		grants = append(grants, types.Grant{Grantee: &types.Grantee{ID: &acs}, Permission: elem.Permission})
+	}
+
+	return GetBucketAclOutput{
+		Owner: &types.Owner{
+			ID: &acl.Owner,
+		},
+		AccessControlList: AccessControlList{
+			Grants: grants,
+		},
+	}, nil
+}
+
+func UpdateACL(input *s3.PutBucketAclInput, acl ACL, iam IAMService) error {
+	if acl.Owner != *input.AccessControlPolicy.Owner.ID {
+		return s3err.GetAPIError(s3err.ErrAccessDenied)
+	}
+
+	// if the ACL is specified, set the ACL, else replace the grantees
+	if input.ACL != "" {
+		acl.ACL = input.ACL
+		acl.Grantees = []Grantee{}
+		return nil
+	}
+
+	grantees := []Grantee{}
+
+	fullControlList, readList, readACPList, writeList, writeACPList := []string{}, []string{}, []string{}, []string{}, []string{}
+
+	if *input.GrantFullControl != "" {
+		fullControlList = splitUnique(*input.GrantFullControl, ",")
+		fmt.Println(fullControlList)
+		for _, str := range fullControlList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "FULL_CONTROL"})
+		}
+	}
+	if *input.GrantRead != "" {
+		readList = splitUnique(*input.GrantRead, ",")
+		for _, str := range readList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "READ"})
+		}
+	}
+	if *input.GrantReadACP != "" {
+		readACPList = splitUnique(*input.GrantReadACP, ",")
+		for _, str := range readACPList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "READ_ACP"})
+		}
+	}
+	if *input.GrantWrite != "" {
+		writeList = splitUnique(*input.GrantWrite, ",")
+		for _, str := range writeList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "WRITE"})
+		}
+	}
+	if *input.GrantWriteACP != "" {
+		writeACPList = splitUnique(*input.GrantWriteACP, ",")
+		for _, str := range writeACPList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "WRITE_ACP"})
+		}
+	}
+
+	accs := append(append(append(append(fullControlList, readList...), writeACPList...), readACPList...), writeList...)
+
+	// Check if the specified accounts exist
+	accList, err := checkIfAccountsExist(accs, iam)
+	if err != nil {
+		return err
+	}
+	if len(accList) > 0 {
+		return fmt.Errorf("accounts does not exist: %s", strings.Join(accList, ", "))
+	}
+
+	acl.Grantees = grantees
+	acl.ACL = ""
+
+	return nil
+}
+
+func checkIfAccountsExist(accs []string, iam IAMService) ([]string, error) {
+	result := []string{}
+
+	for _, acc := range accs {
+		_, err := iam.GetUserAccount(acc)
+		if err != nil && err != ErrNoSuchUser {
+			return nil, fmt.Errorf("check user account: %w", err)
+		}
+		if err == nil {
+			result = append(result, acc)
+		}
+	}
+	return result, nil
+}
+
+func splitUnique(s, divider string) []string {
+	elements := strings.Split(s, divider)
+	uniqueElements := make(map[string]bool)
+	result := make([]string, 0, len(elements))
+
+	for _, element := range elements {
+		if _, ok := uniqueElements[element]; !ok {
+			result = append(result, element)
+			uniqueElements[element] = true
+		}
+	}
+
+	return result
+}
+
+func VerifyACL(acl ACL, bucket, access string, permission types.Permission, isRoot bool) error {
+	if isRoot {
+		return nil
+	}
+
+	if acl.Owner == access {
+		return nil
+	}
+
+	if acl.ACL != "" {
+		if (permission == "READ" || permission == "READ_ACP") && (acl.ACL != "public-read" && acl.ACL != "public-read-write") {
+			return s3err.GetAPIError(s3err.ErrAccessDenied)
+		}
+		if (permission == "WRITE" || permission == "WRITE_ACP") && acl.ACL != "public-read-write" {
+			return s3err.GetAPIError(s3err.ErrAccessDenied)
+		}
+
+		return nil
+	} else {
+		grantee := Grantee{Access: access, Permission: permission}
+		granteeFullCtrl := Grantee{Access: access, Permission: "FULL_CONTROL"}
+
+		isFound := false
+
+		for _, grt := range acl.Grantees {
+			if grt == grantee || grt == granteeFullCtrl {
+				isFound = true
+				break
+			}
+		}
+
+		if isFound {
+			return nil
+		}
+	}
+
+	return s3err.GetAPIError(s3err.ErrAccessDenied)
+}
+
+func IsAdmin(access string, isRoot bool) error {
+	var data IAMConfig
+
+	if isRoot {
+		return nil
+	}
+
+	file, err := os.ReadFile("users.json")
+	if err != nil {
+		return fmt.Errorf("unable to read config file: %w", err)
+	}
+
+	if err := json.Unmarshal(file, &data); err != nil {
+		return err
+	}
+
+	acc, ok := data.AccessAccounts[access]
+	if !ok {
+		return fmt.Errorf("user does not exist")
+	}
+
+	if acc.Role == "admin" {
+		return nil
+	}
+	return fmt.Errorf("only admin users have access to this resource")
+}

auth/iam.go

@@ -0,0 +1,34 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package auth
+
+import (
+	"errors"
+)
+
+// Account is a gateway IAM account
+type Account struct {
+	Secret string `json:"secret"`
+	Role   string `json:"role"`
+}
+
+// IAMService is the interface for all IAM service implementations
+type IAMService interface {
+	CreateAccount(access string, account Account) error
+	GetUserAccount(access string) (Account, error)
+	DeleteUserAccount(access string) error
+}
+
+var ErrNoSuchUser = errors.New("user not found")

auth/iam_internal.go

@@ -0,0 +1,178 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package auth
+
+import (
+	"encoding/json"
+	"fmt"
+	"hash/crc32"
+	"sync"
+)
+
+// IAMServiceInternal manages the internal IAM service
+type IAMServiceInternal struct {
+	storer Storer
+
+	mu     sync.RWMutex
+	accts  IAMConfig
+	serial uint32
+}
+
+// UpdateAcctFunc accepts the current data and returns the new data to be stored
+type UpdateAcctFunc func([]byte) ([]byte, error)
+
+// Storer is the interface to manage the peristent IAM data for the internal
+// IAM service
+type Storer interface {
+	InitIAM() error
+	GetIAM() ([]byte, error)
+	StoreIAM(UpdateAcctFunc) error
+}
+
+// IAMConfig stores all internal IAM accounts
+type IAMConfig struct {
+	AccessAccounts map[string]Account `json:"accessAccounts"`
+}
+
+var _ IAMService = &IAMServiceInternal{}
+
+// NewInternal creates a new instance for the Internal IAM service
+func NewInternal(s Storer) (*IAMServiceInternal, error) {
+	i := &IAMServiceInternal{
+		storer: s,
+	}
+
+	err := i.updateCache()
+	if err != nil {
+		return nil, fmt.Errorf("refresh iam cache: %w", err)
+	}
+
+	return i, nil
+}
+
+// CreateAccount creates a new IAM account. Returns an error if the account
+// already exists.
+func (s *IAMServiceInternal) CreateAccount(access string, account Account) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	return s.storer.StoreIAM(func(data []byte) ([]byte, error) {
+		var conf IAMConfig
+
+		if len(data) > 0 {
+			if err := json.Unmarshal(data, &conf); err != nil {
+				return nil, fmt.Errorf("failed to parse iam: %w", err)
+			}
+		} else {
+			conf.AccessAccounts = make(map[string]Account)
+		}
+
+		_, ok := conf.AccessAccounts[access]
+		if ok {
+			return nil, fmt.Errorf("account already exists")
+		}
+		conf.AccessAccounts[access] = account
+
+		b, err := json.Marshal(s.accts)
+		if err != nil {
+			return nil, fmt.Errorf("failed to serialize iam: %w", err)
+		}
+
+		return b, nil
+	})
+}
+
+// GetUserAccount retrieves account info for the requested user. Returns
+// ErrNoSuchUser if the account does not exist.
+func (s *IAMServiceInternal) GetUserAccount(access string) (Account, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	data, err := s.storer.GetIAM()
+	if err != nil {
+		return Account{}, fmt.Errorf("get iam data: %w", err)
+	}
+
+	serial := crc32.ChecksumIEEE(data)
+	if serial != s.serial {
+		s.mu.RUnlock()
+		err := s.updateCache()
+		s.mu.RLock()
+		if err != nil {
+			return Account{}, fmt.Errorf("refresh iam cache: %w", err)
+		}
+	}
+
+	acct, ok := s.accts.AccessAccounts[access]
+	if !ok {
+		return Account{}, ErrNoSuchUser
+	}
+
+	return acct, nil
+}
+
+// updateCache must be called with no locks held
+func (s *IAMServiceInternal) updateCache() error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	data, err := s.storer.GetIAM()
+	if err != nil {
+		return fmt.Errorf("get iam data: %w", err)
+	}
+
+	serial := crc32.ChecksumIEEE(data)
+
+	if len(data) > 0 {
+		if err := json.Unmarshal(data, &s.accts); err != nil {
+			return fmt.Errorf("failed to parse the config file: %w", err)
+		}
+	} else {
+		s.accts.AccessAccounts = make(map[string]Account)
+	}
+
+	s.serial = serial
+
+	return nil
+}
+
+// DeleteUserAccount deletes the specified user account. Does not check if
+// account exists.
+func (s *IAMServiceInternal) DeleteUserAccount(access string) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	return s.storer.StoreIAM(func(data []byte) ([]byte, error) {
+		if len(data) == 0 {
+			// empty config, do nothing
+			return data, nil
+		}
+
+		var conf IAMConfig
+
+		if err := json.Unmarshal(data, &conf); err != nil {
+			return nil, fmt.Errorf("failed to parse iam: %w", err)
+		}
+
+		delete(conf.AccessAccounts, access)
+
+		b, err := json.Marshal(s.accts)
+		if err != nil {
+			return nil, fmt.Errorf("failed to serialize iam: %w", err)
+		}
+
+		return b, nil
+	})
+}

backend/auth/iam.go

@@ -1,137 +0,0 @@
-// Copyright 2023 Versity Software
-// This file is licensed under the Apache License, Version 2.0
-// (the "License"); you may not use this file except in compliance
-// with the License.  You may obtain a copy of the License at
-//
-//   http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied.  See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package auth
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"sync"
-
-	"github.com/versity/versitygw/s3err"
-)
-
-type Account struct {
-	Secret string `json:"secret"`
-	Role   string `json:"role"`
-	Region string `json:"region"`
-}
-
-type IAMConfig struct {
-	AccessAccounts map[string]Account `json:"accessAccounts"`
-}
-
-type AccountsCache struct {
-	mu       sync.Mutex
-	Accounts map[string]Account
-}
-
-func (c *AccountsCache) getAccount(access string) *Account {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-
-	acc, ok := c.Accounts[access]
-	if !ok {
-		return nil
-	}
-
-	return &acc
-}
-
-func (c *AccountsCache) updateAccounts() error {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-
-	var data IAMConfig
-
-	file, err := os.ReadFile("users.json")
-	if err != nil {
-		return fmt.Errorf("error reading config file: %w", err)
-	}
-
-	if err := json.Unmarshal(file, &data); err != nil {
-		return fmt.Errorf("error parsing the data: %w", err)
-	}
-
-	c.Accounts = data.AccessAccounts
-
-	return nil
-}
-
-type IAMService interface {
-	GetIAMConfig() (*IAMConfig, error)
-	CreateAccount(access string, account *Account) error
-	GetUserAccount(access string) *Account
-}
-
-type IAMServiceUnsupported struct {
-	accCache *AccountsCache
-}
-
-var _ IAMService = &IAMServiceUnsupported{}
-
-func New() IAMService {
-	return &IAMServiceUnsupported{accCache: &AccountsCache{Accounts: map[string]Account{}}}
-}
-
-func (IAMServiceUnsupported) GetIAMConfig() (*IAMConfig, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
-}
-
-func (s IAMServiceUnsupported) CreateAccount(access string, account *Account) error {
-	var data IAMConfig
-
-	file, err := os.ReadFile("users.json")
-	if err != nil {
-		data = IAMConfig{AccessAccounts: map[string]Account{
-			access: *account,
-		}}
-	} else {
-		if err := json.Unmarshal(file, &data); err != nil {
-			return err
-		}
-
-		_, ok := data.AccessAccounts[access]
-		if ok {
-			return fmt.Errorf("user with the given access already exists")
-		}
-
-		data.AccessAccounts[access] = *account
-	}
-
-	updatedJSON, err := json.MarshalIndent(data, "", "  ")
-	if err != nil {
-		return err
-	}
-
-	if err := os.WriteFile("users.json", updatedJSON, 0644); err != nil {
-		return err
-	}
-	return nil
-}
-
-func (s IAMServiceUnsupported) GetUserAccount(access string) *Account {
-	acc := s.accCache.getAccount(access)
-	if acc == nil {
-		err := s.accCache.updateAccounts()
-		if err != nil {
-			return nil
-		}
-
-		return s.accCache.getAccount(access)
-	}
-
-	return acc
-}

backend/backend.go

@@ -30,11 +30,11 @@ type Backend interface {
 	fmt.Stringer
 	Shutdown()
 
-	ListBuckets() (*s3.ListBucketsOutput, error)
+	ListBuckets() (s3response.ListAllMyBucketsResult, error)
 	HeadBucket(bucket string) (*s3.HeadBucketOutput, error)
-	GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error)
-	PutBucket(bucket string) error
-	PutBucketAcl(*s3.PutBucketAclInput) error
+	GetBucketAcl(bucket string) ([]byte, error)
+	PutBucket(bucket, owner string) error
+	PutBucketAcl(bucket string, data []byte) error
 	DeleteBucket(bucket string) error
 
 	CreateMultipartUpload(*s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error)
@@ -44,6 +44,7 @@ type Backend interface {
 	ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error)
 	CopyPart(srcBucket, srcObject, DstBucket, uploadID, rangeHeader string, part int) (*types.CopyPartResult, error)
 	PutObjectPart(bucket, object, uploadID string, part int, length int64, r io.Reader) (etag string, err error)
+	UploadPartCopy(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
 	PutObject(*s3.PutObjectInput) (string, error)
 	HeadObject(bucket, object string) (*s3.HeadObjectOutput, error)
@@ -57,8 +58,6 @@ type Backend interface {
 	DeleteObjects(bucket string, objects *s3.DeleteObjectsInput) error
 	PutObjectAcl(*s3.PutObjectAclInput) error
 	RestoreObject(bucket, object string, restoreRequest *s3.RestoreObjectInput) error
-	UploadPart(bucket, object, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error)
-	UploadPartCopy(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
 	GetTags(bucket, object string) (map[string]string, error)
 	SetTags(bucket, object string, tags map[string]string) error
@@ -76,10 +75,10 @@ func (BackendUnsupported) Shutdown() {}
 func (BackendUnsupported) String() string {
 	return "Unsupported"
 }
-func (BackendUnsupported) ListBuckets() (*s3.ListBucketsOutput, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+func (BackendUnsupported) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
+	return s3response.ListAllMyBucketsResult{}, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) PutBucketAcl(*s3.PutBucketAclInput) error {
+func (BackendUnsupported) PutBucketAcl(bucket string, data []byte) error {
 	return s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) PutObjectAcl(*s3.PutObjectAclInput) error {
@@ -91,16 +90,13 @@ func (BackendUnsupported) RestoreObject(bucket, object string, restoreRequest *s
 func (BackendUnsupported) UploadPartCopy(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) UploadPart(bucket, object, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
-}
-func (BackendUnsupported) GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error) {
+func (BackendUnsupported) GetBucketAcl(bucket string) ([]byte, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) HeadBucket(bucket string) (*s3.HeadBucketOutput, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) PutBucket(bucket string) error {
+func (BackendUnsupported) PutBucket(bucket, owner string) error {
 	return s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) DeleteBucket(bucket string) error {

backend/backend_moq_test.go

@@ -45,7 +45,7 @@ var _ Backend = &BackendMock{}
 //			DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
 //				panic("mock out the DeleteObjects method")
 //			},
-//			GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
+//			GetBucketAclFunc: func(bucket string) ([]byte, error) {
 //				panic("mock out the GetBucketAcl method")
 //			},
 //			GetObjectFunc: func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
@@ -66,7 +66,7 @@ var _ Backend = &BackendMock{}
 //			HeadObjectFunc: func(bucket string, object string) (*s3.HeadObjectOutput, error) {
 //				panic("mock out the HeadObject method")
 //			},
-//			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
+//			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
 //				panic("mock out the ListBuckets method")
 //			},
 //			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
@@ -81,10 +81,10 @@ var _ Backend = &BackendMock{}
 //			ListObjectsV2Func: func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
 //				panic("mock out the ListObjectsV2 method")
 //			},
-//			PutBucketFunc: func(bucket string) error {
+//			PutBucketFunc: func(bucket string, owner string) error {
 //				panic("mock out the PutBucket method")
 //			},
-//			PutBucketAclFunc: func(putBucketAclInput *s3.PutBucketAclInput) error {
+//			PutBucketAclFunc: func(bucket string, data []byte) error {
 //				panic("mock out the PutBucketAcl method")
 //			},
 //			PutObjectFunc: func(putObjectInput *s3.PutObjectInput) (string, error) {
@@ -111,9 +111,6 @@ var _ Backend = &BackendMock{}
 //			StringFunc: func() string {
 //				panic("mock out the String method")
 //			},
-//			UploadPartFunc: func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-//				panic("mock out the UploadPart method")
-//			},
 //			UploadPartCopyFunc: func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 //				panic("mock out the UploadPartCopy method")
 //			},
@@ -149,7 +146,7 @@ type BackendMock struct {
 	DeleteObjectsFunc func(bucket string, objects *s3.DeleteObjectsInput) error
 
 	// GetBucketAclFunc mocks the GetBucketAcl method.
-	GetBucketAclFunc func(bucket string) (*s3.GetBucketAclOutput, error)
+	GetBucketAclFunc func(bucket string) ([]byte, error)
 
 	// GetObjectFunc mocks the GetObject method.
 	GetObjectFunc func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error)
@@ -170,7 +167,7 @@ type BackendMock struct {
 	HeadObjectFunc func(bucket string, object string) (*s3.HeadObjectOutput, error)
 
 	// ListBucketsFunc mocks the ListBuckets method.
-	ListBucketsFunc func() (*s3.ListBucketsOutput, error)
+	ListBucketsFunc func() (s3response.ListAllMyBucketsResult, error)
 
 	// ListMultipartUploadsFunc mocks the ListMultipartUploads method.
 	ListMultipartUploadsFunc func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error)
@@ -185,10 +182,10 @@ type BackendMock struct {
 	ListObjectsV2Func func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error)
 
 	// PutBucketFunc mocks the PutBucket method.
-	PutBucketFunc func(bucket string) error
+	PutBucketFunc func(bucket string, owner string) error
 
 	// PutBucketAclFunc mocks the PutBucketAcl method.
-	PutBucketAclFunc func(putBucketAclInput *s3.PutBucketAclInput) error
+	PutBucketAclFunc func(bucket string, data []byte) error
 
 	// PutObjectFunc mocks the PutObject method.
 	PutObjectFunc func(putObjectInput *s3.PutObjectInput) (string, error)
@@ -214,9 +211,6 @@ type BackendMock struct {
 	// StringFunc mocks the String method.
 	StringFunc func() string
 
-	// UploadPartFunc mocks the UploadPart method.
-	UploadPartFunc func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error)
-
 	// UploadPartCopyFunc mocks the UploadPartCopy method.
 	UploadPartCopyFunc func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
@@ -390,11 +384,15 @@ type BackendMock struct {
 		PutBucket []struct {
 			// Bucket is the bucket argument value.
 			Bucket string
+			// Owner is the owner argument value.
+			Owner string
 		}
 		// PutBucketAcl holds details about calls to the PutBucketAcl method.
 		PutBucketAcl []struct {
-			// PutBucketAclInput is the putBucketAclInput argument value.
-			PutBucketAclInput *s3.PutBucketAclInput
+			// Bucket is the bucket argument value.
+			Bucket string
+			// Data is the data argument value.
+			Data []byte
 		}
 		// PutObject holds details about calls to the PutObject method.
 		PutObject []struct {
@@ -452,17 +450,6 @@ type BackendMock struct {
 		// String holds details about calls to the String method.
 		String []struct {
 		}
-		// UploadPart holds details about calls to the UploadPart method.
-		UploadPart []struct {
-			// Bucket is the bucket argument value.
-			Bucket string
-			// Object is the object argument value.
-			Object string
-			// UploadId is the uploadId argument value.
-			UploadId string
-			// Body is the Body argument value.
-			Body io.ReadSeeker
-		}
 		// UploadPartCopy holds details about calls to the UploadPartCopy method.
 		UploadPartCopy []struct {
 			// UploadPartCopyInput is the uploadPartCopyInput argument value.
@@ -499,7 +486,6 @@ type BackendMock struct {
 	lockSetTags                 sync.RWMutex
 	lockShutdown                sync.RWMutex
 	lockString                  sync.RWMutex
-	lockUploadPart              sync.RWMutex
 	lockUploadPartCopy          sync.RWMutex
 }
 
@@ -812,7 +798,7 @@ func (mock *BackendMock) DeleteObjectsCalls() []struct {
 }
 
 // GetBucketAcl calls GetBucketAclFunc.
-func (mock *BackendMock) GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error) {
+func (mock *BackendMock) GetBucketAcl(bucket string) ([]byte, error) {
 	if mock.GetBucketAclFunc == nil {
 		panic("BackendMock.GetBucketAclFunc: method is nil but Backend.GetBucketAcl was just called")
 	}
@@ -1068,7 +1054,7 @@ func (mock *BackendMock) HeadObjectCalls() []struct {
 }
 
 // ListBuckets calls ListBucketsFunc.
-func (mock *BackendMock) ListBuckets() (*s3.ListBucketsOutput, error) {
+func (mock *BackendMock) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
 	if mock.ListBucketsFunc == nil {
 		panic("BackendMock.ListBucketsFunc: method is nil but Backend.ListBuckets was just called")
 	}
@@ -1271,19 +1257,21 @@ func (mock *BackendMock) ListObjectsV2Calls() []struct {
 }
 
 // PutBucket calls PutBucketFunc.
-func (mock *BackendMock) PutBucket(bucket string) error {
+func (mock *BackendMock) PutBucket(bucket string, owner string) error {
 	if mock.PutBucketFunc == nil {
 		panic("BackendMock.PutBucketFunc: method is nil but Backend.PutBucket was just called")
 	}
 	callInfo := struct {
 		Bucket string
+		Owner  string
 	}{
 		Bucket: bucket,
+		Owner:  owner,
 	}
 	mock.lockPutBucket.Lock()
 	mock.calls.PutBucket = append(mock.calls.PutBucket, callInfo)
 	mock.lockPutBucket.Unlock()
-	return mock.PutBucketFunc(bucket)
+	return mock.PutBucketFunc(bucket, owner)
 }
 
 // PutBucketCalls gets all the calls that were made to PutBucket.
@@ -1292,9 +1280,11 @@ func (mock *BackendMock) PutBucket(bucket string) error {
 //	len(mockedBackend.PutBucketCalls())
 func (mock *BackendMock) PutBucketCalls() []struct {
 	Bucket string
+	Owner  string
 } {
 	var calls []struct {
 		Bucket string
+		Owner  string
 	}
 	mock.lockPutBucket.RLock()
 	calls = mock.calls.PutBucket
@@ -1303,19 +1293,21 @@ func (mock *BackendMock) PutBucketCalls() []struct {
 }
 
 // PutBucketAcl calls PutBucketAclFunc.
-func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) error {
+func (mock *BackendMock) PutBucketAcl(bucket string, data []byte) error {
 	if mock.PutBucketAclFunc == nil {
 		panic("BackendMock.PutBucketAclFunc: method is nil but Backend.PutBucketAcl was just called")
 	}
 	callInfo := struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}{
-		PutBucketAclInput: putBucketAclInput,
+		Bucket: bucket,
+		Data:   data,
 	}
 	mock.lockPutBucketAcl.Lock()
 	mock.calls.PutBucketAcl = append(mock.calls.PutBucketAcl, callInfo)
 	mock.lockPutBucketAcl.Unlock()
-	return mock.PutBucketAclFunc(putBucketAclInput)
+	return mock.PutBucketAclFunc(bucket, data)
 }
 
 // PutBucketAclCalls gets all the calls that were made to PutBucketAcl.
@@ -1323,10 +1315,12 @@ func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) e
 //
 //	len(mockedBackend.PutBucketAclCalls())
 func (mock *BackendMock) PutBucketAclCalls() []struct {
-	PutBucketAclInput *s3.PutBucketAclInput
+	Bucket string
+	Data   []byte
 } {
 	var calls []struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}
 	mock.lockPutBucketAcl.RLock()
 	calls = mock.calls.PutBucketAcl
@@ -1620,50 +1614,6 @@ func (mock *BackendMock) StringCalls() []struct {
 	return calls
 }
 
-// UploadPart calls UploadPartFunc.
-func (mock *BackendMock) UploadPart(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-	if mock.UploadPartFunc == nil {
-		panic("BackendMock.UploadPartFunc: method is nil but Backend.UploadPart was just called")
-	}
-	callInfo := struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}{
-		Bucket:   bucket,
-		Object:   object,
-		UploadId: uploadId,
-		Body:     Body,
-	}
-	mock.lockUploadPart.Lock()
-	mock.calls.UploadPart = append(mock.calls.UploadPart, callInfo)
-	mock.lockUploadPart.Unlock()
-	return mock.UploadPartFunc(bucket, object, uploadId, Body)
-}
-
-// UploadPartCalls gets all the calls that were made to UploadPart.
-// Check the length with:
-//
-//	len(mockedBackend.UploadPartCalls())
-func (mock *BackendMock) UploadPartCalls() []struct {
-	Bucket   string
-	Object   string
-	UploadId string
-	Body     io.ReadSeeker
-} {
-	var calls []struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}
-	mock.lockUploadPart.RLock()
-	calls = mock.calls.UploadPart
-	mock.lockUploadPart.RUnlock()
-	return calls
-}
-
 // UploadPartCopy calls UploadPartCopyFunc.
 func (mock *BackendMock) UploadPartCopy(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 	if mock.UploadPartCopyFunc == nil {

backend/backend_test.go

@@ -18,10 +18,9 @@ import (
 	"context"
 	"testing"
 
-	"github.com/aws/aws-sdk-go-v2/aws"
 	"github.com/aws/aws-sdk-go-v2/service/s3"
-	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/versity/versitygw/s3err"
+	"github.com/versity/versitygw/s3response"
 )
 
 func TestBackend_ListBuckets(t *testing.T) {
@@ -38,11 +37,13 @@ func TestBackend_ListBuckets(t *testing.T) {
 	tests = append(tests, test{
 		name: "list-Bucket",
 		c: &BackendMock{
-			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
-				return &s3.ListBucketsOutput{
-					Buckets: []types.Bucket{
-						{
-							Name: aws.String("t1"),
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{
+					Buckets: s3response.ListAllMyBucketsList{
+						Bucket: []s3response.ListAllMyBucketsEntry{
+							{
+								Name: "t1",
+							},
 						},
 					},
 				}, s3err.GetAPIError(0)
@@ -55,8 +56,8 @@ func TestBackend_ListBuckets(t *testing.T) {
 	}, test{
 		name: "list-Bucket-error",
 		c: &BackendMock{
-			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
-				return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{}, s3err.GetAPIError(s3err.ErrNotImplemented)
 			},
 		},
 		args: args{
@@ -120,7 +121,7 @@ func TestBackend_GetBucketAcl(t *testing.T) {
 	tests = append(tests, test{
 		name: "get bucket acl error",
 		c: &BackendMock{
-			GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
 				return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 			},
 		},
@@ -140,8 +141,9 @@ func TestBackend_GetBucketAcl(t *testing.T) {
 }
 func TestBackend_PutBucket(t *testing.T) {
 	type args struct {
-		ctx        context.Context
-		bucketName string
+		ctx         context.Context
+		bucketName  string
+		bucketOwner string
 	}
 	type test struct {
 		name    string
@@ -153,31 +155,33 @@ func TestBackend_PutBucket(t *testing.T) {
 	tests = append(tests, test{
 		name: "put bucket ",
 		c: &BackendMock{
-			PutBucketFunc: func(bucket string) error {
+			PutBucketFunc: func(bucket, owner string) error {
 				return s3err.GetAPIError(0)
 			},
 		},
 		args: args{
-			ctx:        context.Background(),
-			bucketName: "b1",
+			ctx:         context.Background(),
+			bucketName:  "b1",
+			bucketOwner: "owner",
 		},
 		wantErr: false,
 	}, test{
 		name: "put bucket error",
 		c: &BackendMock{
-			PutBucketFunc: func(bucket string) error {
+			PutBucketFunc: func(bucket, owner string) error {
 				return s3err.GetAPIError(s3err.ErrNotImplemented)
 			},
 		},
 		args: args{
-			ctx:        context.Background(),
-			bucketName: "b2",
+			ctx:         context.Background(),
+			bucketName:  "b2",
+			bucketOwner: "owner",
 		},
 		wantErr: true,
 	})
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			if err := tt.c.PutBucket(tt.args.bucketName); (err.(s3err.APIError).Code != "") != tt.wantErr {
+			if err := tt.c.PutBucket(tt.args.bucketName, tt.args.bucketOwner); (err.(s3err.APIError).Code != "") != tt.wantErr {
 				t.Errorf("Backend.PutBucket() error = %v, wantErr %v", err, tt.wantErr)
 			}
 		})

backend/common.go

@@ -15,13 +15,17 @@
 package backend
 
 import (
+	"crypto/md5"
+	"encoding/hex"
 	"errors"
+	"fmt"
 	"io/fs"
 	"strconv"
 	"strings"
 	"time"
 
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/versity/versitygw/s3response"
 )
 
 var (
@@ -31,11 +35,11 @@ var (
 
 func IsValidBucketName(name string) bool { return true }
 
-type ByBucketName []types.Bucket
+type ByBucketName []s3response.ListAllMyBucketsEntry
 
 func (d ByBucketName) Len() int           { return len(d) }
 func (d ByBucketName) Swap(i, j int)      { d[i], d[j] = d[j], d[i] }
-func (d ByBucketName) Less(i, j int) bool { return *d[i].Name < *d[j].Name }
+func (d ByBucketName) Less(i, j int) bool { return d[i].Name < d[j].Name }
 
 type ByObjectName []types.Object
 
@@ -83,3 +87,25 @@ func ParseRange(file fs.FileInfo, acceptRange string) (int64, int64, error) {
 
 	return int64(startOffset), int64(endOffset - startOffset + 1), nil
 }
+
+func GetMultipartMD5(parts []types.Part) string {
+	var partsEtagBytes []byte
+	for _, part := range parts {
+		partsEtagBytes = append(partsEtagBytes, getEtagBytes(*part.ETag)...)
+	}
+	s3MD5 := fmt.Sprintf("%s-%d", md5String(partsEtagBytes), len(parts))
+	return s3MD5
+}
+
+func getEtagBytes(etag string) []byte {
+	decode, err := hex.DecodeString(strings.ReplaceAll(etag, string('"'), ""))
+	if err != nil {
+		return []byte(etag)
+	}
+	return decode
+}
+
+func md5String(data []byte) string {
+	sum := md5.Sum(data)
+	return hex.EncodeToString(sum[:])
+}

backend/posix/posix.go

@@ -28,31 +28,50 @@ import (
 	"sort"
 	"strconv"
 	"strings"
+	"sync"
 	"syscall"
+	"time"
 
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/google/uuid"
 	"github.com/pkg/xattr"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3err"
 	"github.com/versity/versitygw/s3response"
 )
 
 type Posix struct {
+	backend.BackendUnsupported
+
 	rootfd  *os.File
 	rootdir string
-	backend.BackendUnsupported
+
+	mu        sync.RWMutex
+	iamcache  []byte
+	iamvalid  bool
+	iamexpire time.Time
 }
 
 var _ backend.Backend = &Posix{}
 
+var (
+	cacheDuration = 5 * time.Minute
+)
+
 const (
 	metaTmpDir          = ".sgwtmp"
 	metaTmpMultipartDir = metaTmpDir + "/multipart"
 	onameAttr           = "user.objname"
 	tagHdr              = "X-Amz-Tagging"
+	contentTypeHdr      = "content-type"
+	contentEncHdr       = "content-encoding"
 	emptyMD5            = "d41d8cd98f00b204e9800998ecf8427e"
+	iamFile             = "users.json"
+	iamBackupFile       = "users.json.backup"
+	aclkey              = "user.acl"
+	etagkey             = "user.etag"
 )
 
 func New(rootdir string) (*Posix, error) {
@@ -73,17 +92,18 @@ func (p *Posix) Shutdown() {
 	p.rootfd.Close()
 }
 
-func (p *Posix) Sring() string {
+func (p *Posix) String() string {
 	return "Posix Gateway"
 }
 
-func (p *Posix) ListBuckets() (*s3.ListBucketsOutput, error) {
+func (p *Posix) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
 	entries, err := os.ReadDir(".")
 	if err != nil {
-		return nil, fmt.Errorf("readdir buckets: %w", err)
+		return s3response.ListAllMyBucketsResult{},
+			fmt.Errorf("readdir buckets: %w", err)
 	}
 
-	var buckets []types.Bucket
+	var buckets []s3response.ListAllMyBucketsEntry
 	for _, entry := range entries {
 		if !entry.IsDir() {
 			// buckets must be a directory
@@ -96,16 +116,18 @@ func (p *Posix) ListBuckets() (*s3.ListBucketsOutput, error) {
 			continue
 		}
 
-		buckets = append(buckets, types.Bucket{
-			Name:         backend.GetStringPtr(entry.Name()),
-			CreationDate: backend.GetTimePtr(fi.ModTime()),
+		buckets = append(buckets, s3response.ListAllMyBucketsEntry{
+			Name:         entry.Name(),
+			CreationDate: fi.ModTime(),
 		})
 	}
 
 	sort.Sort(backend.ByBucketName(buckets))
 
-	return &s3.ListBucketsOutput{
-		Buckets: buckets,
+	return s3response.ListAllMyBucketsResult{
+		Buckets: s3response.ListAllMyBucketsList{
+			Bucket: buckets,
+		},
 	}, nil
 }
 
@@ -121,7 +143,7 @@ func (p *Posix) HeadBucket(bucket string) (*s3.HeadBucketOutput, error) {
 	return &s3.HeadBucketOutput{}, nil
 }
 
-func (p *Posix) PutBucket(bucket string) error {
+func (p *Posix) PutBucket(bucket string, owner string) error {
 	err := os.Mkdir(bucket, 0777)
 	if err != nil && os.IsExist(err) {
 		return s3err.GetAPIError(s3err.ErrBucketAlreadyExists)
@@ -130,6 +152,16 @@ func (p *Posix) PutBucket(bucket string) error {
 		return fmt.Errorf("mkdir bucket: %w", err)
 	}
 
+	acl := auth.ACL{ACL: "private", Owner: owner, Grantees: []auth.Grantee{}}
+	jsonACL, err := json.Marshal(acl)
+	if err != nil {
+		return fmt.Errorf("marshal acl: %w", err)
+	}
+
+	if err := xattr.Set(bucket, aclkey, jsonACL); err != nil {
+		return fmt.Errorf("set acl: %w", err)
+	}
+
 	return nil
 }
 
@@ -249,7 +281,7 @@ func (p *Posix) CompleteMultipartUpload(bucket, object, uploadID string, parts [
 			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
 		}
 
-		b, err := xattr.Get(partPath, "user.etag")
+		b, err := xattr.Get(partPath, etagkey)
 		etag := string(b)
 		if err != nil {
 			etag = ""
@@ -303,9 +335,9 @@ func (p *Posix) CompleteMultipartUpload(bucket, object, uploadID string, parts [
 	}
 
 	// Calculate s3 compatible md5sum for complete multipart.
-	s3MD5 := getMultipartMD5(parts)
+	s3MD5 := backend.GetMultipartMD5(parts)
 
-	err = xattr.Set(objname, "user.etag", []byte(s3MD5))
+	err = xattr.Set(objname, etagkey, []byte(s3MD5))
 	if err != nil {
 		// cleanup object if returning error
 		os.Remove(objname)
@@ -359,22 +391,22 @@ func loadUserMetaData(path string, m map[string]string) (contentType, contentEnc
 		m[strings.TrimPrefix(e, "user.")] = string(b)
 	}
 
-	b, err := xattr.Get(path, "user.content-type")
+	b, err := xattr.Get(path, "user."+contentTypeHdr)
 	contentType = string(b)
 	if err != nil {
 		contentType = ""
 	}
 	if contentType != "" {
-		m["content-type"] = contentType
+		m[contentTypeHdr] = contentType
 	}
 
-	b, err = xattr.Get(path, "user.content-encoding")
+	b, err = xattr.Get(path, "user."+contentEncHdr)
 	contentEncoding = string(b)
 	if err != nil {
 		contentEncoding = ""
 	}
 	if contentEncoding != "" {
-		m["content-encoding"] = contentEncoding
+		m[contentEncHdr] = contentEncoding
 	}
 
 	return
@@ -390,8 +422,8 @@ func isValidMeta(val string) bool {
 	return false
 }
 
-// mkdirAll is similar to os.MkdirAll but it will also set uid/gid when
-// making new directories
+// mkdirAll is similar to os.MkdirAll but it will return ErrObjectParentIsFile
+// when appropriate
 func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
 	// Fast path: if we can tell whether path is a directory or file, stop with success or error.
 	dir, err := os.Stat(path)
@@ -399,7 +431,7 @@ func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
 		if dir.IsDir() {
 			return nil
 		}
-		return &os.PathError{Op: "mkdir", Path: path, Err: syscall.ENOTDIR}
+		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
 	}
 
 	// Slow path: make sure parent exists and then call Mkdir for path.
@@ -435,28 +467,6 @@ func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
 	return nil
 }
 
-func getMultipartMD5(parts []types.Part) string {
-	var partsEtagBytes []byte
-	for _, part := range parts {
-		partsEtagBytes = append(partsEtagBytes, getEtagBytes(*part.ETag)...)
-	}
-	s3MD5 := fmt.Sprintf("%s-%d", md5String(partsEtagBytes), len(parts))
-	return s3MD5
-}
-
-func getEtagBytes(etag string) []byte {
-	decode, err := hex.DecodeString(strings.ReplaceAll(etag, string('"'), ""))
-	if err != nil {
-		return []byte(etag)
-	}
-	return decode
-}
-
-func md5String(data []byte) string {
-	sum := md5.Sum(data)
-	return hex.EncodeToString(sum[:])
-}
-
 func (p *Posix) AbortMultipartUpload(mpu *s3.AbortMultipartUploadInput) error {
 	bucket := *mpu.Bucket
 	object := *mpu.Key
@@ -634,7 +644,7 @@ func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarke
 		}
 
 		partPath := filepath.Join(objdir, uploadID, e.Name())
-		b, err := xattr.Get(partPath, "user.etag")
+		b, err := xattr.Get(partPath, etagkey)
 		etag := string(b)
 		if err != nil {
 			etag = ""
@@ -721,7 +731,7 @@ func (p *Posix) PutObjectPart(bucket, object, uploadID string, part int, length
 
 	dataSum := hash.Sum(nil)
 	etag := hex.EncodeToString(dataSum)
-	xattr.Set(partPath, "user.etag", []byte(etag))
+	xattr.Set(partPath, etagkey, []byte(etag))
 
 	return etag, nil
 }
@@ -749,7 +759,7 @@ func (p *Posix) PutObject(po *s3.PutObjectInput) (string, error) {
 		}
 
 		// set etag attribute to signify this dir was specifically put
-		xattr.Set(name, "user.etag", []byte(emptyMD5))
+		xattr.Set(name, etagkey, []byte(emptyMD5))
 
 		return emptyMD5, nil
 	}
@@ -787,7 +797,7 @@ func (p *Posix) PutObject(po *s3.PutObjectInput) (string, error) {
 
 	dataSum := hash.Sum(nil)
 	etag := hex.EncodeToString(dataSum[:])
-	xattr.Set(name, "user.etag", []byte(etag))
+	xattr.Set(name, etagkey, []byte(etag))
 
 	return etag, nil
 }
@@ -824,11 +834,15 @@ func (p *Posix) removeParents(bucket, object string) error {
 		parent := filepath.Dir(objPath)
 
 		if filepath.Base(parent) == bucket {
+			// stop removing parents if we hit the bucket directory.
 			break
 		}
 
-		_, err := xattr.Get(parent, "user.etag")
+		_, err := xattr.Get(parent, etagkey)
 		if err == nil {
+			// a directory with a valid etag means this was specifically
+			// uploaded with a put object, so stop here and leave this
+			// directory in place.
 			break
 		}
 
@@ -901,7 +915,7 @@ func (p *Posix) GetObject(bucket, object, acceptRange string, writer io.Writer)
 
 	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
 
-	b, err := xattr.Get(objPath, "user.etag")
+	b, err := xattr.Get(objPath, etagkey)
 	etag := string(b)
 	if err != nil {
 		etag = ""
@@ -945,7 +959,7 @@ func (p *Posix) HeadObject(bucket, object string) (*s3.HeadObjectOutput, error)
 	userMetaData := make(map[string]string)
 	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
 
-	b, err := xattr.Get(objPath, "user.etag")
+	b, err := xattr.Get(objPath, etagkey)
 	etag := string(b)
 	if err != nil {
 		etag = ""
@@ -1017,19 +1031,7 @@ func (p *Posix) ListObjects(bucket, prefix, marker, delim string, maxkeys int) (
 
 	fileSystem := os.DirFS(bucket)
 	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
-		func(path string) (bool, error) {
-			_, err := xattr.Get(filepath.Join(bucket, path), "user.etag")
-			if isNoAttr(err) {
-				return false, nil
-			}
-			if err != nil {
-				return false, err
-			}
-			return true, nil
-		}, func(path string) (string, error) {
-			etag, err := xattr.Get(filepath.Join(bucket, path), "user.etag")
-			return string(etag), err
-		}, []string{metaTmpDir})
+		fileToObj(bucket), []string{metaTmpDir})
 	if err != nil {
 		return nil, fmt.Errorf("walk %v: %w", bucket, err)
 	}
@@ -1047,6 +1049,67 @@ func (p *Posix) ListObjects(bucket, prefix, marker, delim string, maxkeys int) (
 	}, nil
 }
 
+func fileToObj(bucket string) backend.GetObjFunc {
+	return func(path string, d fs.DirEntry) (types.Object, error) {
+		if d.IsDir() {
+			// directory object only happens if directory empty
+			// check to see if this is a directory object by checking etag
+			etagBytes, err := xattr.Get(filepath.Join(bucket, path), etagkey)
+			if isNoAttr(err) || errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get etag: %w", err)
+			}
+			etag := string(etagBytes)
+
+			fi, err := d.Info()
+			if errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+			}
+
+			key := path + "/"
+
+			return types.Object{
+				ETag:         &etag,
+				Key:          &key,
+				LastModified: backend.GetTimePtr(fi.ModTime()),
+			}, nil
+		}
+
+		// file object, get object info and fill out object data
+		etagBytes, err := xattr.Get(filepath.Join(bucket, path), etagkey)
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil && !isNoAttr(err) {
+			return types.Object{}, fmt.Errorf("get etag: %w", err)
+		}
+		// note: isNoAttr(err) will return etagBytes = []byte{}
+		// so this will just set etag to "" if its not already set
+
+		etag := string(etagBytes)
+
+		fi, err := d.Info()
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil {
+			return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+		}
+
+		return types.Object{
+			ETag:         &etag,
+			Key:          &path,
+			LastModified: backend.GetTimePtr(fi.ModTime()),
+			Size:         fi.Size(),
+		}, nil
+	}
+}
+
 func (p *Posix) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
 	_, err := os.Stat(bucket)
 	if errors.Is(err, fs.ErrNotExist) {
@@ -1058,19 +1121,7 @@ func (p *Posix) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int)
 
 	fileSystem := os.DirFS(bucket)
 	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
-		func(path string) (bool, error) {
-			_, err := xattr.Get(filepath.Join(bucket, path), "user.etag")
-			if isNoAttr(err) {
-				return false, nil
-			}
-			if err != nil {
-				return false, err
-			}
-			return true, nil
-		}, func(path string) (string, error) {
-			etag, err := xattr.Get(filepath.Join(bucket, path), "user.etag")
-			return string(etag), err
-		}, []string{metaTmpDir})
+		fileToObj(bucket), []string{metaTmpDir})
 	if err != nil {
 		return nil, fmt.Errorf("walk %v: %w", bucket, err)
 	}
@@ -1088,6 +1139,41 @@ func (p *Posix) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int)
 	}, nil
 }
 
+func (p *Posix) PutBucketAcl(bucket string, data []byte) error {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return fmt.Errorf("stat bucket: %w", err)
+	}
+
+	if err := xattr.Set(bucket, aclkey, data); err != nil {
+		return fmt.Errorf("set acl: %w", err)
+	}
+
+	return nil
+}
+
+func (p *Posix) GetBucketAcl(bucket string) ([]byte, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	b, err := xattr.Get(bucket, aclkey)
+	if isNoAttr(err) {
+		return []byte{}, nil
+	}
+	if err != nil {
+		return nil, fmt.Errorf("get acl: %w", err)
+	}
+	return b, nil
+}
+
 func (p *Posix) GetTags(bucket, object string) (map[string]string, error) {
 	_, err := os.Stat(bucket)
 	if errors.Is(err, fs.ErrNotExist) {
@@ -1161,6 +1247,198 @@ func (p *Posix) RemoveTags(bucket, object string) error {
 	return p.SetTags(bucket, object, nil)
 }
 
+const (
+	iamMode = 0600
+)
+
+func (p *Posix) InitIAM() error {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+
+	_, err := os.ReadFile(iamFile)
+	if errors.Is(err, fs.ErrNotExist) {
+		b, err := json.Marshal(auth.IAMConfig{})
+		if err != nil {
+			return fmt.Errorf("marshal default iam: %w", err)
+		}
+		err = os.WriteFile(iamFile, b, iamMode)
+		if err != nil {
+			return fmt.Errorf("write default iam: %w", err)
+		}
+	}
+
+	return nil
+}
+
+func (p *Posix) GetIAM() ([]byte, error) {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+
+	if !p.iamvalid || !p.iamexpire.After(time.Now()) {
+		p.mu.RUnlock()
+		err := p.refreshIAM()
+		p.mu.RLock()
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return p.iamcache, nil
+}
+
+const (
+	backoff  = 100 * time.Millisecond
+	maxretry = 300
+)
+
+func (p *Posix) refreshIAM() error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	// We are going to be racing with other running gateways without any
+	// coordination. So we might find the file does not exist at times.
+	// For this case we need to retry for a while assuming the other gateway
+	// will eventually write the file. If it doesn't after the max retries,
+	// then we will return the error.
+
+	retries := 0
+
+	for {
+		b, err := os.ReadFile(iamFile)
+		if errors.Is(err, fs.ErrNotExist) {
+			// racing with someone else updating
+			// keep retrying after backoff
+			retries++
+			if retries < maxretry {
+				time.Sleep(backoff)
+				continue
+			}
+			return fmt.Errorf("read iam file: %w", err)
+		}
+		if err != nil {
+			return err
+		}
+
+		p.iamcache = b
+		p.iamvalid = true
+		p.iamexpire = time.Now().Add(cacheDuration)
+		break
+	}
+
+	return nil
+}
+
+func (p *Posix) StoreIAM(update auth.UpdateAcctFunc) error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	// We are going to be racing with other running gateways without any
+	// coordination. So the strategy here is to read the current file data.
+	// If the file doesn't exist, then we assume someone else is currently
+	// updating the file. So we just need to keep retrying. We also need
+	// to make sure the data is consistent within a single update. So racing
+	// writes to a file would possibly leave this in some invalid state.
+	// We can get atomic updates with rename. If we read the data, update
+	// the data, write to a temp file, then rename the tempfile back to the
+	// data file. This should always result in a complete data image.
+
+	// There is at least one unsolved failure mode here.
+	// If a gateway removes the data file and then crashes, all other
+	// gateways will retry forever thinking that the original will eventually
+	// write the file.
+
+	retries := 0
+
+	for {
+		b, err := os.ReadFile(iamFile)
+		if errors.Is(err, fs.ErrNotExist) {
+			// racing with someone else updating
+			// keep retrying after backoff
+			retries++
+			if retries < maxretry {
+				time.Sleep(backoff)
+				continue
+			}
+
+			// we have been unsuccessful trying to read the iam file
+			// so this must be the case where something happened and
+			// the file did not get updated successfully, and probably
+			// isn't going to be. The recovery procedure would be to
+			// copy the backup file into place of the original.
+			return fmt.Errorf("no iam file, needs backup recovery")
+		}
+		if err != nil && !errors.Is(err, fs.ErrNotExist) {
+			return fmt.Errorf("read iam file: %w", err)
+		}
+
+		// reset retries on successful read
+		retries = 0
+
+		err = os.Remove(iamFile)
+		if errors.Is(err, fs.ErrNotExist) {
+			// racing with someone else updating
+			// keep retrying after backoff
+			time.Sleep(backoff)
+			continue
+		}
+		if err != nil && !errors.Is(err, fs.ErrNotExist) {
+			return fmt.Errorf("remove old iam file: %w", err)
+		}
+
+		// save copy of data
+		datacopy := make([]byte, len(b))
+		copy(datacopy, b)
+
+		// make a backup copy in case we crash before update
+		// this is after remove, so there is a small window something
+		// can go wrong, but the remove should barrier other gateways
+		// from trying to write backup at the same time. Only one
+		// gateway will successfully remove the file.
+		os.WriteFile(iamBackupFile, b, iamMode)
+
+		b, err = update(b)
+		if err != nil {
+			// update failed, try to write old data back out
+			os.WriteFile(iamFile, datacopy, iamMode)
+			return fmt.Errorf("update iam data: %w", err)
+		}
+
+		err = writeTempFile(b)
+		if err != nil {
+			// update failed, try to write old data back out
+			os.WriteFile(iamFile, datacopy, iamMode)
+			return err
+		}
+
+		p.iamcache = b
+		p.iamvalid = true
+		p.iamexpire = time.Now().Add(cacheDuration)
+		break
+	}
+
+	return nil
+}
+
+func writeTempFile(b []byte) error {
+	f, err := os.CreateTemp(".", iamFile)
+	if err != nil {
+		return fmt.Errorf("create temp file: %w", err)
+	}
+	defer os.Remove(f.Name())
+
+	_, err = f.Write(b)
+	if err != nil {
+		return fmt.Errorf("write temp file: %w", err)
+	}
+
+	err = os.Rename(f.Name(), iamFile)
+	if err != nil {
+		return fmt.Errorf("rename temp file: %w", err)
+	}
+
+	return nil
+}
+
 func isNoAttr(err error) bool {
 	if err == nil {
 		return false

backend/posix/posix_linux.go

@@ -68,7 +68,7 @@ func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
 	// later to link file into namespace
 	f := os.NewFile(uintptr(fd), filepath.Join(procfddir, strconv.Itoa(fd)))
 
-	tmp := &tmpfile{f: f, isOTmp: true, size: size}
+	tmp := &tmpfile{f: f, bucket: bucket, objname: obj, isOTmp: true, size: size}
 	// falloc is best effort, its fine if this fails
 	if size > 0 {
 		tmp.falloc()
@@ -117,7 +117,8 @@ func (tmp *tmpfile) link() error {
 	err = unix.Linkat(int(procdir.Fd()), filepath.Base(tmp.f.Name()),
 		int(dir.Fd()), filepath.Base(objPath), unix.AT_SYMLINK_FOLLOW)
 	if err != nil {
-		return fmt.Errorf("link tmpfile: %w", err)
+		return fmt.Errorf("link tmpfile (%q in %q): %w",
+			filepath.Dir(objPath), filepath.Base(tmp.f.Name()), err)
 	}
 
 	err = tmp.f.Close()

backend/scoutfs/scoutfs.go

@@ -15,12 +15,731 @@
 package scoutfs
 
 import (
+	"crypto/sha256"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/pkg/xattr"
+	"github.com/versity/scoutfs-go"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/backend/posix"
+	"github.com/versity/versitygw/s3err"
 )
 
 type ScoutFS struct {
 	*posix.Posix
+	rootfd  *os.File
+	rootdir string
+
+	// glaciermode enables the following behavior:
+	// GET object:  if file offline, return invalid object state
+	// HEAD object: if file offline, set obj storage class to GLACIER
+	//              if file offline and staging, x-amz-restore: ongoing-request="true"
+	//              if file offline and not staging, x-amz-restore: ongoing-request="false"
+	//              if file online, x-amz-restore: ongoing-request="false", expiry-date="Fri, 2 Dec 2050 00:00:00 GMT"
+	//              note: this expiry-date is not used but provided for client glacier compatibility
+	// ListObjects: if file offline, set obj storage class to GLACIER
+	// RestoreObject: add batch stage request to file
+	glaciermode bool
 }
 
-var _ backend.Backend = ScoutFS{}
+var _ backend.Backend = &ScoutFS{}
+
+const (
+	metaTmpDir          = ".sgwtmp"
+	metaTmpMultipartDir = metaTmpDir + "/multipart"
+	tagHdr              = "X-Amz-Tagging"
+	emptyMD5            = "d41d8cd98f00b204e9800998ecf8427e"
+	etagkey             = "user.etag"
+)
+
+var (
+	stageComplete      = "ongoing-request=\"false\", expiry-date=\"Fri, 2 Dec 2050 00:00:00 GMT\""
+	stageInProgress    = "true"
+	stageNotInProgress = "false"
+)
+
+const (
+	// ScoutFS special xattr types
+
+	systemPrefix = "scoutfs.hide."
+	onameAttr    = systemPrefix + "objname"
+	flagskey     = systemPrefix + "sam_flags"
+	stagecopykey = systemPrefix + "sam_stagereq"
+)
+
+const (
+	// ScoutAM Flags
+
+	// Staging - file requested stage
+	Staging uint64 = 1 << iota
+	// StageFail - all copies failed to stage
+	StageFail
+	// NoArchive - no archive copies of file should be made
+	NoArchive
+	// ExtCacheRequested means file policy requests Ext Cache
+	ExtCacheRequested
+	// ExtCacheDone means this file ext cache copy has been
+	// created already (and possibly pruned, so may not exist)
+	ExtCacheDone
+)
+
+// Option sets various options for scoutfs
+type Option func(s *ScoutFS)
+
+// WithGlacierEmulation sets glacier mode emulation
+func WithGlacierEmulation() Option {
+	return func(s *ScoutFS) { s.glaciermode = true }
+}
+
+func (s *ScoutFS) Shutdown() {
+	s.Posix.Shutdown()
+	s.rootfd.Close()
+	_ = s.rootdir
+}
+
+func (*ScoutFS) String() string {
+	return "ScoutFS Gateway"
+}
+
+// CompleteMultipartUpload scoutfs complete upload uses scoutfs move blocks
+// ioctl to not have to read and copy the part data to the final object. This
+// saves a read and write cycle for all mutlipart uploads.
+func (s *ScoutFS) CompleteMultipartUpload(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	sum, err := s.checkUploadIDExists(bucket, object, uploadID)
+	if err != nil {
+		return nil, err
+	}
+
+	objdir := filepath.Join(bucket, metaTmpMultipartDir, fmt.Sprintf("%x", sum))
+
+	// check all parts ok
+	last := len(parts) - 1
+	partsize := int64(0)
+	var totalsize int64
+	for i, p := range parts {
+		partPath := filepath.Join(objdir, uploadID, fmt.Sprintf("%v", p.PartNumber))
+		fi, err := os.Lstat(partPath)
+		if err != nil {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
+		}
+
+		if i == 0 {
+			partsize = fi.Size()
+		}
+		totalsize += fi.Size()
+		// all parts except the last need to be the same size
+		if i < last && partsize != fi.Size() {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
+		}
+		// non-last part sizes need to be multiples of 4k for move blocks
+		// TODO: fallback to no move blocks if not 4k aligned?
+		if i == 0 && i < last && fi.Size()%4096 != 0 {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
+		}
+
+		b, err := xattr.Get(partPath, "user.etag")
+		etag := string(b)
+		if err != nil {
+			etag = ""
+		}
+		parts[i].ETag = &etag
+	}
+
+	// use totalsize=0 because we wont be writing to the file, only moving
+	// extents around.  so we dont want to fallocate this.
+	f, err := openTmpFile(filepath.Join(bucket, metaTmpDir), bucket, object, 0)
+	if err != nil {
+		return nil, fmt.Errorf("open temp file: %w", err)
+	}
+	defer f.cleanup()
+
+	for _, p := range parts {
+		pf, err := os.Open(filepath.Join(objdir, uploadID, fmt.Sprintf("%v", p.PartNumber)))
+		if err != nil {
+			return nil, fmt.Errorf("open part %v: %v", p.PartNumber, err)
+		}
+
+		// scoutfs move data is a metadata only operation that moves the data
+		// extent references from the source, appeding to the destination.
+		// this needs to be 4k aligned.
+		err = scoutfs.MoveData(pf, f.f)
+		pf.Close()
+		if err != nil {
+			return nil, fmt.Errorf("move blocks part %v: %v", p.PartNumber, err)
+		}
+	}
+
+	userMetaData := make(map[string]string)
+	upiddir := filepath.Join(objdir, uploadID)
+	loadUserMetaData(upiddir, userMetaData)
+
+	objname := filepath.Join(bucket, object)
+	dir := filepath.Dir(objname)
+	if dir != "" {
+		if err = mkdirAll(dir, os.FileMode(0755), bucket, object); err != nil {
+			if err != nil {
+				return nil, s3err.GetAPIError(s3err.ErrExistingObjectIsDirectory)
+			}
+		}
+	}
+	err = f.link()
+	if err != nil {
+		return nil, fmt.Errorf("link object in namespace: %w", err)
+	}
+
+	for k, v := range userMetaData {
+		err = xattr.Set(objname, "user."+k, []byte(v))
+		if err != nil {
+			// cleanup object if returning error
+			os.Remove(objname)
+			return nil, fmt.Errorf("set user attr %q: %w", k, err)
+		}
+	}
+
+	// Calculate s3 compatible md5sum for complete multipart.
+	s3MD5 := backend.GetMultipartMD5(parts)
+
+	err = xattr.Set(objname, "user.etag", []byte(s3MD5))
+	if err != nil {
+		// cleanup object if returning error
+		os.Remove(objname)
+		return nil, fmt.Errorf("set etag attr: %w", err)
+	}
+
+	// cleanup tmp dirs
+	os.RemoveAll(upiddir)
+	// use Remove for objdir in case there are still other uploads
+	// for same object name outstanding
+	os.Remove(objdir)
+
+	return &s3.CompleteMultipartUploadOutput{
+		Bucket: &bucket,
+		ETag:   &s3MD5,
+		Key:    &object,
+	}, nil
+}
+
+func (s *ScoutFS) checkUploadIDExists(bucket, object, uploadID string) ([32]byte, error) {
+	sum := sha256.Sum256([]byte(object))
+	objdir := filepath.Join(bucket, metaTmpMultipartDir, fmt.Sprintf("%x", sum))
+
+	_, err := os.Stat(filepath.Join(objdir, uploadID))
+	if errors.Is(err, fs.ErrNotExist) {
+		return [32]byte{}, s3err.GetAPIError(s3err.ErrNoSuchUpload)
+	}
+	if err != nil {
+		return [32]byte{}, fmt.Errorf("stat upload: %w", err)
+	}
+	return sum, nil
+}
+
+func loadUserMetaData(path string, m map[string]string) (contentType, contentEncoding string) {
+	ents, err := xattr.List(path)
+	if err != nil || len(ents) == 0 {
+		return
+	}
+	for _, e := range ents {
+		if !isValidMeta(e) {
+			continue
+		}
+		b, err := xattr.Get(path, e)
+		if err == syscall.ENODATA {
+			m[strings.TrimPrefix(e, "user.")] = ""
+			continue
+		}
+		if err != nil {
+			continue
+		}
+		m[strings.TrimPrefix(e, "user.")] = string(b)
+	}
+
+	b, err := xattr.Get(path, "user.content-type")
+	contentType = string(b)
+	if err != nil {
+		contentType = ""
+	}
+	if contentType != "" {
+		m["content-type"] = contentType
+	}
+
+	b, err = xattr.Get(path, "user.content-encoding")
+	contentEncoding = string(b)
+	if err != nil {
+		contentEncoding = ""
+	}
+	if contentEncoding != "" {
+		m["content-encoding"] = contentEncoding
+	}
+
+	return
+}
+
+func isValidMeta(val string) bool {
+	if strings.HasPrefix(val, "user.X-Amz-Meta") {
+		return true
+	}
+	if strings.EqualFold(val, "user.Expires") {
+		return true
+	}
+	return false
+}
+
+// mkdirAll is similar to os.MkdirAll but it will return ErrObjectParentIsFile
+// when appropriate
+func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
+	// Fast path: if we can tell whether path is a directory or file, stop with success or error.
+	dir, err := os.Stat(path)
+	if err == nil {
+		if dir.IsDir() {
+			return nil
+		}
+		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
+	}
+
+	// Slow path: make sure parent exists and then call Mkdir for path.
+	i := len(path)
+	for i > 0 && os.IsPathSeparator(path[i-1]) { // Skip trailing path separator.
+		i--
+	}
+
+	j := i
+	for j > 0 && !os.IsPathSeparator(path[j-1]) { // Scan backward over element.
+		j--
+	}
+
+	if j > 1 {
+		// Create parent.
+		err = mkdirAll(path[:j-1], perm, bucket, object)
+		if err != nil {
+			return err
+		}
+	}
+
+	// Parent now exists; invoke Mkdir and use its result.
+	err = os.Mkdir(path, perm)
+	if err != nil {
+		// Handle arguments like "foo/." by
+		// double-checking that directory doesn't exist.
+		dir, err1 := os.Lstat(path)
+		if err1 == nil && dir.IsDir() {
+			return nil
+		}
+		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
+	}
+	return nil
+}
+
+func (s *ScoutFS) HeadObject(bucket, object string) (*s3.HeadObjectOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	objPath := filepath.Join(bucket, object)
+	fi, err := os.Stat(objPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat object: %w", err)
+	}
+
+	userMetaData := make(map[string]string)
+	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
+
+	b, err := xattr.Get(objPath, etagkey)
+	etag := string(b)
+	if err != nil {
+		etag = ""
+	}
+
+	stclass := types.StorageClassStandard
+	requestOngoing := ""
+	if s.glaciermode {
+		requestOngoing = stageComplete
+
+		// Check if there are any offline exents associated with this file.
+		// If so, we will set storage class to glacier.
+		st, err := scoutfs.StatMore(objPath)
+		if errors.Is(err, fs.ErrNotExist) {
+			return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+		}
+		if err != nil {
+			return nil, fmt.Errorf("stat more: %w", err)
+		}
+		if st.Offline_blocks != 0 {
+			stclass = types.StorageClassGlacier
+			requestOngoing = stageNotInProgress
+
+			ok, err := isStaging(objPath)
+			if errors.Is(err, fs.ErrNotExist) {
+				return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+			}
+			if err != nil {
+				return nil, fmt.Errorf("check stage status: %w", err)
+			}
+			if ok {
+				requestOngoing = stageInProgress
+			}
+		}
+	}
+
+	return &s3.HeadObjectOutput{
+		ContentLength:   fi.Size(),
+		ContentType:     &contentType,
+		ContentEncoding: &contentEncoding,
+		ETag:            &etag,
+		LastModified:    backend.GetTimePtr(fi.ModTime()),
+		Metadata:        userMetaData,
+		StorageClass:    stclass,
+		Restore:         &requestOngoing,
+	}, nil
+}
+
+func (s *ScoutFS) GetObject(bucket, object, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	objPath := filepath.Join(bucket, object)
+	fi, err := os.Stat(objPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat object: %w", err)
+	}
+
+	startOffset, length, err := backend.ParseRange(fi, acceptRange)
+	if err != nil {
+		return nil, err
+	}
+
+	if startOffset+length > fi.Size() {
+		// TODO: is ErrInvalidRequest correct here?
+		return nil, s3err.GetAPIError(s3err.ErrInvalidRequest)
+	}
+
+	if s.glaciermode {
+		// Check if there are any offline exents associated with this file.
+		// If so, we will return the InvalidObjectState error.
+		st, err := scoutfs.StatMore(objPath)
+		if errors.Is(err, fs.ErrNotExist) {
+			return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+		}
+		if err != nil {
+			return nil, fmt.Errorf("stat more: %w", err)
+		}
+		if st.Offline_blocks != 0 {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidObjectState)
+		}
+	}
+
+	f, err := os.Open(objPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("open object: %w", err)
+	}
+	defer f.Close()
+
+	rdr := io.NewSectionReader(f, startOffset, length)
+	_, err = io.Copy(writer, rdr)
+	if err != nil {
+		return nil, fmt.Errorf("copy data: %w", err)
+	}
+
+	userMetaData := make(map[string]string)
+
+	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
+
+	b, err := xattr.Get(objPath, etagkey)
+	etag := string(b)
+	if err != nil {
+		etag = ""
+	}
+
+	tags, err := s.getXattrTags(bucket, object)
+	if err != nil {
+		return nil, fmt.Errorf("get object tags: %w", err)
+	}
+
+	return &s3.GetObjectOutput{
+		AcceptRanges:    &acceptRange,
+		ContentLength:   length,
+		ContentEncoding: &contentEncoding,
+		ContentType:     &contentType,
+		ETag:            &etag,
+		LastModified:    backend.GetTimePtr(fi.ModTime()),
+		Metadata:        userMetaData,
+		TagCount:        int32(len(tags)),
+		StorageClass:    types.StorageClassStandard,
+	}, nil
+}
+
+func (s *ScoutFS) getXattrTags(bucket, object string) (map[string]string, error) {
+	tags := make(map[string]string)
+	b, err := xattr.Get(filepath.Join(bucket, object), "user."+tagHdr)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if isNoAttr(err) {
+		return tags, nil
+	}
+	if err != nil {
+		return nil, fmt.Errorf("get tags: %w", err)
+	}
+
+	err = json.Unmarshal(b, &tags)
+	if err != nil {
+		return nil, fmt.Errorf("unmarshal tags: %w", err)
+	}
+
+	return tags, nil
+}
+
+func (s *ScoutFS) ListObjects(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	fileSystem := os.DirFS(bucket)
+	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
+		s.fileToObj(bucket), []string{metaTmpDir})
+	if err != nil {
+		return nil, fmt.Errorf("walk %v: %w", bucket, err)
+	}
+
+	return &s3.ListObjectsOutput{
+		CommonPrefixes: results.CommonPrefixes,
+		Contents:       results.Objects,
+		Delimiter:      &delim,
+		IsTruncated:    results.Truncated,
+		Marker:         &marker,
+		MaxKeys:        int32(maxkeys),
+		Name:           &bucket,
+		NextMarker:     &results.NextMarker,
+		Prefix:         &prefix,
+	}, nil
+}
+
+func (s *ScoutFS) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	fileSystem := os.DirFS(bucket)
+	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
+		s.fileToObj(bucket), []string{metaTmpDir})
+	if err != nil {
+		return nil, fmt.Errorf("walk %v: %w", bucket, err)
+	}
+
+	return &s3.ListObjectsV2Output{
+		CommonPrefixes:        results.CommonPrefixes,
+		Contents:              results.Objects,
+		Delimiter:             &delim,
+		IsTruncated:           results.Truncated,
+		ContinuationToken:     &marker,
+		MaxKeys:               int32(maxkeys),
+		Name:                  &bucket,
+		NextContinuationToken: &results.NextMarker,
+		Prefix:                &prefix,
+	}, nil
+}
+
+func (s *ScoutFS) fileToObj(bucket string) backend.GetObjFunc {
+	return func(path string, d fs.DirEntry) (types.Object, error) {
+		objPath := filepath.Join(bucket, path)
+		if d.IsDir() {
+			// directory object only happens if directory empty
+			// check to see if this is a directory object by checking etag
+			etagBytes, err := xattr.Get(objPath, etagkey)
+			if isNoAttr(err) || errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get etag: %w", err)
+			}
+			etag := string(etagBytes)
+
+			fi, err := d.Info()
+			if errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+			}
+
+			key := path + "/"
+
+			return types.Object{
+				ETag:         &etag,
+				Key:          &key,
+				LastModified: backend.GetTimePtr(fi.ModTime()),
+			}, nil
+		}
+
+		// file object, get object info and fill out object data
+		etagBytes, err := xattr.Get(objPath, etagkey)
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil && !isNoAttr(err) {
+			return types.Object{}, fmt.Errorf("get etag: %w", err)
+		}
+		etag := string(etagBytes)
+
+		fi, err := d.Info()
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil {
+			return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+		}
+
+		sc := types.ObjectStorageClassStandard
+		if s.glaciermode {
+			// Check if there are any offline exents associated with this file.
+			// If so, we will return the InvalidObjectState error.
+			st, err := scoutfs.StatMore(objPath)
+			if errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("stat more: %w", err)
+			}
+			if st.Offline_blocks != 0 {
+				sc = types.ObjectStorageClassGlacier
+			}
+		}
+
+		return types.Object{
+			ETag:         &etag,
+			Key:          &path,
+			LastModified: backend.GetTimePtr(fi.ModTime()),
+			Size:         fi.Size(),
+			StorageClass: sc,
+		}, nil
+	}
+}
+
+// RestoreObject will set stage request on file if offline and do nothing if
+// file is online
+func (s *ScoutFS) RestoreObject(bucket, object string, restoreRequest *s3.RestoreObjectInput) error {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return fmt.Errorf("stat bucket: %w", err)
+	}
+
+	err = setStaging(filepath.Join(bucket, object))
+	if errors.Is(err, fs.ErrNotExist) {
+		return s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return fmt.Errorf("stage object: %w", err)
+	}
+
+	return nil
+}
+
+func setStaging(objname string) error {
+	b, err := xattr.Get(objname, flagskey)
+	if err != nil && !isNoAttr(err) {
+		return err
+	}
+
+	var oldflags uint64
+	if !isNoAttr(err) {
+		err = json.Unmarshal(b, &oldflags)
+		if err != nil {
+			return err
+		}
+	}
+
+	newflags := oldflags | Staging
+
+	if newflags == oldflags {
+		// no flags change, just return
+		return nil
+	}
+
+	return fSetNewGlobalFlags(objname, newflags)
+}
+
+func isStaging(objname string) (bool, error) {
+	b, err := xattr.Get(objname, flagskey)
+	if err != nil && !isNoAttr(err) {
+		return false, err
+	}
+
+	var flags uint64
+	if !isNoAttr(err) {
+		err = json.Unmarshal(b, &flags)
+		if err != nil {
+			return false, err
+		}
+	}
+
+	return flags&Staging == Staging, nil
+}
+
+func fSetNewGlobalFlags(objname string, flags uint64) error {
+	b, err := json.Marshal(&flags)
+	if err != nil {
+		return err
+	}
+
+	return xattr.Set(objname, flagskey, b)
+}
+
+func isNoAttr(err error) bool {
+	if err == nil {
+		return false
+	}
+	xerr, ok := err.(*xattr.Error)
+	if ok && xerr.Err == xattr.ENOATTR {
+		return true
+	}
+	if err == syscall.ENODATA {
+		return true
+	}
+	return false
+}

backend/scoutfs/scoutfs_darwin.go

@@ -0,0 +1,48 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scoutfs
+
+import (
+	"errors"
+	"fmt"
+	"os"
+)
+
+func New(rootdir string, opts ...Option) (*ScoutFS, error) {
+	return nil, fmt.Errorf("scoutfs only available on linux")
+}
+
+type tmpfile struct {
+	f *os.File
+}
+
+var (
+	errNotSupported = errors.New("not supported")
+)
+
+func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
+	return nil, errNotSupported
+}
+
+func (tmp *tmpfile) link() error {
+	return errNotSupported
+}
+
+func (tmp *tmpfile) Write(b []byte) (int, error) {
+	return 0, errNotSupported
+}
+
+func (tmp *tmpfile) cleanup() {
+}

backend/scoutfs/scoutfs_linux.go

@@ -0,0 +1,184 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scoutfs
+
+import (
+	"crypto/sha256"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strconv"
+	"syscall"
+
+	"golang.org/x/sys/unix"
+
+	"github.com/versity/versitygw/backend/posix"
+)
+
+func New(rootdir string, opts ...Option) (*ScoutFS, error) {
+	p, err := posix.New(rootdir)
+	if err != nil {
+		return nil, err
+	}
+
+	f, err := os.Open(rootdir)
+	if err != nil {
+		return nil, fmt.Errorf("open %v: %w", rootdir, err)
+	}
+
+	s := &ScoutFS{Posix: p, rootfd: f, rootdir: rootdir}
+	for _, opt := range opts {
+		opt(s)
+	}
+
+	return s, nil
+}
+
+const procfddir = "/proc/self/fd"
+
+type tmpfile struct {
+	f       *os.File
+	bucket  string
+	objname string
+	isOTmp  bool
+	size    int64
+}
+
+func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
+	// O_TMPFILE allows for a file handle to an unnamed file in the filesystem.
+	// This can help reduce contention within the namespace (parent directories),
+	// etc. And will auto cleanup the inode on close if we never link this
+	// file descriptor into the namespace.
+	// Not all filesystems support this, so fallback to CreateTemp for when
+	// this is not supported.
+	fd, err := unix.Open(dir, unix.O_RDWR|unix.O_TMPFILE|unix.O_CLOEXEC, 0666)
+	if err != nil {
+		// O_TMPFILE not supported, try fallback
+		err := os.MkdirAll(dir, 0700)
+		if err != nil {
+			return nil, fmt.Errorf("make temp dir: %w", err)
+		}
+		f, err := os.CreateTemp(dir,
+			fmt.Sprintf("%x.", sha256.Sum256([]byte(obj))))
+		if err != nil {
+			return nil, err
+		}
+		tmp := &tmpfile{f: f, bucket: bucket, objname: obj, size: size}
+		// falloc is best effort, its fine if this fails
+		if size > 0 {
+			tmp.falloc()
+		}
+		return tmp, nil
+	}
+
+	// for O_TMPFILE, filename is /proc/self/fd/<fd> to be used
+	// later to link file into namespace
+	f := os.NewFile(uintptr(fd), filepath.Join(procfddir, strconv.Itoa(fd)))
+
+	tmp := &tmpfile{f: f, bucket: bucket, objname: obj, isOTmp: true, size: size}
+	// falloc is best effort, its fine if this fails
+	if size > 0 {
+		tmp.falloc()
+	}
+	return tmp, nil
+}
+
+func (tmp *tmpfile) falloc() error {
+	err := syscall.Fallocate(int(tmp.f.Fd()), 0, 0, tmp.size)
+	if err != nil {
+		return fmt.Errorf("fallocate: %v", err)
+	}
+	return nil
+}
+
+func (tmp *tmpfile) link() error {
+	// We use Linkat/Rename as the atomic operation for object puts. The
+	// upload is written to a temp (or unnamed/O_TMPFILE) file to not conflict
+	// with any other simultaneous uploads. The final operation is to move the
+	// temp file into place for the object. This ensures the object semantics
+	// of last upload completed wins and is not some combination of writes
+	// from simultaneous uploads.
+	objPath := filepath.Join(tmp.bucket, tmp.objname)
+	err := os.Remove(objPath)
+	if err != nil && !errors.Is(err, fs.ErrNotExist) {
+		return fmt.Errorf("remove stale path: %w", err)
+	}
+
+	if !tmp.isOTmp {
+		// O_TMPFILE not suported, use fallback
+		return tmp.fallbackLink()
+	}
+
+	procdir, err := os.Open(procfddir)
+	if err != nil {
+		return fmt.Errorf("open proc dir: %w", err)
+	}
+	defer procdir.Close()
+
+	dir, err := os.Open(filepath.Dir(objPath))
+	if err != nil {
+		return fmt.Errorf("open parent dir: %w", err)
+	}
+	defer dir.Close()
+
+	err = unix.Linkat(int(procdir.Fd()), filepath.Base(tmp.f.Name()),
+		int(dir.Fd()), filepath.Base(objPath), unix.AT_SYMLINK_FOLLOW)
+	if err != nil {
+		return fmt.Errorf("link tmpfile: %w", err)
+	}
+
+	err = tmp.f.Close()
+	if err != nil {
+		return fmt.Errorf("close tmpfile: %w", err)
+	}
+
+	return nil
+}
+
+func (tmp *tmpfile) fallbackLink() error {
+	tempname := tmp.f.Name()
+	// cleanup in case anything goes wrong, if rename succeeds then
+	// this will no longer exist
+	defer os.Remove(tempname)
+
+	err := tmp.f.Close()
+	if err != nil {
+		return fmt.Errorf("close tmpfile: %w", err)
+	}
+
+	objPath := filepath.Join(tmp.bucket, tmp.objname)
+	err = os.Rename(tempname, objPath)
+	if err != nil {
+		return fmt.Errorf("rename tmpfile: %w", err)
+	}
+
+	return nil
+}
+
+func (tmp *tmpfile) Write(b []byte) (int, error) {
+	if int64(len(b)) > tmp.size {
+		return 0, fmt.Errorf("write exceeds content length %v", tmp.size)
+	}
+
+	n, err := tmp.f.Write(b)
+	tmp.size -= int64(n)
+	return n, err
+}
+
+func (tmp *tmpfile) cleanup() {
+	tmp.f.Close()
+}

backend/scoutfs/scoutfs_windows.go

@@ -0,0 +1,48 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scoutfs
+
+import (
+	"errors"
+	"fmt"
+	"os"
+)
+
+func New(rootdir string, opts ...Option) (*ScoutFS, error) {
+	return nil, fmt.Errorf("scoutfs only available on linux")
+}
+
+type tmpfile struct {
+	f *os.File
+}
+
+var (
+	errNotSupported = errors.New("not supported")
+)
+
+func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
+	return nil, errNotSupported
+}
+
+func (tmp *tmpfile) link() error {
+	return errNotSupported
+}
+
+func (tmp *tmpfile) Write(b []byte) (int, error) {
+	return 0, errNotSupported
+}
+
+func (tmp *tmpfile) cleanup() {
+}

backend/walk.go

@@ -15,6 +15,7 @@
 package backend
 
 import (
+	"errors"
 	"fmt"
 	"io/fs"
 	"os"
@@ -31,12 +32,13 @@ type WalkResults struct {
 	NextMarker     string
 }
 
-type DirObjCheck func(path string) (bool, error)
-type GetETag func(path string) (string, error)
+type GetObjFunc func(path string, d fs.DirEntry) (types.Object, error)
+
+var ErrSkipObj = errors.New("skip this object")
 
 // Walk walks the supplied fs.FS and returns results compatible with list
 // objects responses
-func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, dirchk DirObjCheck, getetag GetETag, skipdirs []string) (WalkResults, error) {
+func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, getObj GetObjFunc, skipdirs []string) (WalkResults, error) {
 	cpmap := make(map[string]struct{})
 	var objects []types.Object
 
@@ -90,26 +92,14 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, dirchk Di
 				return fmt.Errorf("readdir %q: %w", path, err)
 			}
 			if len(ents) == 0 {
-				dirobj, err := dirchk(path)
+				dirobj, err := getObj(path, d)
+				if err == ErrSkipObj {
+					return nil
+				}
 				if err != nil {
-					return fmt.Errorf("directory object check %q: %w", path, err)
-				}
-				if dirobj {
-					fi, err := d.Info()
-					if err != nil {
-						return fmt.Errorf("dir info %q: %w", path, err)
-					}
-					etag, err := getetag(path)
-					if err != nil {
-						return fmt.Errorf("get etag %q: %w", path, err)
-					}
-					path := path + "/"
-					objects = append(objects, types.Object{
-						ETag:         &etag,
-						Key:          &path,
-						LastModified: GetTimePtr(fi.ModTime()),
-					})
+					return fmt.Errorf("directory to object %q: %w", path, err)
 				}
+				objects = append(objects, dirobj)
 			}
 
 			return nil
@@ -122,29 +112,22 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, dirchk Di
 			pastMarker = true
 		}
 
-		// If object doesnt have prefix, dont include in results.
+		// If object doesn't have prefix, don't include in results.
 		if prefix != "" && !strings.HasPrefix(path, prefix) {
 			return nil
 		}
 
 		if delimiter == "" {
-			// If no delimeter specified, then all files with matching
+			// If no delimiter specified, then all files with matching
 			// prefix are included in results
-			fi, err := d.Info()
-			if err != nil {
-				return fmt.Errorf("get info for %v: %w", path, err)
+			obj, err := getObj(path, d)
+			if err == ErrSkipObj {
+				return nil
 			}
-			etag, err := getetag(path)
 			if err != nil {
-				return fmt.Errorf("get etag %q: %w", path, err)
+				return fmt.Errorf("file to object %q: %w", path, err)
 			}
-
-			objects = append(objects, types.Object{
-				ETag:         &etag,
-				Key:          &path,
-				LastModified: GetTimePtr(fi.ModTime()),
-				Size:         fi.Size(),
-			})
+			objects = append(objects, obj)
 
 			if max > 0 && (len(objects)+len(cpmap)) == max {
 				pastMax = true
@@ -160,7 +143,7 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, dirchk Di
 		//
 		// For example:
 		// prefix = A/
-		// delimeter = /
+		// delimiter = /
 		// and objects:
 		// A/file
 		// A/B/file
@@ -169,28 +152,22 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, dirchk Di
 		// objects: A/file
 		// common prefix: A/B/
 		//
-		// Note: No obects are included past the common prefix since
+		// Note: No objects are included past the common prefix since
 		// these are all rolled up into the common prefix.
-		// Note: The delimeter can be anything, so we have to operate on
-		// the full path without any assumptions on posix directory heirarchy
-		// here.  Usually the delimeter will be "/", but thats not required.
+		// Note: The delimiter can be anything, so we have to operate on
+		// the full path without any assumptions on posix directory hierarchy
+		// here.  Usually the delimiter will be "/", but thats not required.
 		suffix := strings.TrimPrefix(path, prefix)
 		before, _, found := strings.Cut(suffix, delimiter)
 		if !found {
-			fi, err := d.Info()
-			if err != nil {
-				return fmt.Errorf("get info for %v: %w", path, err)
+			obj, err := getObj(path, d)
+			if err == ErrSkipObj {
+				return nil
 			}
-			etag, err := getetag(path)
 			if err != nil {
-				return fmt.Errorf("get etag %q: %w", path, err)
+				return fmt.Errorf("file to object %q: %w", path, err)
 			}
-			objects = append(objects, types.Object{
-				ETag:         &etag,
-				Key:          &path,
-				LastModified: GetTimePtr(fi.ModTime()),
-				Size:         fi.Size(),
-			})
+			objects = append(objects, obj)
 			if (len(objects) + len(cpmap)) == max {
 				pastMax = true
 			}
@@ -199,7 +176,7 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, dirchk Di
 
 		// Common prefixes are a set, so should not have duplicates.
 		// These are abstractly a "directory", so need to include the
-		// delimeter at the end.
+		// delimiter at the end.
 		cpmap[prefix+before+delimiter] = struct{}{}
 		if (len(objects) + len(cpmap)) == max {
 			pastMax = true

backend/walk_test.go

@@ -15,6 +15,9 @@
 package backend_test
 
 import (
+	"crypto/md5"
+	"encoding/hex"
+	"fmt"
 	"io/fs"
 	"testing"
 	"testing/fstest"
@@ -26,10 +29,44 @@ import (
 type walkTest struct {
 	fsys     fs.FS
 	expected backend.WalkResults
-	dc       backend.DirObjCheck
+	getobj   backend.GetObjFunc
 }
 
-func gettag(string) (string, error) { return "myetag", nil }
+func getObj(path string, d fs.DirEntry) (types.Object, error) {
+	if d.IsDir() {
+		etag := getMD5(path)
+
+		fi, err := d.Info()
+		if err != nil {
+			return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+		}
+
+		return types.Object{
+			ETag:         &etag,
+			Key:          &path,
+			LastModified: backend.GetTimePtr(fi.ModTime()),
+		}, nil
+	}
+
+	etag := getMD5(path)
+
+	fi, err := d.Info()
+	if err != nil {
+		return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+	}
+
+	return types.Object{
+		ETag:         &etag,
+		Key:          &path,
+		LastModified: backend.GetTimePtr(fi.ModTime()),
+		Size:         fi.Size(),
+	}, nil
+}
+
+func getMD5(text string) string {
+	hash := md5.Sum([]byte(text))
+	return hex.EncodeToString(hash[:])
+}
 
 func TestWalk(t *testing.T) {
 	tests := []walkTest{
@@ -51,7 +88,7 @@ func TestWalk(t *testing.T) {
 					Key: backend.GetStringPtr("sample.jpg"),
 				}},
 			},
-			dc: func(string) (bool, error) { return false, nil },
+			getobj: getObj,
 		},
 		{
 			// test case single dir/single file
@@ -64,12 +101,12 @@ func TestWalk(t *testing.T) {
 				}},
 				Objects: []types.Object{},
 			},
-			dc: func(string) (bool, error) { return true, nil },
+			getobj: getObj,
 		},
 	}
 
 	for _, tt := range tests {
-		res, err := backend.Walk(tt.fsys, "", "/", "", 1000, tt.dc, gettag, []string{})
+		res, err := backend.Walk(tt.fsys, "", "/", "", 1000, tt.getobj, []string{})
 		if err != nil {
 			t.Fatalf("walk: %v", err)
 		}

cmd/versitygw/admin.go

@@ -73,6 +73,19 @@ func adminCommand() *cli.Command {
 					},
 				},
 			},
+			{
+				Name:   "delete-user",
+				Usage:  "Delete a user",
+				Action: deleteUser,
+				Flags: []cli.Flag{
+					&cli.StringFlag{
+						Name:     "access",
+						Usage:    "access value for the user to be deleted",
+						Required: true,
+						Aliases:  []string{"a"},
+					},
+				},
+			},
 		},
 		Flags: []cli.Flag{
 			// TODO: create a configuration file for this
@@ -143,3 +156,43 @@ func createUser(ctx *cli.Context) error {
 
 	return nil
 }
+
+func deleteUser(ctx *cli.Context) error {
+	access := ctx.String("access")
+	if access == "" {
+		return fmt.Errorf("invalid input parameter for the new user")
+	}
+
+	req, err := http.NewRequest(http.MethodDelete, fmt.Sprintf("http://localhost:7070/delete-user?access=%v", access), nil)
+	if err != nil {
+		return fmt.Errorf("failed to send the request: %w", err)
+	}
+
+	signer := v4.NewSigner()
+
+	hashedPayload := sha256.Sum256([]byte{})
+	hexPayload := hex.EncodeToString(hashedPayload[:])
+
+	req.Header.Set("X-Amz-Content-Sha256", hexPayload)
+
+	signErr := signer.SignHTTP(req.Context(), aws.Credentials{AccessKeyID: adminAccess, SecretAccessKey: adminSecret}, req, hexPayload, "s3", adminRegion, time.Now())
+	if signErr != nil {
+		return fmt.Errorf("failed to sign the request: %w", err)
+	}
+
+	client := http.Client{}
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return fmt.Errorf("failed to send the request: %w", err)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+
+	fmt.Printf("%s", body)
+
+	return nil
+}

cmd/versitygw/main.go

@@ -22,8 +22,8 @@ import (
 
 	"github.com/gofiber/fiber/v2"
 	"github.com/urfave/cli/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api"
 	"github.com/versity/versitygw/s3api/middlewares"
 )
@@ -51,7 +51,9 @@ func main() {
 
 	app.Commands = []*cli.Command{
 		posixCommand(),
+		scoutfsCommand(),
 		adminCommand(),
+		testCommand(),
 	}
 
 	if err := app.Run(os.Args); err != nil {
@@ -132,7 +134,7 @@ func initFlags() []cli.Flag {
 	}
 }
 
-func runGateway(be backend.Backend) error {
+func runGateway(be backend.Backend, s auth.Storer) error {
 	app := fiber.New(fiber.Config{
 		AppName:      "versitygw",
 		ServerHeader: "VERSITYGW",
@@ -160,11 +162,20 @@ func runGateway(be backend.Backend) error {
 		opts = append(opts, s3api.WithDebug())
 	}
 
+	err := s.InitIAM()
+	if err != nil {
+		return fmt.Errorf("init iam: %w", err)
+	}
+
+	iam, err := auth.NewInternal(s)
+	if err != nil {
+		return fmt.Errorf("setup internal iam service: %w", err)
+	}
+
 	srv, err := s3api.New(app, be, middlewares.RootUserConfig{
 		Access: rootUserAccess,
 		Secret: rootUserSecret,
-		Region: region,
-	}, port, auth.New(), opts...)
+	}, port, region, iam, opts...)
 	if err != nil {
 		return fmt.Errorf("init gateway: %v", err)
 	}

cmd/versitygw/posix.go

@@ -49,5 +49,5 @@ func runPosix(ctx *cli.Context) error {
 		return fmt.Errorf("init posix: %v", err)
 	}
 
-	return runGateway(be)
+	return runGateway(be, be)
 }

cmd/versitygw/scoutfs.go

@@ -0,0 +1,73 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package main
+
+import (
+	"fmt"
+
+	"github.com/urfave/cli/v2"
+	"github.com/versity/versitygw/backend/scoutfs"
+)
+
+var (
+	glacier bool
+)
+
+func scoutfsCommand() *cli.Command {
+	return &cli.Command{
+		Name:  "scoutfs",
+		Usage: "scoutfs filesystem storage backend",
+		Description: `Support for ScoutFS.
+The top level directory for the gateway must be provided. All sub directories
+of the top level directory are treated as buckets, and all files/directories
+below the "bucket directory" are treated as the objects. The object name is
+split on "/" separator to translate to posix storage.
+For example:
+top level: /mnt/fs/gwroot
+bucket: mybucket
+object: a/b/c/myobject
+will be translated into the file /mnt/fs/gwroot/mybucket/a/b/c/myobject
+
+ScoutFS contains optimizations for multipart uploads using extent
+move interfaces as well as support for tiered filesystems.`,
+		Action: runScoutfs,
+		Flags: []cli.Flag{
+			&cli.BoolFlag{
+				Name:        "glacier",
+				Usage:       "enable glacier emulation mode",
+				Aliases:     []string{"g"},
+				Destination: &glacier,
+			},
+		},
+	}
+}
+
+func runScoutfs(ctx *cli.Context) error {
+	if ctx.NArg() == 0 {
+		return fmt.Errorf("no directory provided for operation")
+	}
+
+	var opts []scoutfs.Option
+	if glacier {
+		opts = append(opts, scoutfs.WithGlacierEmulation())
+	}
+
+	be, err := scoutfs.New(ctx.Args().Get(0), opts...)
+	if err != nil {
+		return fmt.Errorf("init scoutfs: %v", err)
+	}
+
+	return runGateway(be, be)
+}

cmd/versitygw/test.go

@@ -0,0 +1,180 @@
+package main
+
+import (
+	"fmt"
+
+	"github.com/urfave/cli/v2"
+	"github.com/versity/versitygw/integration"
+)
+
+var (
+	awsID     string
+	awsSecret string
+	endpoint  string
+)
+
+func testCommand() *cli.Command {
+	return &cli.Command{
+		Name:  "test",
+		Usage: "Client side testing command for the gateway",
+		Description: `The testing CLI is used to test group of versitygw actions.
+		It also includes some performance and stress testing`,
+		Subcommands: initTestCommands(),
+		Flags:       initTestFlags(),
+	}
+}
+
+func initTestFlags() []cli.Flag {
+	return []cli.Flag{
+		&cli.StringFlag{
+			Name:        "access",
+			Usage:       "aws user access key",
+			EnvVars:     []string{"AWS_ACCESS_KEY_ID", "AWS_ACCESS_KEY"},
+			Aliases:     []string{"a"},
+			Destination: &awsID,
+		},
+		&cli.StringFlag{
+			Name:        "secret",
+			Usage:       "aws user secret access key",
+			EnvVars:     []string{"AWS_SECRET_ACCESS_KEY", "AWS_SECRET_KEY"},
+			Aliases:     []string{"s"},
+			Destination: &awsSecret,
+		},
+		&cli.StringFlag{
+			Name:        "endpoint",
+			Usage:       "s3 server endpoint",
+			Destination: &endpoint,
+			Aliases:     []string{"e"},
+		},
+		&cli.BoolFlag{
+			Name:        "debug",
+			Usage:       "enable debug mode",
+			Aliases:     []string{"d"},
+			Destination: &debug,
+		},
+	}
+}
+
+func initTestCommands() []*cli.Command {
+	return []*cli.Command{
+		{
+			Name:  "make-bucket",
+			Usage: "Test bucket creation.",
+			Description: `Calls s3 gateway create-bucket action to create a new bucket,
+		then calls delete-bucket action to delete the bucket.`,
+			Action: getAction(integration.TestMakeBucket),
+		},
+		{
+			Name:  "put-get-object",
+			Usage: "Test put & get object.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it,
+			gets the object from the bucket, deletes both the object and bucket.`,
+			Action: getAction(integration.TestPutGetObject),
+		},
+		{
+			Name:  "put-get-mp-object",
+			Usage: "Test put & get multipart object.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it with multipart upload,
+			gets the object from the bucket, deletes both the object and bucket.`,
+			Action: getAction(integration.TestPutGetMPObject),
+		},
+		{
+			Name:  "put-dir-object",
+			Usage: "Test put directory object.",
+			Description: `Creates a bucket with s3 gateway action, puts a directory object in it,
+			lists the bucket's objects, deletes both the objects and bucket.`,
+			Action: getAction(integration.TestPutDirObject),
+		},
+		{
+			Name:  "list-objects",
+			Usage: "Test list-objects action.",
+			Description: `Creates a bucket with s3 gateway action, puts 2 directory objects in it,
+			lists the bucket's objects, deletes both the objects and bucket.`,
+			Action: getAction(integration.TestListObject),
+		},
+		{
+			Name:  "abort-mp",
+			Usage: "Tests abort-multipart-upload action.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			lists the multipart upload, aborts the multipart upload, lists the multipart upload again,
+			deletes both the objects and bucket.`,
+			Action: getAction(integration.TestListAbortMultiPartObject),
+		},
+		{
+			Name:  "list-parts",
+			Usage: "Tests list-parts action.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			lists the upload parts, deletes both the objects and bucket.`,
+			Action: getAction(integration.TestListMultiParts),
+		},
+		{
+			Name:  "incorrect-mp",
+			Usage: "Tests incorrect multipart case.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			uploads different parts, completes the multipart upload with incorrect part numbers,
+			calls the head-object action, compares the content length, removes both the object and bucket`,
+			Action: getAction(integration.TestIncorrectMultiParts),
+		},
+		{
+			Name:  "incomplete-mp",
+			Usage: "Tests incomplete multi parts.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			upload a part, lists the parts, checks if the uploaded part is in the list, 
+			removes both the object and the bucket`,
+			Action: getAction(integration.TestIncompleteMultiParts),
+		},
+		{
+			Name:  "incomplete-put-object",
+			Usage: "Tests incomplete put objects case.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it,
+			gets the object with head-object action, expects the object to be got, 
+			removes both the object and bucket`,
+			Action: getAction(integration.TestIncompletePutObject),
+		},
+		{
+			Name:  "get-range",
+			Usage: "Tests get object by range.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it,
+			gets the object by specifying the object range, compares the range with the original one,
+			removes both the object and the bucket`,
+			Action: getAction(integration.TestRangeGet),
+		},
+		{
+			Name:  "invalid-mp",
+			Usage: "Tests invalid multi part case.",
+			Description: `Creates a bucket with s3 gateway action, creates a multi part upload,
+			uploads an invalid part, gets the object with head-object action, expects to get error,
+			removes both the object and bucket`,
+			Action: getAction(integration.TestInvalidMultiParts),
+		},
+		{
+			Name:        "full-flow",
+			Usage:       "Tests the full flow of gateway.",
+			Description: `Runs all the available tests to test the full flow of the gateway.`,
+			Action:      getAction(integration.TestFullFlow),
+		},
+	}
+}
+
+type testFunc func(*integration.S3Conf)
+
+func getAction(tf testFunc) func(*cli.Context) error {
+	return func(ctx *cli.Context) error {
+		opts := []integration.Option{
+			integration.WithAccess(awsID),
+			integration.WithSecret(awsSecret),
+			integration.WithRegion(region),
+			integration.WithEndpoint(endpoint),
+		}
+		if debug {
+			opts = append(opts, integration.WithDebug())
+		}
+
+		s := integration.NewS3Conf(opts...)
+		tf(s)
+
+		fmt.Println()
+		fmt.Println("RAN:", integration.RunCount, "PASS:", integration.PassCount, "FAIL:", integration.FailCount)
+		return nil
+	}
+}

go.mod

@@ -3,29 +3,42 @@ module github.com/versity/versitygw
 go 1.20
 
 require (
-	github.com/aws/aws-sdk-go-v2 v1.18.0
-	github.com/aws/aws-sdk-go-v2/service/s3 v1.33.1
+	github.com/aws/aws-sdk-go-v2 v1.18.1
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0
 	github.com/aws/smithy-go v1.13.5
 	github.com/gofiber/fiber/v2 v2.46.0
 	github.com/google/uuid v1.3.0
 	github.com/pkg/xattr v0.4.9
-	github.com/urfave/cli/v2 v2.25.4
+	github.com/urfave/cli/v2 v2.25.6
 	github.com/valyala/fasthttp v1.47.0
-	golang.org/x/sys v0.8.0
+	github.com/versity/scoutfs-go v0.0.0-20230606232754-0474b14343b9
+	golang.org/x/sys v0.9.0
+)
+
+require (
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.19.2 // indirect
+	github.com/jmespath/go-jmespath v0.4.0 // indirect
 )
 
 require (
 	github.com/andybalholm/brotli v1.0.5 // indirect
 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25 // indirect
+	github.com/aws/aws-sdk-go-v2/config v1.18.27
+	github.com/aws/aws-sdk-go-v2/credentials v1.13.26
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.28 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
-	github.com/klauspost/compress v1.16.5 // indirect
+	github.com/klauspost/compress v1.16.6 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.19 // indirect
 	github.com/mattn/go-runewidth v0.0.14 // indirect

go.sum

@@ -1,25 +1,43 @@
 github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
 github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
-github.com/aws/aws-sdk-go-v2 v1.18.0 h1:882kkTpSFhdgYRKVZ/VCgf7sd0ru57p2JCxz4/oN5RY=
-github.com/aws/aws-sdk-go-v2 v1.18.0/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
+github.com/aws/aws-sdk-go-v2 v1.18.1 h1:+tefE750oAb7ZQGzla6bLkOwfcQCEtC5y2RqoqCeqKo=
+github.com/aws/aws-sdk-go-v2 v1.18.1/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 h1:dK82zF6kkPeCo8J1e+tGx4JdvDIQzj7ygIoLg8WMuGs=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10/go.mod h1:VeTZetY5KRJLuD/7fkQXMU6Mw7H5m/KP2J5Iy9osMno=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33 h1:kG5eQilShqmJbv11XL1VpyDbaEJzWxd4zRiCG30GSn4=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33/go.mod h1:7i0PF1ME/2eUPFcjkVIwq+DOygHEoK92t5cDqNgYbIw=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27 h1:vFQlirhuM8lLlpI7imKOMsjdQLuN9CPi+k44F/OFVsk=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27/go.mod h1:UrHnn3QV/d0pBZ6QBAEQcqFLf8FAzLmoUfPVIueOvoM=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25 h1:AzwRi5OKKwo4QNqPf7TjeO+tK8AyOK3GVSwmRPo7/Cs=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25/go.mod h1:SUbB4wcbSEyCvqBxv/O/IBf93RbEze7U7OnoTlpPB+g=
+github.com/aws/aws-sdk-go-v2/config v1.18.27 h1:Az9uLwmssTE6OGTpsFqOnaGpLnKDqNYOJzWuC6UAYzA=
+github.com/aws/aws-sdk-go-v2/config v1.18.27/go.mod h1:0My+YgmkGxeqjXZb5BYme5pc4drjTnM+x1GJ3zv42Nw=
+github.com/aws/aws-sdk-go-v2/credentials v1.13.26 h1:qmU+yhKmOCyujmuPY7tf5MxR/RKyZrOPO3V4DobiTUk=
+github.com/aws/aws-sdk-go-v2/credentials v1.13.26/go.mod h1:GoXt2YC8jHUBbA4jr+W3JiemnIbkXOfxSXcisUsZ3os=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 h1:LxK/bitrAr4lnh9LnIS6i7zWbCOdMsfzKFBI6LUCS0I=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4/go.mod h1:E1hLXN/BL2e6YizK1zFlYd8vsfi2GTjbjBazinMmeaM=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70 h1:4bh28MeeXoBFTjb0JjQ5sVatzlf5xA1DziV8mZed9v4=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70/go.mod h1:9yI5NXzqy2yOiMytv6QLZHvlyHLwYxO9iIq+bZIbrFg=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 h1:A5UqQEmPaCFpedKouS4v+dHCTUo2sKqhoKO9U5kxyWo=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34/go.mod h1:wZpTEecJe0Btj3IYnDx/VlUzor9wm3fJHyvLpQF0VwY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 h1:srIVS45eQuewqz6fKKu6ZGXaq6FuFg5NzgQBAM6g8Y4=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28/go.mod h1:7VRpKQQedkfIEXb4k52I7swUnZP0wohVajJMRn3vsUw=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35 h1:LWA+3kDM8ly001vJ1X1waCuLJdtTl48gwkPKWy9sosI=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35/go.mod h1:0Eg1YjxE0Bhn56lx+SHJwCzhW+2JGtizsrx+lCqrfm0=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26 h1:wscW+pnn3J1OYnanMnza5ZVYXLX4cKk5rAvUAl4Qu+c=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26/go.mod h1:MtYiox5gvyB+OyP0Mr0Sm/yzbEAIPL9eijj/ouHAPw0=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 h1:y2+VQzC6Zh2ojtV2LoC0MNwHWc6qXv/j2vrQtlftkdA=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11/go.mod h1:iV4q2hsqtNECrfmlXyord9u4zyuFEJX9eLgLpSPzWA8=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.28 h1:vGWm5vTpMr39tEZfQeDiDAMgk+5qsnvRny3FjLpnH5w=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.28/go.mod h1:spfrICMD6wCAhjhzHuy6DOZZ+LAIY10UxhUmLzpJTTs=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27 h1:0iKliEXAcCa2qVtRs7Ot5hItA2MsufrphbRFlz1Owxo=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27/go.mod h1:EOwBD4J4S5qYszS5/3DpkejfuK+Z5/1uzICfPaZLtqw=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.2 h1:NbWkRxEEIRSCqxhsHQuMiTH7yo+JZW1gp8v3elSVMTQ=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.2/go.mod h1:4tfW5l4IAB32VWCDEBxCRtR9T4BWy4I4kr1spr8NgZM=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.33.1 h1:O+9nAy9Bb6bJFTpeNFtd9UfHbgxO1o4ZDAM9rQp5NsY=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.33.1/go.mod h1:J9kLNzEiHSeGMyN7238EjJmBpCniVzFda75Gxl/NqB8=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29 h1:zZSLP3v3riMOP14H7b4XP0uyfREDQOYv2cqIrvTXDNQ=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29/go.mod h1:z7EjRjVwZ6pWcWdI2H64dKttvzaP99jRIj5hphW0M5U=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 h1:bkRyG4a929RCnpVSTvLM2j/T4ls015ZhhYApbmYs15s=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28/go.mod h1:jj7znCIg05jXlaGBlFMGP8+7UN3VtCkRBG2spnmRQkU=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 h1:dBL3StFxHtpBzJJ/mNEsjXVgfO+7jR0dAIEwLqMapEA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3/go.mod h1:f1QyiAsvIv4B49DmCqrhlXqyaR+0IxMmyX+1P+AnzOM=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.34.1 h1:rYYwwsGqbwvGgQHjBkqgDt8MynXk+I8xgS0IEj5gOT0=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.34.1/go.mod h1:aVbf0sko/TsLWHx30c/uVu7c62+0EAJ3vbxaJga0xCw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0 h1:ya7fmrN2fE7s1P2gaPbNg5MTkERVWfsH8ToP1YC4Z9o=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0/go.mod h1:aVbf0sko/TsLWHx30c/uVu7c62+0EAJ3vbxaJga0xCw=
+github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 h1:nneMBM2p79PGWBQovYO/6Xnc2ryRMw3InnDJq1FHkSY=
+github.com/aws/aws-sdk-go-v2/service/sso v1.12.12/go.mod h1:HuCOxYsF21eKrerARYO6HapNeh9GBNq7fius2AcwodY=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12 h1:2qTR7IFk7/0IN/adSFhYu9Xthr0zVFTgBrmPldILn80=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12/go.mod h1:E4VrHCPzmVB/KFXtqBGKb3c8zpbNBgKe3fisDNLAW5w=
+github.com/aws/aws-sdk-go-v2/service/sts v1.19.2 h1:XFJ2Z6sNUUcAz9poj+245DMkrHE4h2j5I9/xD50RHfE=
+github.com/aws/aws-sdk-go-v2/service/sts v1.19.2/go.mod h1:dp0yLPsLBOi++WTxzCjA/oZqi6NPIhoR+uF7GeMU9eg=
 github.com/aws/smithy-go v1.13.5 h1:hgz0X/DX0dGqTYpGALqXJoRKRj5oQ7150i5FdTePzO8=
 github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
 github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
@@ -31,10 +49,11 @@ github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
 github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
 github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
 github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
 github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
-github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI=
-github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
+github.com/klauspost/compress v1.16.6 h1:91SKEy4K37vkp255cJ8QesJhjyRO0hn9i9G0GoUwLsk=
+github.com/klauspost/compress v1.16.6/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
 github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
 github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
 github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
@@ -62,14 +81,16 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/tinylib/msgp v1.1.6/go.mod h1:75BAfg2hauQhs3qedfdDZmWAPcFMAvJE5b9rGOMufyw=
 github.com/tinylib/msgp v1.1.8 h1:FCXC1xanKO4I8plpHGH2P7koL/RzZs12l/+r7vakfm0=
 github.com/tinylib/msgp v1.1.8/go.mod h1:qkpG+2ldGg4xRFmx+jfTvZPxfGFhi64BcnL9vkCm/Tw=
-github.com/urfave/cli/v2 v2.25.4 h1:HyYwPrTO3im9rYhUff/ZNs78eolxt0nJ4LN+9yJKSH4=
-github.com/urfave/cli/v2 v2.25.4/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
+github.com/urfave/cli/v2 v2.25.6 h1:yuSkgDSZfH3L1CjF2/5fNNg2KbM47pY2EvjBq4ESQnU=
+github.com/urfave/cli/v2 v2.25.6/go.mod h1:8qnjx1vcq5s2/wpsqoZFndg2CE5tNFyrTvS6SinrnYQ=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/fasthttp v1.47.0 h1:y7moDoxYzMooFpT5aHgNgVOQDrS3qlkfiP9mDtGGK9c=
 github.com/valyala/fasthttp v1.47.0/go.mod h1:k2zXd82h/7UZc3VOdJ2WaUqt1uZ/XpXAfE9i+HBC3lA=
 github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8=
 github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
+github.com/versity/scoutfs-go v0.0.0-20230606232754-0474b14343b9 h1:ZfmQR01Kk6/kQh6+zlqfBYszVY02fzf9xYrchOY4NFM=
+github.com/versity/scoutfs-go v0.0.0-20230606232754-0474b14343b9/go.mod h1:gJsq73k+4685y+rbDIpPY8i/5GbsiwP6JFoFyUDB1fQ=
 github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU=
 github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673/go.mod h1:N3UwUGtsrSj3ccvlPHLoLsHnpR27oXr4ZE984MbSER8=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -102,8 +123,8 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
-golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
+golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=

integration/output.go

@@ -0,0 +1,31 @@
+package integration
+
+import "fmt"
+
+var (
+	colorReset = "\033[0m"
+	colorRed   = "\033[31m"
+	colorGreen = "\033[32m"
+	colorCyan  = "\033[36m"
+)
+
+var (
+	RunCount  = 0
+	PassCount = 0
+	FailCount = 0
+)
+
+func runF(format string, a ...interface{}) {
+	RunCount++
+	fmt.Printf(colorCyan+"RUN  "+colorReset+format+"\n", a...)
+}
+
+func failF(format string, a ...interface{}) {
+	FailCount++
+	fmt.Printf(colorRed+"FAIL "+colorReset+format+"\n", a...)
+}
+
+func passF(format string, a ...interface{}) {
+	PassCount++
+	fmt.Printf(colorGreen+"PASS "+colorReset+format+"\n", a...)
+}

integration/reader.go

@@ -0,0 +1,54 @@
+package integration
+
+import (
+	"crypto/rand"
+	"crypto/sha256"
+	"hash"
+	"io"
+)
+
+type RReader struct {
+	buf      []byte
+	dataleft int
+	hash     hash.Hash
+}
+
+func NewDataReader(totalsize, bufsize int) *RReader {
+	b := make([]byte, bufsize)
+	rand.Read(b)
+	return &RReader{
+		buf:      b,
+		dataleft: totalsize,
+		hash:     sha256.New(),
+	}
+}
+
+func (r *RReader) Read(p []byte) (int, error) {
+	n := min(len(p), len(r.buf), r.dataleft)
+	r.dataleft -= n
+	err := error(nil)
+	if n == 0 {
+		err = io.EOF
+	}
+	r.hash.Write(r.buf[:n])
+	return copy(p, r.buf[:n]), err
+}
+
+func (r *RReader) Sum() []byte {
+	return r.hash.Sum(nil)
+}
+
+func min(values ...int) int {
+	if len(values) == 0 {
+		return 0
+	}
+
+	min := values[0]
+	for _, v := range values {
+		if v < min {
+			min = v
+		}
+	}
+
+	return min
+}

integration/s3conf.go

@@ -0,0 +1,125 @@
+package integration
+
+import (
+	"context"
+	"log"
+	"net/http"
+	"os"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
+	"github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/credentials"
+	"github.com/aws/smithy-go/middleware"
+)
+
+type S3Conf struct {
+	awsID           string
+	awsSecret       string
+	awsRegion       string
+	endpoint        string
+	checksumDisable bool
+	pathStyle       bool
+	PartSize        int64
+	Concurrency     int
+	debug           bool
+}
+
+func NewS3Conf(opts ...Option) *S3Conf {
+	s := &S3Conf{
+		PartSize:    64 * 1024 * 1024, // 64B default chunksize
+		Concurrency: 1,                // 1 default concurrency
+	}
+
+	for _, opt := range opts {
+		opt(s)
+	}
+	return s
+}
+
+type Option func(*S3Conf)
+
+func WithAccess(ak string) Option {
+	return func(s *S3Conf) { s.awsID = ak }
+}
+func WithSecret(sk string) Option {
+	return func(s *S3Conf) { s.awsSecret = sk }
+}
+func WithRegion(r string) Option {
+	return func(s *S3Conf) { s.awsRegion = r }
+}
+func WithEndpoint(e string) Option {
+	return func(s *S3Conf) { s.endpoint = e }
+}
+func WithDisableChecksum() Option {
+	return func(s *S3Conf) { s.checksumDisable = true }
+}
+func WithPathStyle() Option {
+	return func(s *S3Conf) { s.pathStyle = true }
+}
+func WithPartSize(p int64) Option {
+	return func(s *S3Conf) { s.PartSize = p }
+}
+func WithConcurrency(c int) Option {
+	return func(s *S3Conf) { s.Concurrency = c }
+}
+func WithDebug() Option {
+	return func(s *S3Conf) { s.debug = true }
+}
+
+func (c *S3Conf) getCreds() credentials.StaticCredentialsProvider {
+	// TODO support token/IAM
+	if c.awsSecret == "" {
+		c.awsSecret = os.Getenv("AWS_SECRET_ACCESS_KEY")
+	}
+	if c.awsSecret == "" {
+		log.Fatal("no AWS_SECRET_ACCESS_KEY found")
+	}
+
+	return credentials.NewStaticCredentialsProvider(c.awsID, c.awsSecret, "")
+}
+
+func (c *S3Conf) ResolveEndpoint(service, region string, options ...interface{}) (aws.Endpoint, error) {
+	return aws.Endpoint{
+		PartitionID:       "aws",
+		URL:               c.endpoint,
+		SigningRegion:     c.awsRegion,
+		HostnameImmutable: true,
+	}, nil
+}
+
+func (c *S3Conf) Config() aws.Config {
+	creds := c.getCreds()
+
+	tr := &http.Transport{}
+	client := &http.Client{Transport: tr}
+
+	opts := []func(*config.LoadOptions) error{
+		config.WithRegion(c.awsRegion),
+		config.WithCredentialsProvider(creds),
+		config.WithHTTPClient(client),
+	}
+
+	if c.endpoint != "" && c.endpoint != "aws" {
+		opts = append(opts,
+			config.WithEndpointResolverWithOptions(c))
+	}
+
+	if c.checksumDisable {
+		opts = append(opts,
+			config.WithAPIOptions([]func(*middleware.Stack) error{v4.SwapComputePayloadSHA256ForUnsignedPayloadMiddleware}))
+	}
+
+	if c.debug {
+		opts = append(opts,
+			config.WithClientLogMode(aws.LogSigning|aws.LogRetries|aws.LogRequest|aws.LogResponse|aws.LogRequestEventMessage|aws.LogResponseEventMessage))
+	}
+
+	cfg, err := config.LoadDefaultConfig(
+		context.TODO(), opts...)
+	if err != nil {
+		log.Fatalln("error:", err)
+	}
+
+	return cfg
+}

s3api/controllers/admin.go

@@ -18,28 +18,24 @@ import (
 	"fmt"
 
 	"github.com/gofiber/fiber/v2"
-	"github.com/versity/versitygw/backend/auth"
+	"github.com/versity/versitygw/auth"
 )
 
 type AdminController struct {
 	IAMService auth.IAMService
 }
 
-func NewAdminController() AdminController {
-	return AdminController{IAMService: auth.New()}
-}
-
 func (c AdminController) CreateUser(ctx *fiber.Ctx) error {
-	access, secret, role, region := ctx.Query("access"), ctx.Query("secret"), ctx.Query("role"), ctx.Query("region")
+	access, secret, role := ctx.Query("access"), ctx.Query("secret"), ctx.Query("role")
 	requesterRole := ctx.Locals("role")
 
 	if requesterRole != "admin" {
 		return fmt.Errorf("access denied: only admin users have access to this resource")
 	}
 
-	user := auth.Account{Secret: secret, Role: role, Region: region}
+	user := auth.Account{Secret: secret, Role: role}
 
-	err := c.IAMService.CreateAccount(access, &user)
+	err := c.IAMService.CreateAccount(access, user)
 	if err != nil {
 		return fmt.Errorf("failed to create a user: %w", err)
 	}
@@ -47,3 +43,19 @@ func (c AdminController) CreateUser(ctx *fiber.Ctx) error {
 	ctx.SendString("The user has been created successfully")
 	return nil
 }
+
+func (c AdminController) DeleteUser(ctx *fiber.Ctx) error {
+	access := ctx.Query("access")
+	requesterRole := ctx.Locals("role")
+	if requesterRole != "admin" {
+		return fmt.Errorf("access denied: only admin users have access to this resource")
+	}
+
+	err := c.IAMService.DeleteUserAccount(access)
+	if err != nil {
+		return err
+	}
+
+	ctx.SendString("The user has been created successfully")
+	return nil
+}

s3api/controllers/backend_moq_test.go

@@ -46,7 +46,7 @@ var _ backend.Backend = &BackendMock{}
 //			DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
 //				panic("mock out the DeleteObjects method")
 //			},
-//			GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
+//			GetBucketAclFunc: func(bucket string) ([]byte, error) {
 //				panic("mock out the GetBucketAcl method")
 //			},
 //			GetObjectFunc: func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
@@ -67,7 +67,7 @@ var _ backend.Backend = &BackendMock{}
 //			HeadObjectFunc: func(bucket string, object string) (*s3.HeadObjectOutput, error) {
 //				panic("mock out the HeadObject method")
 //			},
-//			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
+//			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
 //				panic("mock out the ListBuckets method")
 //			},
 //			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
@@ -82,10 +82,10 @@ var _ backend.Backend = &BackendMock{}
 //			ListObjectsV2Func: func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
 //				panic("mock out the ListObjectsV2 method")
 //			},
-//			PutBucketFunc: func(bucket string) error {
+//			PutBucketFunc: func(bucket string, owner string) error {
 //				panic("mock out the PutBucket method")
 //			},
-//			PutBucketAclFunc: func(putBucketAclInput *s3.PutBucketAclInput) error {
+//			PutBucketAclFunc: func(bucket string, data []byte) error {
 //				panic("mock out the PutBucketAcl method")
 //			},
 //			PutObjectFunc: func(putObjectInput *s3.PutObjectInput) (string, error) {
@@ -112,9 +112,6 @@ var _ backend.Backend = &BackendMock{}
 //			StringFunc: func() string {
 //				panic("mock out the String method")
 //			},
-//			UploadPartFunc: func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-//				panic("mock out the UploadPart method")
-//			},
 //			UploadPartCopyFunc: func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 //				panic("mock out the UploadPartCopy method")
 //			},
@@ -150,7 +147,7 @@ type BackendMock struct {
 	DeleteObjectsFunc func(bucket string, objects *s3.DeleteObjectsInput) error
 
 	// GetBucketAclFunc mocks the GetBucketAcl method.
-	GetBucketAclFunc func(bucket string) (*s3.GetBucketAclOutput, error)
+	GetBucketAclFunc func(bucket string) ([]byte, error)
 
 	// GetObjectFunc mocks the GetObject method.
 	GetObjectFunc func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error)
@@ -171,7 +168,7 @@ type BackendMock struct {
 	HeadObjectFunc func(bucket string, object string) (*s3.HeadObjectOutput, error)
 
 	// ListBucketsFunc mocks the ListBuckets method.
-	ListBucketsFunc func() (*s3.ListBucketsOutput, error)
+	ListBucketsFunc func() (s3response.ListAllMyBucketsResult, error)
 
 	// ListMultipartUploadsFunc mocks the ListMultipartUploads method.
 	ListMultipartUploadsFunc func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error)
@@ -186,10 +183,10 @@ type BackendMock struct {
 	ListObjectsV2Func func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error)
 
 	// PutBucketFunc mocks the PutBucket method.
-	PutBucketFunc func(bucket string) error
+	PutBucketFunc func(bucket string, owner string) error
 
 	// PutBucketAclFunc mocks the PutBucketAcl method.
-	PutBucketAclFunc func(putBucketAclInput *s3.PutBucketAclInput) error
+	PutBucketAclFunc func(bucket string, data []byte) error
 
 	// PutObjectFunc mocks the PutObject method.
 	PutObjectFunc func(putObjectInput *s3.PutObjectInput) (string, error)
@@ -215,9 +212,6 @@ type BackendMock struct {
 	// StringFunc mocks the String method.
 	StringFunc func() string
 
-	// UploadPartFunc mocks the UploadPart method.
-	UploadPartFunc func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error)
-
 	// UploadPartCopyFunc mocks the UploadPartCopy method.
 	UploadPartCopyFunc func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
@@ -391,11 +385,15 @@ type BackendMock struct {
 		PutBucket []struct {
 			// Bucket is the bucket argument value.
 			Bucket string
+			// Owner is the owner argument value.
+			Owner string
 		}
 		// PutBucketAcl holds details about calls to the PutBucketAcl method.
 		PutBucketAcl []struct {
-			// PutBucketAclInput is the putBucketAclInput argument value.
-			PutBucketAclInput *s3.PutBucketAclInput
+			// Bucket is the bucket argument value.
+			Bucket string
+			// Data is the data argument value.
+			Data []byte
 		}
 		// PutObject holds details about calls to the PutObject method.
 		PutObject []struct {
@@ -453,17 +451,6 @@ type BackendMock struct {
 		// String holds details about calls to the String method.
 		String []struct {
 		}
-		// UploadPart holds details about calls to the UploadPart method.
-		UploadPart []struct {
-			// Bucket is the bucket argument value.
-			Bucket string
-			// Object is the object argument value.
-			Object string
-			// UploadId is the uploadId argument value.
-			UploadId string
-			// Body is the Body argument value.
-			Body io.ReadSeeker
-		}
 		// UploadPartCopy holds details about calls to the UploadPartCopy method.
 		UploadPartCopy []struct {
 			// UploadPartCopyInput is the uploadPartCopyInput argument value.
@@ -500,7 +487,6 @@ type BackendMock struct {
 	lockSetTags                 sync.RWMutex
 	lockShutdown                sync.RWMutex
 	lockString                  sync.RWMutex
-	lockUploadPart              sync.RWMutex
 	lockUploadPartCopy          sync.RWMutex
 }
 
@@ -813,7 +799,7 @@ func (mock *BackendMock) DeleteObjectsCalls() []struct {
 }
 
 // GetBucketAcl calls GetBucketAclFunc.
-func (mock *BackendMock) GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error) {
+func (mock *BackendMock) GetBucketAcl(bucket string) ([]byte, error) {
 	if mock.GetBucketAclFunc == nil {
 		panic("BackendMock.GetBucketAclFunc: method is nil but Backend.GetBucketAcl was just called")
 	}
@@ -1069,7 +1055,7 @@ func (mock *BackendMock) HeadObjectCalls() []struct {
 }
 
 // ListBuckets calls ListBucketsFunc.
-func (mock *BackendMock) ListBuckets() (*s3.ListBucketsOutput, error) {
+func (mock *BackendMock) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
 	if mock.ListBucketsFunc == nil {
 		panic("BackendMock.ListBucketsFunc: method is nil but Backend.ListBuckets was just called")
 	}
@@ -1272,19 +1258,21 @@ func (mock *BackendMock) ListObjectsV2Calls() []struct {
 }
 
 // PutBucket calls PutBucketFunc.
-func (mock *BackendMock) PutBucket(bucket string) error {
+func (mock *BackendMock) PutBucket(bucket string, owner string) error {
 	if mock.PutBucketFunc == nil {
 		panic("BackendMock.PutBucketFunc: method is nil but Backend.PutBucket was just called")
 	}
 	callInfo := struct {
 		Bucket string
+		Owner  string
 	}{
 		Bucket: bucket,
+		Owner:  owner,
 	}
 	mock.lockPutBucket.Lock()
 	mock.calls.PutBucket = append(mock.calls.PutBucket, callInfo)
 	mock.lockPutBucket.Unlock()
-	return mock.PutBucketFunc(bucket)
+	return mock.PutBucketFunc(bucket, owner)
 }
 
 // PutBucketCalls gets all the calls that were made to PutBucket.
@@ -1293,9 +1281,11 @@ func (mock *BackendMock) PutBucket(bucket string) error {
 //	len(mockedBackend.PutBucketCalls())
 func (mock *BackendMock) PutBucketCalls() []struct {
 	Bucket string
+	Owner  string
 } {
 	var calls []struct {
 		Bucket string
+		Owner  string
 	}
 	mock.lockPutBucket.RLock()
 	calls = mock.calls.PutBucket
@@ -1304,19 +1294,21 @@ func (mock *BackendMock) PutBucketCalls() []struct {
 }
 
 // PutBucketAcl calls PutBucketAclFunc.
-func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) error {
+func (mock *BackendMock) PutBucketAcl(bucket string, data []byte) error {
 	if mock.PutBucketAclFunc == nil {
 		panic("BackendMock.PutBucketAclFunc: method is nil but Backend.PutBucketAcl was just called")
 	}
 	callInfo := struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}{
-		PutBucketAclInput: putBucketAclInput,
+		Bucket: bucket,
+		Data:   data,
 	}
 	mock.lockPutBucketAcl.Lock()
 	mock.calls.PutBucketAcl = append(mock.calls.PutBucketAcl, callInfo)
 	mock.lockPutBucketAcl.Unlock()
-	return mock.PutBucketAclFunc(putBucketAclInput)
+	return mock.PutBucketAclFunc(bucket, data)
 }
 
 // PutBucketAclCalls gets all the calls that were made to PutBucketAcl.
@@ -1324,10 +1316,12 @@ func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) e
 //
 //	len(mockedBackend.PutBucketAclCalls())
 func (mock *BackendMock) PutBucketAclCalls() []struct {
-	PutBucketAclInput *s3.PutBucketAclInput
+	Bucket string
+	Data   []byte
 } {
 	var calls []struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}
 	mock.lockPutBucketAcl.RLock()
 	calls = mock.calls.PutBucketAcl
@@ -1621,50 +1615,6 @@ func (mock *BackendMock) StringCalls() []struct {
 	return calls
 }
 
-// UploadPart calls UploadPartFunc.
-func (mock *BackendMock) UploadPart(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-	if mock.UploadPartFunc == nil {
-		panic("BackendMock.UploadPartFunc: method is nil but Backend.UploadPart was just called")
-	}
-	callInfo := struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}{
-		Bucket:   bucket,
-		Object:   object,
-		UploadId: uploadId,
-		Body:     Body,
-	}
-	mock.lockUploadPart.Lock()
-	mock.calls.UploadPart = append(mock.calls.UploadPart, callInfo)
-	mock.lockUploadPart.Unlock()
-	return mock.UploadPartFunc(bucket, object, uploadId, Body)
-}
-
-// UploadPartCalls gets all the calls that were made to UploadPart.
-// Check the length with:
-//
-//	len(mockedBackend.UploadPartCalls())
-func (mock *BackendMock) UploadPartCalls() []struct {
-	Bucket   string
-	Object   string
-	UploadId string
-	Body     io.ReadSeeker
-} {
-	var calls []struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}
-	mock.lockUploadPart.RLock()
-	calls = mock.calls.UploadPart
-	mock.lockUploadPart.RUnlock()
-	return calls
-}
-
 // UploadPartCopy calls UploadPartCopyFunc.
 func (mock *BackendMock) UploadPartCopy(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 	if mock.UploadPartCopyFunc == nil {

s3api/controllers/base.go

@@ -20,8 +20,8 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"log"
 	"net/http"
-	"os"
 	"strconv"
 	"strings"
 
@@ -29,13 +29,15 @@ import (
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3api/utils"
 	"github.com/versity/versitygw/s3err"
 )
 
 type S3ApiController struct {
-	be backend.Backend
+	be  backend.Backend
+	iam auth.IAMService
 }
 
 func New(be backend.Backend) S3ApiController {
@@ -43,6 +45,10 @@ func New(be backend.Backend) S3ApiController {
 }
 
 func (c S3ApiController) ListBuckets(ctx *fiber.Ctx) error {
+	access, isRoot := ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+	if err := auth.IsAdmin(access, isRoot); err != nil {
+		return SendXMLResponse(ctx, nil, err)
+	}
 	res, err := c.be.ListBuckets()
 	return SendXMLResponse(ctx, res, err)
 }
@@ -55,10 +61,22 @@ func (c S3ApiController) GetActions(ctx *fiber.Ctx) error {
 	maxParts := ctx.QueryInt("max-parts", 0)
 	partNumberMarker := ctx.QueryInt("part-number-marker", 0)
 	acceptRange := ctx.Get("Range")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	if uploadId != "" {
 		if maxParts < 0 || (maxParts == 0 && ctx.Query("max-parts") != "") {
 			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidMaxParts))
@@ -66,20 +84,36 @@ func (c S3ApiController) GetActions(ctx *fiber.Ctx) error {
 		if partNumberMarker < 0 || (partNumberMarker == 0 && ctx.Query("part-number-marker") != "") {
 			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidPartNumberMarker))
 		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+
 		res, err := c.be.ListObjectParts(bucket, key, uploadId, partNumberMarker, maxParts)
 		return SendXMLResponse(ctx, res, err)
 	}
 
 	if ctx.Request().URI().QueryArgs().Has("acl") {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ_ACP", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.GetObjectAcl(bucket, key)
 		return SendXMLResponse(ctx, res, err)
 	}
 
 	if attrs := ctx.Get("X-Amz-Object-Attributes"); attrs != "" {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.GetObjectAttributes(bucket, key, strings.Split(attrs, ","))
 		return SendXMLResponse(ctx, res, err)
 	}
 
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ_ACP", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	ctx.Locals("logResBody", false)
 	res, err := c.be.GetObject(bucket, key, acceptRange, ctx.Response().BodyWriter())
 	if err != nil {
 		return SendResponse(ctx, err)
@@ -114,8 +148,12 @@ func (c S3ApiController) GetActions(ctx *fiber.Ctx) error {
 			Key:   "Last-Modified",
 			Value: lastmod,
 		},
+		{
+			Key:   "x-amz-storage-class",
+			Value: string(res.StorageClass),
+		},
 	})
-	return ctx.SendStatus(http.StatusOK)
+	return SendResponse(ctx, err)
 }
 
 func getstring(s *string) string {
@@ -131,35 +169,63 @@ func (c S3ApiController) ListActions(ctx *fiber.Ctx) error {
 	marker := ctx.Query("continuation-token")
 	delimiter := ctx.Query("delimiter")
 	maxkeys := ctx.QueryInt("max-keys")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
 
 	if ctx.Request().URI().QueryArgs().Has("acl") {
-		res, err := c.be.GetBucketAcl(ctx.Params("bucket"))
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ_ACP", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+
+		res, err := auth.ParseACLOutput(data)
 		return SendXMLResponse(ctx, res, err)
 	}
 
 	if ctx.Request().URI().QueryArgs().Has("uploads") {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.ListMultipartUploads(&s3.ListMultipartUploadsInput{Bucket: aws.String(ctx.Params("bucket"))})
 		return SendXMLResponse(ctx, res, err)
 	}
 
 	if ctx.QueryInt("list-type") == 2 {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.ListObjectsV2(bucket, prefix, marker, delimiter, maxkeys)
 		return SendXMLResponse(ctx, res, err)
 	}
 
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+		return SendXMLResponse(ctx, nil, err)
+	}
+
 	res, err := c.be.ListObjects(bucket, prefix, marker, delimiter, maxkeys)
 	return SendXMLResponse(ctx, res, err)
 }
 
 func (c S3ApiController) PutBucketActions(ctx *fiber.Ctx) error {
-	bucket, acl, grantFullControl, grantRead, grantReadACP, granWrite, grantWriteACP :=
+	bucket, acl, grantFullControl, grantRead, grantReadACP, granWrite, grantWriteACP, access, isRoot :=
 		ctx.Params("bucket"),
 		ctx.Get("X-Amz-Acl"),
 		ctx.Get("X-Amz-Grant-Full-Control"),
 		ctx.Get("X-Amz-Grant-Read"),
 		ctx.Get("X-Amz-Grant-Read-Acp"),
 		ctx.Get("X-Amz-Grant-Write"),
-		ctx.Get("X-Amz-Grant-Write-Acp")
+		ctx.Get("X-Amz-Grant-Write-Acp"),
+		ctx.Locals("access").(string),
+		ctx.Locals("isRoot").(bool)
 
 	grants := grantFullControl + grantRead + grantReadACP + granWrite + grantWriteACP
 
@@ -167,20 +233,41 @@ func (c S3ApiController) PutBucketActions(ctx *fiber.Ctx) error {
 		if grants != "" && acl != "" {
 			return errors.New("wrong api call")
 		}
-		err := c.be.PutBucketAcl(&s3.PutBucketAclInput{
-			Bucket:           &bucket,
-			ACL:              types.BucketCannedACL(acl),
-			GrantFullControl: &grantFullControl,
-			GrantRead:        &grantRead,
-			GrantReadACP:     &grantReadACP,
-			GrantWrite:       &granWrite,
-			GrantWriteACP:    &grantWriteACP,
-		})
 
+		if acl != "" && acl != "private" && acl != "public-read" && acl != "public-read-write" {
+			return errors.New("wrong api call")
+		}
+
+		data, err := c.be.GetBucketAcl(bucket)
+		if err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		parsedAcl, err := auth.ParseACL(data)
+		if err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE_ACP", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		input := &s3.PutBucketAclInput{
+			Bucket:              &bucket,
+			ACL:                 types.BucketCannedACL(acl),
+			GrantFullControl:    &grantFullControl,
+			GrantRead:           &grantRead,
+			GrantReadACP:        &grantReadACP,
+			GrantWrite:          &granWrite,
+			GrantWriteACP:       &grantWriteACP,
+			AccessControlPolicy: &types.AccessControlPolicy{Owner: &types.Owner{ID: &access}},
+		}
+
+		err = auth.UpdateACL(input, parsedAcl, c.iam)
 		return SendResponse(ctx, err)
 	}
 
-	err := c.be.PutBucket(bucket)
+	err := c.be.PutBucket(bucket, access)
 	return SendResponse(ctx, err)
 }
 
@@ -190,6 +277,8 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 	keyEnd := ctx.Params("*1")
 	uploadId := ctx.Query("uploadId")
 	partNumberStr := ctx.Query("partNumber")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
 
 	// Copy source headers
 	copySource := ctx.Get("X-Amz-Copy-Source")
@@ -228,13 +317,28 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 		}
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	if uploadId != "" && partNumberStr != "" {
 		partNumber := ctx.QueryInt("partNumber", -1)
 		if partNumber < 1 {
 			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidPart))
 		}
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		body := io.ReadSeeker(bytes.NewReader([]byte(ctx.Body())))
+		ctx.Locals("logReqBody", false)
 		etag, err := c.be.PutObjectPart(bucket, keyStart, uploadId,
 			partNumber, contentLength, body)
 		ctx.Response().Header.Set("Etag", etag)
@@ -246,6 +350,10 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 			return errors.New("wrong api call")
 		}
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE_ACP", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		err := c.be.PutObjectAcl(&s3.PutObjectAclInput{
 			Bucket:           &bucket,
 			Key:              &keyStart,
@@ -265,12 +373,21 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 		copySourceSplit := strings.Split(copySource, "/")
 		srcBucket, srcObject := copySourceSplit[0], copySourceSplit[1:]
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+
 		res, err := c.be.CopyObject(srcBucket, strings.Join(srcObject, "/"), bucket, keyStart)
 		return SendXMLResponse(ctx, res, err)
 	}
 
 	metadata := utils.GetUserMetaData(&ctx.Request().Header)
 
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	ctx.Locals("logReqBody", false)
 	etag, err := c.be.PutObject(&s3.PutObjectInput{
 		Bucket:        &bucket,
 		Key:           &keyStart,
@@ -283,17 +400,49 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 }
 
 func (c S3ApiController) DeleteBucket(ctx *fiber.Ctx) error {
-	err := c.be.DeleteBucket(ctx.Params("bucket"))
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	err = c.be.DeleteBucket(bucket)
 	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) DeleteObjects(ctx *fiber.Ctx) error {
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
 	var dObj types.Delete
+
 	if err := xml.Unmarshal(ctx.Body(), &dObj); err != nil {
 		return errors.New("wrong api call")
 	}
 
-	err := c.be.DeleteObjects(ctx.Params("bucket"), &s3.DeleteObjectsInput{Delete: &dObj})
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	err = c.be.DeleteObjects(bucket, &s3.DeleteObjectsInput{Delete: &dObj})
 	return SendResponse(ctx, err)
 }
 
@@ -302,14 +451,30 @@ func (c S3ApiController) DeleteActions(ctx *fiber.Ctx) error {
 	key := ctx.Params("key")
 	keyEnd := ctx.Params("*1")
 	uploadId := ctx.Query("uploadId")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
 
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	if uploadId != "" {
 		expectedBucketOwner, requestPayer := ctx.Get("X-Amz-Expected-Bucket-Owner"), ctx.Get("X-Amz-Request-Payer")
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		err := c.be.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
 			UploadId:            &uploadId,
 			Bucket:              &bucket,
@@ -320,12 +485,32 @@ func (c S3ApiController) DeleteActions(ctx *fiber.Ctx) error {
 		return SendResponse(ctx, err)
 	}
 
-	err := c.be.DeleteObject(bucket, key)
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	err = c.be.DeleteObject(bucket, key)
 	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) HeadBucket(ctx *fiber.Ctx) error {
-	_, err := c.be.HeadBucket(ctx.Params("bucket"))
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	_, err = c.be.HeadBucket(bucket)
 	// TODO: set bucket response headers
 	return SendResponse(ctx, err)
 }
@@ -335,13 +520,27 @@ const (
 )
 
 func (c S3ApiController) HeadObject(ctx *fiber.Ctx) error {
-	bucket := ctx.Params("bucket")
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
 	key := ctx.Params("key")
 	keyEnd := ctx.Params("*1")
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	res, err := c.be.HeadObject(bucket, key)
 	if err != nil {
 		return SendResponse(ctx, err)
@@ -376,6 +575,14 @@ func (c S3ApiController) HeadObject(ctx *fiber.Ctx) error {
 			Key:   "Last-Modified",
 			Value: lastmod,
 		},
+		{
+			Key:   "x-amz-storage-class",
+			Value: string(res.StorageClass),
+		},
+		{
+			Key:   "x-amz-restore",
+			Value: getstring(res.Restore),
+		},
 	})
 
 	return SendResponse(ctx, nil)
@@ -386,17 +593,34 @@ func (c S3ApiController) CreateActions(ctx *fiber.Ctx) error {
 	key := ctx.Params("key")
 	keyEnd := ctx.Params("*1")
 	uploadId := ctx.Query("uploadId")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
 
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	var restoreRequest s3.RestoreObjectInput
 	if ctx.Request().URI().QueryArgs().Has("restore") {
 		xmlErr := xml.Unmarshal(ctx.Body(), &restoreRequest)
 		if xmlErr != nil {
 			return errors.New("wrong api call")
 		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		err := c.be.RestoreObject(bucket, key, &restoreRequest)
 		return SendResponse(ctx, err)
 	}
@@ -410,9 +634,18 @@ func (c S3ApiController) CreateActions(ctx *fiber.Ctx) error {
 			return errors.New("wrong api call")
 		}
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+
 		res, err := c.be.CompleteMultipartUpload(bucket, key, uploadId, data.Parts)
 		return SendXMLResponse(ctx, res, err)
 	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendXMLResponse(ctx, nil, err)
+	}
+
 	res, err := c.be.CreateMultipartUpload(&s3.CreateMultipartUploadInput{Bucket: &bucket, Key: &key})
 	return SendXMLResponse(ctx, res, err)
 }
@@ -424,10 +657,15 @@ func SendResponse(ctx *fiber.Ctx, err error) error {
 			ctx.Status(serr.HTTPStatusCode)
 			return ctx.Send(s3err.GetAPIErrorResponse(serr, "", "", ""))
 		}
+
+		log.Printf("Internal Error, %v", err)
+		ctx.Status(http.StatusInternalServerError)
 		return ctx.Send(s3err.GetAPIErrorResponse(
 			s3err.GetAPIError(s3err.ErrInternalError), "", "", ""))
 	}
 
+	utils.LogCtxDetails(ctx, []byte{})
+
 	// https://github.com/gofiber/fiber/issues/2080
 	// ctx.SendStatus() sets incorrect content length on HEAD request
 	ctx.Status(http.StatusOK)
@@ -442,8 +680,8 @@ func SendXMLResponse(ctx *fiber.Ctx, resp any, err error) error {
 			return ctx.Send(s3err.GetAPIErrorResponse(serr, "", "", ""))
 		}
 
-		fmt.Fprintf(os.Stderr, "Internal Error, req:\n%v\nerr:\n%v\n",
-			ctx.Request(), err)
+		log.Printf("Internal Error, %v", err)
+		ctx.Status(http.StatusInternalServerError)
 
 		return ctx.Send(s3err.GetAPIErrorResponse(
 			s3err.GetAPIError(s3err.ErrInternalError), "", "", ""))
@@ -461,5 +699,7 @@ func SendXMLResponse(ctx *fiber.Ctx, resp any, err error) error {
 		}
 	}
 
+	utils.LogCtxDetails(ctx, b)
+
 	return ctx.Send(b)
 }

s3api/controllers/base_test.go

@@ -15,6 +15,7 @@
 package controllers
 
 import (
+	"encoding/json"
 	"io"
 	"net/http"
 	"net/http/httptest"
@@ -26,12 +27,25 @@ import (
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/gofiber/fiber/v2"
-	"github.com/valyala/fasthttp"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3err"
 	"github.com/versity/versitygw/s3response"
 )
 
+var (
+	acl     auth.ACL
+	acldata []byte
+)
+
+func init() {
+	var err error
+	acldata, err = json.Marshal(acl)
+	if err != nil {
+		panic(err)
+	}
+}
+
 func TestNew(t *testing.T) {
 	type args struct {
 		be backend.Backend
@@ -65,58 +79,86 @@ func TestNew(t *testing.T) {
 
 func TestS3ApiController_ListBuckets(t *testing.T) {
 	type args struct {
-		ctx *fiber.Ctx
+		req *http.Request
 	}
 
 	app := fiber.New()
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{}, nil
+			},
+		},
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
+	app.Get("/", s3ApiController.ListBuckets)
+
+	// Error case
+	appErr := fiber.New()
+	s3ApiControllerErr := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{}, s3err.GetAPIError(s3err.ErrMethodNotAllowed)
+			},
+		},
+	}
+
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
+	appErr.Get("/", s3ApiControllerErr.ListBuckets)
 
 	tests := []struct {
 		name       string
-		c          S3ApiController
 		args       args
+		app        *fiber.App
 		wantErr    bool
 		statusCode int
 	}{
 		{
-			name: "List-bucket-not-implemented",
-			c: S3ApiController{
-				be: backend.BackendUnsupported{},
-			},
+			name: "List-bucket-method-not-allowed",
 			args: args{
-				ctx: app.AcquireCtx(&fasthttp.RequestCtx{}),
+				req: httptest.NewRequest(http.MethodGet, "/", nil),
 			},
+			app:        appErr,
 			wantErr:    false,
-			statusCode: 501,
+			statusCode: 405,
 		},
 		{
 			name: "list-bucket-success",
-			c: S3ApiController{
-				be: &BackendMock{
-					ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
-						return &s3.ListBucketsOutput{}, nil
-					},
-				},
-			},
 			args: args{
-				ctx: app.AcquireCtx(&fasthttp.RequestCtx{}),
+				req: httptest.NewRequest(http.MethodGet, "/", nil),
 			},
+			app:        app,
 			wantErr:    false,
 			statusCode: 200,
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			err := tt.c.ListBuckets(tt.args.ctx)
+			resp, err := tt.app.Test(tt.args.req)
 
 			if (err != nil) != tt.wantErr {
 				t.Errorf("S3ApiController.ListBuckets() error = %v, wantErr %v", err, tt.wantErr)
-				return
 			}
 
-			statusCode := tt.args.ctx.Response().StatusCode()
-
-			if statusCode != tt.statusCode {
-				t.Errorf("S3ApiController.ListBuckets() code = %v, wantErr %v", statusCode, tt.wantErr)
+			if resp.StatusCode != tt.statusCode {
+				t.Errorf("S3ApiController.ListBuckets() statusCode = %v, wantStatusCode = %v", resp.StatusCode, tt.statusCode)
 			}
 		})
 	}
@@ -128,20 +170,31 @@ func TestS3ApiController_GetActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		ListObjectPartsFunc: func(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
-			return s3response.ListPartsResponse{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListObjectPartsFunc: func(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
+				return s3response.ListPartsResponse{}, nil
+			},
+			GetObjectAclFunc: func(bucket, object string) (*s3.GetObjectAclOutput, error) {
+				return &s3.GetObjectAclOutput{}, nil
+			},
+			GetObjectAttributesFunc: func(bucket, object string, attributes []string) (*s3.GetObjectAttributesOutput, error) {
+				return &s3.GetObjectAttributesOutput{}, nil
+			},
+			GetObjectFunc: func(bucket, object, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
+				return &s3.GetObjectOutput{Metadata: nil}, nil
+			},
 		},
-		GetObjectAclFunc: func(bucket, object string) (*s3.GetObjectAclOutput, error) {
-			return &s3.GetObjectAclOutput{}, nil
-		},
-		GetObjectAttributesFunc: func(bucket, object string, attributes []string) (*s3.GetObjectAttributesOutput, error) {
-			return &s3.GetObjectAttributesOutput{}, nil
-		},
-		GetObjectFunc: func(bucket, object, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
-			return &s3.GetObjectOutput{Metadata: nil}, nil
-		},
-	}}
+	}
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Get("/:bucket/:key/*", s3ApiController.GetActions)
 
 	// GetObjectACL
@@ -230,29 +283,50 @@ func TestS3ApiController_ListActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
-			return &s3.GetBucketAclOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
+				return s3response.ListMultipartUploadsResponse{}, nil
+			},
+			ListObjectsV2Func: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
+				return &s3.ListObjectsV2Output{}, nil
+			},
+			ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
+				return &s3.ListObjectsOutput{}, nil
+			},
 		},
-		ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
-			return s3response.ListMultipartUploadsResponse{}, nil
-		},
-		ListObjectsV2Func: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
-			return &s3.ListObjectsV2Output{}, nil
-		},
-		ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
-			return &s3.ListObjectsOutput{}, nil
-		},
-	}}
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
+
 	app.Get("/:bucket", s3ApiController.ListActions)
 
 	//Error case
-	s3ApiControllerError := S3ApiController{be: &BackendMock{
-		ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
-			return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+	s3ApiControllerError := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
+				return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+			},
 		},
-	}}
+	}
 	appError := fiber.New()
+	appError.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appError.Get("/:bucket", s3ApiControllerError.ListActions)
 
 	tests := []struct {
@@ -329,14 +403,26 @@ func TestS3ApiController_PutBucketActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		PutBucketAclFunc: func(*s3.PutBucketAclInput) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			PutBucketAclFunc: func(string, []byte) error {
+				return nil
+			},
+			PutBucketFunc: func(bucket, owner string) error {
+				return nil
+			},
 		},
-		PutBucketFunc: func(bucket string) error {
-			return nil
-		},
-	}}
+	}
+	// Mock ctx.Locals
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Put("/:bucket", s3ApiController.PutBucketActions)
 
 	// Error case
@@ -402,23 +488,31 @@ func TestS3ApiController_PutActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		UploadPartCopyFunc: func(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
-			return &s3.UploadPartCopyOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			UploadPartCopyFunc: func(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
+				return &s3.UploadPartCopyOutput{}, nil
+			},
+			PutObjectAclFunc: func(*s3.PutObjectAclInput) error {
+				return nil
+			},
+			CopyObjectFunc: func(srcBucket, srcObject, DstBucket, dstObject string) (*s3.CopyObjectOutput, error) {
+				return &s3.CopyObjectOutput{}, nil
+			},
+			PutObjectFunc: func(*s3.PutObjectInput) (string, error) {
+				return "Hey", nil
+			},
 		},
-		UploadPartFunc: func(bucket, object, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-			return &s3.UploadPartOutput{}, nil
-		},
-		PutObjectAclFunc: func(*s3.PutObjectAclInput) error {
-			return nil
-		},
-		CopyObjectFunc: func(srcBucket, srcObject, DstBucket, dstObject string) (*s3.CopyObjectOutput, error) {
-			return &s3.CopyObjectOutput{}, nil
-		},
-		PutObjectFunc: func(*s3.PutObjectInput) (string, error) {
-			return "Hey", nil
-		},
-	}}
+	}
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Put("/:bucket/:key/*", s3ApiController.PutActions)
 
 	//PutObjectAcl error
@@ -535,23 +629,46 @@ func TestS3ApiController_DeleteBucket(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		DeleteBucketFunc: func(bucket string) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteBucketFunc: func(bucket string) error {
+				return nil
+			},
 		},
-	}}
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 
 	app.Delete("/:bucket", s3ApiController.DeleteBucket)
 
 	// error case
 	appErr := fiber.New()
 
-	s3ApiControllerErr := S3ApiController{be: &BackendMock{
-		DeleteBucketFunc: func(bucket string) error {
-			return s3err.GetAPIError(48)
+	s3ApiControllerErr := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteBucketFunc: func(bucket string) error {
+				return s3err.GetAPIError(48)
+			},
 		},
-	}}
+	}
 
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appErr.Delete("/:bucket", s3ApiControllerErr.DeleteBucket)
 
 	tests := []struct {
@@ -599,12 +716,23 @@ func TestS3ApiController_DeleteObjects(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
+				return nil
+			},
 		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Post("/:bucket", s3ApiController.DeleteObjects)
 
 	// Valid request body
@@ -658,26 +786,46 @@ func TestS3ApiController_DeleteActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		DeleteObjectFunc: func(bucket, object string) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteObjectFunc: func(bucket, object string) error {
+				return nil
+			},
+			AbortMultipartUploadFunc: func(*s3.AbortMultipartUploadInput) error {
+				return nil
+			},
 		},
-		AbortMultipartUploadFunc: func(*s3.AbortMultipartUploadInput) error {
-			return nil
-		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Delete("/:bucket/:key/*", s3ApiController.DeleteActions)
 
 	//Error case
 	appErr := fiber.New()
 
 	s3ApiControllerErr := S3ApiController{be: &BackendMock{
+		GetBucketAclFunc: func(bucket string) ([]byte, error) {
+			return acldata, nil
+		},
 		DeleteObjectFunc: func(bucket, object string) error {
 			return s3err.GetAPIError(7)
 		},
 	}}
 
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appErr.Delete("/:bucket", s3ApiControllerErr.DeleteBucket)
 
 	tests := []struct {
@@ -734,22 +882,45 @@ func TestS3ApiController_HeadBucket(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		HeadBucketFunc: func(bucket string) (*s3.HeadBucketOutput, error) {
-			return &s3.HeadBucketOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			HeadBucketFunc: func(bucket string) (*s3.HeadBucketOutput, error) {
+				return &s3.HeadBucketOutput{}, nil
+			},
 		},
-	}}
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 
 	app.Head("/:bucket", s3ApiController.HeadBucket)
 
-	//Error case
+	// Error case
 	appErr := fiber.New()
 
 	s3ApiControllerErr := S3ApiController{be: &BackendMock{
+		GetBucketAclFunc: func(bucket string) ([]byte, error) {
+			return acldata, nil
+		},
 		HeadBucketFunc: func(bucket string) (*s3.HeadBucketOutput, error) {
 			return nil, s3err.GetAPIError(3)
 		},
-	}}
+	},
+	}
+
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 
 	appErr.Head("/:bucket", s3ApiControllerErr.HeadBucket)
 
@@ -805,29 +976,51 @@ func TestS3ApiController_HeadObject(t *testing.T) {
 	eTag := "Valid etag"
 	lastModifie := time.Now()
 
-	s3ApiController := S3ApiController{be: &BackendMock{
-		HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
-			return &s3.HeadObjectOutput{
-				ContentEncoding: &contentEncoding,
-				ContentLength:   64,
-				ContentType:     &contentType,
-				LastModified:    &lastModifie,
-				ETag:            &eTag,
-			}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
+				return &s3.HeadObjectOutput{
+					ContentEncoding: &contentEncoding,
+					ContentLength:   64,
+					ContentType:     &contentType,
+					LastModified:    &lastModifie,
+					ETag:            &eTag,
+				}, nil
+			},
 		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Head("/:bucket/:key/*", s3ApiController.HeadObject)
 
 	//Error case
 	appErr := fiber.New()
 
-	s3ApiControllerErr := S3ApiController{be: &BackendMock{
-		HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
-			return nil, s3err.GetAPIError(42)
+	s3ApiControllerErr := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
+				return nil, s3err.GetAPIError(42)
+			},
 		},
-	}}
+	}
 
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appErr.Head("/:bucket/:key/*", s3ApiControllerErr.HeadObject)
 
 	tests := []struct {
@@ -874,18 +1067,29 @@ func TestS3ApiController_CreateActions(t *testing.T) {
 		req *http.Request
 	}
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		RestoreObjectFunc: func(bucket, object string, restoreRequest *s3.RestoreObjectInput) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			RestoreObjectFunc: func(bucket, object string, restoreRequest *s3.RestoreObjectInput) error {
+				return nil
+			},
+			CompleteMultipartUploadFunc: func(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error) {
+				return &s3.CompleteMultipartUploadOutput{}, nil
+			},
+			CreateMultipartUploadFunc: func(*s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error) {
+				return &s3.CreateMultipartUploadOutput{}, nil
+			},
 		},
-		CompleteMultipartUploadFunc: func(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error) {
-			return &s3.CompleteMultipartUploadOutput{}, nil
-		},
-		CreateMultipartUploadFunc: func(*s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error) {
-			return &s3.CreateMultipartUploadOutput{}, nil
-		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Post("/:bucket/:key/*", s3ApiController.CreateActions)
 
 	tests := []struct {
@@ -962,6 +1166,15 @@ func Test_XMLresponse(t *testing.T) {
 	}
 	app := fiber.New()
 
+	var ctx fiber.Ctx
+	// Mocking the fiber ctx
+	app.Get("/:bucket/:key", func(c *fiber.Ctx) error {
+		ctx = *c
+		return nil
+	})
+
+	app.Test(httptest.NewRequest(http.MethodGet, "/my-bucket/my-key", nil))
+
 	tests := []struct {
 		name       string
 		args       args
@@ -971,7 +1184,7 @@ func Test_XMLresponse(t *testing.T) {
 		{
 			name: "Internal-server-error",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: nil,
 				err:  s3err.GetAPIError(16),
 			},
@@ -981,7 +1194,7 @@ func Test_XMLresponse(t *testing.T) {
 		{
 			name: "Error-not-implemented",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: nil,
 				err:  s3err.GetAPIError(50),
 			},
@@ -991,7 +1204,7 @@ func Test_XMLresponse(t *testing.T) {
 		{
 			name: "Invalid-request-body",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: make(chan int),
 				err:  nil,
 			},
@@ -1001,7 +1214,7 @@ func Test_XMLresponse(t *testing.T) {
 		{
 			name: "Successful-response",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: "Valid response",
 				err:  nil,
 			},
@@ -1012,14 +1225,16 @@ func Test_XMLresponse(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			if err := SendXMLResponse(tt.args.ctx, tt.args.resp, tt.args.err); (err != nil) != tt.wantErr {
-				t.Errorf("responce() %v error = %v, wantErr %v", tt.name, err, tt.wantErr)
+				t.Errorf("response() %v error = %v, wantErr %v", tt.name, err, tt.wantErr)
 			}
 
 			statusCode := tt.args.ctx.Response().StatusCode()
 
 			if statusCode != tt.statusCode {
-				t.Errorf("responce() %v code = %v, wantErr %v", tt.name, statusCode, tt.wantErr)
+				t.Errorf("response() %v code = %v, wantErr %v", tt.name, statusCode, tt.wantErr)
 			}
+
+			tt.args.ctx.Status(http.StatusOK)
 		})
 	}
 }
@@ -1031,6 +1246,14 @@ func Test_response(t *testing.T) {
 		err  error
 	}
 	app := fiber.New()
+	var ctx fiber.Ctx
+	// Mocking the fiber ctx
+	app.Get("/:bucket/:key", func(c *fiber.Ctx) error {
+		ctx = *c
+		return nil
+	})
+
+	app.Test(httptest.NewRequest(http.MethodGet, "/my-bucket/my-key", nil))
 
 	tests := []struct {
 		name       string
@@ -1041,7 +1264,7 @@ func Test_response(t *testing.T) {
 		{
 			name: "Internal-server-error",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: nil,
 				err:  s3err.GetAPIError(16),
 			},
@@ -1051,7 +1274,7 @@ func Test_response(t *testing.T) {
 		{
 			name: "Error-not-implemented",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: nil,
 				err:  s3err.GetAPIError(50),
 			},
@@ -1061,7 +1284,7 @@ func Test_response(t *testing.T) {
 		{
 			name: "Successful-response",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: "Valid response",
 				err:  nil,
 			},
@@ -1072,13 +1295,13 @@ func Test_response(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			if err := SendResponse(tt.args.ctx, tt.args.err); (err != nil) != tt.wantErr {
-				t.Errorf("responce() %v error = %v, wantErr %v", tt.name, err, tt.wantErr)
+				t.Errorf("response() %v error = %v, wantErr %v", tt.name, err, tt.wantErr)
 			}
 
 			statusCode := tt.args.ctx.Response().StatusCode()
 
 			if statusCode != tt.statusCode {
-				t.Errorf("responce() %v code = %v, wantErr %v", tt.name, statusCode, tt.wantErr)
+				t.Errorf("response() %v code = %v, wantErr %v", tt.name, statusCode, tt.wantErr)
 			}
 		})
 	}

s3api/middlewares/authentication.go

@@ -25,7 +25,7 @@ import (
 	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
 	"github.com/aws/smithy-go/logging"
 	"github.com/gofiber/fiber/v2"
-	"github.com/versity/versitygw/backend/auth"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/s3api/controllers"
 	"github.com/versity/versitygw/s3api/utils"
 	"github.com/versity/versitygw/s3err"
@@ -38,10 +38,9 @@ const (
 type RootUserConfig struct {
 	Access string
 	Secret string
-	Region string
 }
 
-func VerifyV4Signature(root RootUserConfig, iam auth.IAMService, debug bool) fiber.Handler {
+func VerifyV4Signature(root RootUserConfig, iam auth.IAMService, region string, debug bool) fiber.Handler {
 	acct := accounts{root: root, iam: iam}
 
 	return func(ctx *fiber.Ctx) error {
@@ -68,16 +67,19 @@ func VerifyV4Signature(root RootUserConfig, iam auth.IAMService, debug bool) fib
 			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrCredMalformed))
 		}
 
-		signHdrKv := strings.Split(authParts[2], "=")
+		signHdrKv := strings.Split(authParts[2][:len(authParts[2])-1], "=")
 		if len(signHdrKv) != 2 {
 			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrCredMalformed))
 		}
 		signedHdrs := strings.Split(signHdrKv[1], ";")
 
-		account := acct.getAccount(creds[0])
-		if account == nil {
+		account, err := acct.getAccount(creds[0])
+		if err == auth.ErrNoSuchUser {
 			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidAccessKeyID))
 		}
+		if err != nil {
+			return controllers.SendResponse(ctx, err)
+		}
 
 		// Check X-Amz-Date header
 		date := ctx.Get("X-Amz-Date")
@@ -113,7 +115,7 @@ func VerifyV4Signature(root RootUserConfig, iam auth.IAMService, debug bool) fib
 		signErr := signer.SignHTTP(req.Context(), aws.Credentials{
 			AccessKeyID:     creds[0],
 			SecretAccessKey: account.Secret,
-		}, req, hexPayload, creds[3], account.Region, tdate, func(options *v4.SignerOptions) {
+		}, req, hexPayload, creds[3], region, tdate, func(options *v4.SignerOptions) {
 			if debug {
 				options.LogSigning = true
 				options.Logger = logging.NewStandardLogger(os.Stderr)
@@ -135,6 +137,8 @@ func VerifyV4Signature(root RootUserConfig, iam auth.IAMService, debug bool) fib
 		}
 
 		ctx.Locals("role", account.Role)
+		ctx.Locals("access", creds[0])
+		ctx.Locals("isRoot", creds[0] == root.Access)
 
 		return ctx.Next()
 	}
@@ -145,16 +149,13 @@ type accounts struct {
 	iam  auth.IAMService
 }
 
-func (a accounts) getAccount(access string) *auth.Account {
-	var account *auth.Account
+func (a accounts) getAccount(access string) (auth.Account, error) {
 	if access == a.root.Access {
-		account = &auth.Account{
+		return auth.Account{
 			Secret: a.root.Secret,
 			Role:   "admin",
-			Region: a.root.Region,
-		}
-	} else {
-		account = a.iam.GetUserAccount(access)
+		}, nil
 	}
-	return account
+
+	return a.iam.GetUserAccount(access)
 }

s3api/middlewares/logger.go

@@ -0,0 +1,44 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package middlewares
+
+import (
+	"fmt"
+	"log"
+
+	"github.com/gofiber/fiber/v2"
+)
+
+func RequestLogger(isDebug bool) fiber.Handler {
+	return func(ctx *fiber.Ctx) error {
+		ctx.Locals("isDebug", isDebug)
+		if isDebug {
+			log.Println("Request headers: ")
+			ctx.Request().Header.VisitAll(func(key, val []byte) {
+				log.Printf("%s: %s", key, val)
+			})
+
+			if ctx.Request().URI().QueryArgs().Len() != 0 {
+				fmt.Println()
+				log.Println("Request query arguments: ")
+				ctx.Request().URI().QueryArgs().VisitAll(func(key, val []byte) {
+					log.Printf("%s: %s", key, val)
+				})
+			}
+		}
+
+		return ctx.Next()
+	}
+}

s3api/router.go

@@ -16,8 +16,8 @@ package s3api
 
 import (
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api/controllers"
 )
 
@@ -29,6 +29,9 @@ func (sa *S3ApiRouter) Init(app *fiber.App, be backend.Backend, iam auth.IAMServ
 
 	// TODO: think of better routing system
 	app.Post("/create-user", adminController.CreateUser)
+
+	// Admin Delete api
+	app.Delete("/delete-user", adminController.DeleteUser)
 	// ListBuckets action
 	app.Get("/", s3ApiController.ListBuckets)
 

s3api/router_test.go

@@ -18,8 +18,8 @@ import (
 	"testing"
 
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 )
 
 func TestS3ApiRouter_Init(t *testing.T) {
@@ -39,7 +39,7 @@ func TestS3ApiRouter_Init(t *testing.T) {
 			args: args{
 				app: fiber.New(),
 				be:  backend.BackendUnsupported{},
-				iam: auth.IAMServiceUnsupported{},
+				iam: &auth.IAMServiceInternal{},
 			},
 		},
 	}

s3api/server.go

@@ -19,8 +19,8 @@ import (
 
 	"github.com/gofiber/fiber/v2"
 	"github.com/gofiber/fiber/v2/middleware/logger"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api/middlewares"
 )
 
@@ -33,7 +33,7 @@ type S3ApiServer struct {
 	debug   bool
 }
 
-func New(app *fiber.App, be backend.Backend, root middlewares.RootUserConfig, port string, iam auth.IAMService, opts ...Option) (*S3ApiServer, error) {
+func New(app *fiber.App, be backend.Backend, root middlewares.RootUserConfig, port, region string, iam auth.IAMService, opts ...Option) (*S3ApiServer, error) {
 	server := &S3ApiServer{
 		app:     app,
 		backend: be,
@@ -45,10 +45,16 @@ func New(app *fiber.App, be backend.Backend, root middlewares.RootUserConfig, po
 		opt(server)
 	}
 
-	app.Use(middlewares.VerifyV4Signature(root, iam, server.debug))
+	// Logging middlewares
 	app.Use(logger.New())
+	app.Use(middlewares.RequestLogger(server.debug))
+
+	// Authentication middlewares
+	app.Use(middlewares.VerifyV4Signature(root, iam, region, server.debug))
 	app.Use(middlewares.VerifyMD5Body())
+
 	server.router.Init(app, be, iam)
+
 	return server, nil
 }
 

s3api/server_test.go

@@ -19,8 +19,8 @@ import (
 	"testing"
 
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api/middlewares"
 )
 
@@ -63,7 +63,7 @@ func TestNew(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			gotS3ApiServer, err := New(tt.args.app, tt.args.be, tt.args.root,
-				tt.args.port, auth.IAMServiceUnsupported{})
+				tt.args.port, "us-east-1", &auth.IAMServiceInternal{})
 			if (err != nil) != tt.wantErr {
 				t.Errorf("New() error = %v, wantErr %v", err, tt.wantErr)
 				return

s3api/utils/logger.go

@@ -0,0 +1,55 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package utils
+
+import (
+	"fmt"
+	"log"
+
+	"github.com/gofiber/fiber/v2"
+)
+
+func LogCtxDetails(ctx *fiber.Ctx, respBody []byte) {
+	isDebug, ok := ctx.Locals("isDebug").(bool)
+	_, notLogReqBody := ctx.Locals("logReqBody").(bool)
+	_, notLogResBody := ctx.Locals("logResBody").(bool)
+	if isDebug && ok {
+		// Log request body
+		if !notLogReqBody {
+			fmt.Println()
+			log.Printf("Request Body: %s", ctx.Request().Body())
+		}
+
+		// Log path parameters
+		fmt.Println()
+		log.Println("Path parameters: ")
+		for key, val := range ctx.AllParams() {
+			log.Printf("%s: %s", key, val)
+		}
+
+		// Log response headers
+		fmt.Println()
+		log.Println("Response Headers: ")
+		ctx.Response().Header.VisitAll(func(key, val []byte) {
+			log.Printf("%s: %s", key, val)
+		})
+
+		// Log response body
+		if !notLogResBody && len(respBody) > 0 {
+			fmt.Println()
+			log.Printf("Response body %s", ctx.Response().Body())
+		}
+	}
+}

s3err/s3err.go

@@ -105,6 +105,7 @@ const (
 	ErrAuthNotSetup
 	ErrNotImplemented
 	ErrPreconditionFailed
+	ErrInvalidObjectState
 
 	ErrExistingObjectIsDirectory
 	ErrObjectParentIsFile
@@ -368,6 +369,11 @@ var errorCodeResponse = map[ErrorCode]APIError{
 		Description:    "At least one of the pre-conditions you specified did not hold",
 		HTTPStatusCode: http.StatusPreconditionFailed,
 	},
+	ErrInvalidObjectState: {
+		Code:           "InvalidObjectState",
+		Description:    "The operation is not valid for the current state of the object",
+		HTTPStatusCode: http.StatusForbidden,
+	},
 	ErrExistingObjectIsDirectory: {
 		Code:           "ExistingObjectIsDirectory",
 		Description:    "Existing Object is a directory.",

s3response/AmazonS3.xsd

@@ -0,0 +1,692 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xsd:schema
+  xmlns:tns="http://s3.amazonaws.com/doc/2006-03-01/"
+  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
+  elementFormDefault="qualified"
+  targetNamespace="http://s3.amazonaws.com/doc/2006-03-01/">
+
+  <xsd:element name="CreateBucket">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="MetadataEntry">
+    <xsd:sequence>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="Value" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="CreateBucketResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="CreateBucketReturn" type="tns:CreateBucketResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="Status">
+    <xsd:sequence>
+      <xsd:element name="Code" type="xsd:int"/>
+      <xsd:element name="Description" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="Result">
+    <xsd:sequence>
+      <xsd:element name="Status" type="tns:Status"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="CreateBucketResult">
+    <xsd:sequence>
+      <xsd:element name="BucketName" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="DeleteBucket">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+  <xsd:element name="DeleteBucketResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="DeleteBucketResponse" type="tns:Status"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="BucketLoggingStatus">
+    <xsd:sequence>
+      <xsd:element name="LoggingEnabled" type="tns:LoggingSettings" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="LoggingSettings">
+    <xsd:sequence>
+      <xsd:element name="TargetBucket" type="xsd:string"/>
+      <xsd:element name="TargetPrefix" type="xsd:string"/>
+      <xsd:element name="TargetGrants" type="tns:AccessControlList" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="GetBucketLoggingStatus">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+  
+  <xsd:element name="GetBucketLoggingStatusResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetBucketLoggingStatusResponse" type="tns:BucketLoggingStatus"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+  
+  <xsd:element name="SetBucketLoggingStatus">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="BucketLoggingStatus" type="tns:BucketLoggingStatus"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetBucketLoggingStatusResponse">
+    <xsd:complexType>
+      <xsd:sequence/>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetObjectAccessControlPolicyResponse" type="tns:AccessControlPolicy"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetBucketAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetBucketAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetBucketAccessControlPolicyResponse" type="tns:AccessControlPolicy"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType abstract="true" name="Grantee"/>
+
+  <xsd:complexType name="User" abstract="true">
+    <xsd:complexContent>
+      <xsd:extension base="tns:Grantee"/>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="AmazonCustomerByEmail">
+    <xsd:complexContent>
+      <xsd:extension base="tns:User">
+        <xsd:sequence>
+          <xsd:element name="EmailAddress" type="xsd:string"/>              
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="CanonicalUser">
+    <xsd:complexContent>
+      <xsd:extension base="tns:User">
+        <xsd:sequence>
+          <xsd:element name="ID" type="xsd:string"/>              
+          <xsd:element name="DisplayName" type="xsd:string" minOccurs="0"/>
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="Group">
+    <xsd:complexContent>
+      <xsd:extension base="tns:Grantee">
+        <xsd:sequence>
+          <xsd:element name="URI" type="xsd:string"/>
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:simpleType name="Permission">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="READ"/>
+      <xsd:enumeration value="WRITE"/>
+      <xsd:enumeration value="READ_ACP"/>
+      <xsd:enumeration value="WRITE_ACP"/>
+      <xsd:enumeration value="FULL_CONTROL"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:simpleType name="StorageClass">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="STANDARD"/>
+      <xsd:enumeration value="REDUCED_REDUNDANCY"/>
+      <xsd:enumeration value="GLACIER"/>
+      <xsd:enumeration value="UNKNOWN"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:complexType name="Grant">
+    <xsd:sequence>
+      <xsd:element name="Grantee" type="tns:Grantee"/>
+      <xsd:element name="Permission" type="tns:Permission"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="AccessControlList">
+    <xsd:sequence>
+      <xsd:element name="Grant" type="tns:Grant" minOccurs="0" maxOccurs="100"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="CreateBucketConfiguration">
+    <xsd:sequence>
+      <xsd:element name="LocationConstraint" type="tns:LocationConstraint"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="LocationConstraint">
+    <xsd:simpleContent>
+      <xsd:extension base="xsd:string"/>
+    </xsd:simpleContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="AccessControlPolicy">
+    <xsd:sequence>
+      <xsd:element name="Owner" type="tns:CanonicalUser"/>
+      <xsd:element name="AccessControlList" type="tns:AccessControlList"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="SetObjectAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetObjectAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence/>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetBucketAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetBucketAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence/>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="GetMetadata" type="xsd:boolean"/>
+        <xsd:element name="GetData" type="xsd:boolean"/>
+        <xsd:element name="InlineData" type="xsd:boolean"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+        
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetObjectResponse" type="tns:GetObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="GetObjectResult">
+    <xsd:complexContent>
+      <xsd:extension base="tns:Result">
+        <xsd:sequence>
+          <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
+          <xsd:element name="Data" type="xsd:base64Binary" nillable="true"/>
+          <xsd:element name="LastModified" type="xsd:dateTime"/>
+          <xsd:element name="ETag" type="xsd:string"/>
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:element name="GetObjectExtended">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="GetMetadata" type="xsd:boolean"/>
+        <xsd:element name="GetData" type="xsd:boolean"/>
+        <xsd:element name="InlineData" type="xsd:boolean"/>
+        <xsd:element name="ByteRangeStart" type="xsd:long" minOccurs="0"/>
+        <xsd:element name="ByteRangeEnd" type="xsd:long" minOccurs="0"/>
+        <xsd:element name="IfModifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="IfUnmodifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="IfMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="IfNoneMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="ReturnCompleteObjectOnConditionFailure" type="xsd:boolean" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectExtendedResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetObjectResponse" type="tns:GetObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="PutObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="ContentLength" type="xsd:long"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="PutObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="PutObjectResponse" type="tns:PutObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="PutObjectResult">
+    <xsd:sequence>
+      <xsd:element name="ETag" type="xsd:string"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="PutObjectInline">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element minOccurs="0" maxOccurs="100" name="Metadata" type="tns:MetadataEntry"/>
+        <xsd:element name="Data" type="xsd:base64Binary"/>
+        <xsd:element name="ContentLength" type="xsd:long"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="PutObjectInlineResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="PutObjectInlineResponse" type="tns:PutObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="DeleteObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="DeleteObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="DeleteObjectResponse" type="tns:Status"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListBucket">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Prefix" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Marker" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="MaxKeys" type="xsd:int" minOccurs="0"/>
+        <xsd:element name="Delimiter" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListBucketResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="ListBucketResponse" type="tns:ListBucketResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListVersionsResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="ListVersionsResponse" type="tns:ListVersionsResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>  
+
+  <xsd:complexType name="ListEntry">
+    <xsd:sequence>
+      <xsd:element name="Key" type="xsd:string"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="ETag" type="xsd:string"/>
+      <xsd:element name="Size" type="xsd:long"/>
+      <xsd:element name="Owner" type="tns:CanonicalUser" minOccurs="0"/>
+      <xsd:element name="StorageClass" type="tns:StorageClass"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="VersionEntry">
+    <xsd:sequence>
+      <xsd:element name="Key" type="xsd:string"/>
+      <xsd:element name="VersionId" type="xsd:string"/>
+      <xsd:element name="IsLatest" type="xsd:boolean"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="ETag" type="xsd:string"/>
+      <xsd:element name="Size" type="xsd:long"/>
+      <xsd:element name="Owner" type="tns:CanonicalUser" minOccurs="0"/>
+      <xsd:element name="StorageClass" type="tns:StorageClass"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="DeleteMarkerEntry">
+    <xsd:sequence>
+      <xsd:element name="Key" type="xsd:string"/>
+      <xsd:element name="VersionId" type="xsd:string"/>
+      <xsd:element name="IsLatest" type="xsd:boolean"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="Owner" type="tns:CanonicalUser" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="PrefixEntry">
+    <xsd:sequence>
+      <xsd:element name="Prefix" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListBucketResult">
+    <xsd:sequence>
+      <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="Prefix" type="xsd:string"/>
+      <xsd:element name="Marker" type="xsd:string"/>
+      <xsd:element name="NextMarker" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="MaxKeys" type="xsd:int"/>
+      <xsd:element name="Delimiter" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="IsTruncated" type="xsd:boolean"/>
+      <xsd:element name="Contents" type="tns:ListEntry" minOccurs="0" maxOccurs="unbounded"/>
+      <xsd:element name="CommonPrefixes" type="tns:PrefixEntry" minOccurs="0" maxOccurs="unbounded"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListVersionsResult">
+    <xsd:sequence>
+      <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="Prefix" type="xsd:string"/>
+      <xsd:element name="KeyMarker" type="xsd:string"/>
+      <xsd:element name="VersionIdMarker" type="xsd:string"/>
+      <xsd:element name="NextKeyMarker" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="NextVersionIdMarker" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="MaxKeys" type="xsd:int"/>
+      <xsd:element name="Delimiter" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="IsTruncated" type="xsd:boolean"/>
+      <xsd:choice minOccurs="0" maxOccurs="unbounded">
+          <xsd:element name="Version" type="tns:VersionEntry"/>
+          <xsd:element name="DeleteMarker" type="tns:DeleteMarkerEntry"/>
+      </xsd:choice>
+      <xsd:element name="CommonPrefixes" type="tns:PrefixEntry" minOccurs="0" maxOccurs="unbounded"/>
+    </xsd:sequence>
+  </xsd:complexType>  
+
+  <xsd:element name="ListAllMyBuckets">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListAllMyBucketsResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="ListAllMyBucketsResponse" type="tns:ListAllMyBucketsResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="ListAllMyBucketsEntry">
+    <xsd:sequence>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="CreationDate" type="xsd:dateTime"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListAllMyBucketsResult">
+    <xsd:sequence>
+      <xsd:element name="Owner" type="tns:CanonicalUser"/>
+      <xsd:element name="Buckets" type="tns:ListAllMyBucketsList"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListAllMyBucketsList">
+    <xsd:sequence>
+      <xsd:element name="Bucket" type="tns:ListAllMyBucketsEntry" minOccurs="0" maxOccurs="unbounded"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="PostResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Location" type="xsd:anyURI"/>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="ETag" type="xsd:string"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:simpleType name="MetadataDirective">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="COPY"/>
+      <xsd:enumeration value="REPLACE"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:element name="CopyObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="SourceBucket" type="xsd:string"/>
+        <xsd:element name="SourceKey" type="xsd:string"/>
+        <xsd:element name="DestinationBucket" type="xsd:string"/>
+        <xsd:element name="DestinationKey" type="xsd:string"/>
+        <xsd:element name="MetadataDirective" type="tns:MetadataDirective" minOccurs="0"/>
+        <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+	<xsd:element name="CopySourceIfModifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="CopySourceIfUnmodifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="CopySourceIfMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="CopySourceIfNoneMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="CopyObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="CopyObjectResult" type="tns:CopyObjectResult" />
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="CopyObjectResult">
+    <xsd:sequence>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="ETag" type="xsd:string"/>      
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="RequestPaymentConfiguration">
+    <xsd:sequence>
+      <xsd:element name="Payer" type="tns:Payer" minOccurs="1" maxOccurs="1"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:simpleType name="Payer">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="BucketOwner"/>
+      <xsd:enumeration value="Requester"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+  
+  <xsd:complexType name="VersioningConfiguration">
+    <xsd:sequence>
+      <xsd:element name="Status" type="tns:VersioningStatus" minOccurs="0"/>
+      <xsd:element name="MfaDelete" type="tns:MfaDeleteStatus" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:simpleType name="MfaDeleteStatus">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="Enabled"/>
+      <xsd:enumeration value="Disabled"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:simpleType name="VersioningStatus">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="Enabled"/>
+      <xsd:enumeration value="Suspended"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:complexType name="NotificationConfiguration">
+    <xsd:sequence>
+      <xsd:element name="TopicConfiguration" minOccurs="0" maxOccurs="unbounded" type="tns:TopicConfiguration"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="TopicConfiguration">
+    <xsd:sequence>
+      <xsd:element name="Topic" minOccurs="1" maxOccurs="1" type="xsd:string"/>
+      <xsd:element name="Event" minOccurs="1" maxOccurs="unbounded" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+</xsd:schema>

s3response/README.txt

@@ -0,0 +1,6 @@
+https://doc.s3.amazonaws.com/2006-03-01/AmazonS3.xsd
+
+see https://blog.aqwari.net/xml-schema-go/
+
+go install aqwari.net/xml/cmd/xsdgen@latest
+xsdgen -o s3api_xsd_generated.go -pkg s3response AmazonS3.xsd