Merge pull request #107 from versity/test-cli-setup

Test CLI setup

.github/workflows/go.yml

@@ -8,13 +8,13 @@ jobs:
     steps:
 
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v4
       with:
-        go-version: "1.20"
+        go-version: 'stable'
       id: go
 
     - name: Check out code into the Go module directory
-      uses: actions/checkout@v1
+      uses: actions/checkout@v3
 
     - name: Verify all files pass gofmt formatting
       run: if [ "$(gofmt -s -l . | wc -l)" -gt 0 ]; then gofmt -s -d .; exit 1; fi

CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+versitygw@versity.com.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

README.md

@@ -1,24 +1,37 @@
-# Versity S3 Gateway
+# The Versity Gateway: A High-Performance Open Source S3 to File Translation Tool
 
-[![Versity Logo](https://www.versity.com/wp-content/themes/versity-theme/assets/img/svg/logo.svg)](https://www.versity.com)
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://github.com/versity/versitygw/blob/assets/assets/logo-white.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://github.com/versity/versitygw/blob/assets/assets/logo.svg">
+  <a href="https://www.versity.com"><img alt="Versity Software logo image." src="https://github.com/versity/versitygw/blob/assets/assets/logo.svg"></a>
+</picture>
 
  [![Apache V2 License](https://img.shields.io/badge/license-Apache%20V2-blue.svg)](https://github.com/versity/versitygw/blob/main/LICENSE)  
 
-The Versity S3 Gateway provides  an S3 server that translates S3 client access to a modular backend service. The server translates incoming S3 API requests and transforms them into equivalent operations to the backend service. By leveraging this gateway server, applications can interact with the S3-compatible API on top of already existing storage systems. This project enables leveraging existing infrastructure investments while seamlessly integrating with S3-compatible systems, offering increased flexibility and compatibility in managing data storage.
+The Versity Gateway: A High-Performance Open Source S3 to File Translation Tool
 
-The Versity S3 Gateway is focused on performance, simplicity, and expandability. New backend types can be added to support new storage systems. The initial backend is a posix filesystem. The posix backend allows standing up an S3 compatible server from an existing filesystem mount with a simple command.
+Current status: Alpha, in development not yet suited for production use
 
-The gateway is completely stateless. Mutliple gateways can host the same backend service and clients can load balance across the gateways.
+See project [documentation](https://github.com/versity/versitygw/wiki) on the wiki.
+
+Versity Gateway, a simple to use tool for seamless inline translation between AWS S3 object commands and file-based storage systems. The Versity Gateway bridges the gap between S3-reliant applications and file storage systems, enabling enhanced compatibility and integration with file based systems while offering exceptional scalability.
+
+The server translates incoming S3 API requests and transforms them into equivalent operations to the backend service. By leveraging this gateway server, applications can interact with the S3-compatible API on top of already existing storage systems. This project enables leveraging existing infrastructure investments while seamlessly integrating with S3-compatible systems, offering increased flexibility and compatibility in managing data storage.
+
+The Versity Gateway is focused on performance, simplicity, and expandability. The Versity Gateway is designed with modularity in mind, enabling future extensions to support additional backend storage systems. At present, the Versity Gateway supports any generic POSIX file backend storage and Versity’s open source ScoutFS filesystem.  
+
+The gateway is completely stateless. Multiple Versity Gateway instances may be deployed in a cluster to increase aggregate throughput. The Versity Gateway’s stateless architecture allows any request to be serviced by any gateway thereby distributing workloads and enhancing performance. Load balancers may be used to evenly distribute requests across the cluster of gateways for optimal performance. 
 
 The S3 HTTP(S) server and routing is implemented using the [Fiber](https://gofiber.io) web framework.  This framework is actively developed with a focus on performance.  S3 API compatibility leverages the official [aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) whenever possible for maximum service compatibility with AWS S3. 
 
 ## Getting Started
+See the [Quickstart](https://github.com/versity/versitygw/wiki/Quickstart) documentation.
 
 ### Run the gateway with posix backend:
 
 ```
 mkdir /tmp/vgw
-ADMIN_ACCESS_KEY="testuser" ADMIN_SECRET_KEY="secret" ./versitygw --port :10000 posix /tmp/vgw
+ROOT_ACCESS_KEY="testuser" ROOT_SECRET_KEY="secret" ./versitygw --port :10000 posix /tmp/vgw
 ```
 This will enable an S3 server on the current host listening on port 10000 and hosting the directory `/tmp/vgw`.
 
@@ -34,3 +47,19 @@ The command format is
 versitygw [global options] command [command options] [arguments...]
 ```
 The global options are specified before the backend type and the backend options are specified after.
+
+***
+
+#### Versity gives you clarity and control over your archival storage, so you can allocate more resources to your core mission.
+
+### Contact
+![versity logo](https://www.versity.com/wp-content/uploads/2022/12/cropped-android-chrome-512x512-1-32x32.png)
+info@versity.com <br />
++1 844 726 8826
+
+### @versitysoftware 
+[![linkedin](https://github.com/versity/versitygw/blob/assets/assets/linkedin.jpg)](https://www.linkedin.com/company/versity/) &nbsp; 
+[![twitter](https://github.com/versity/versitygw/blob/assets/assets/twitter.jpg)](https://twitter.com/VersitySoftware) &nbsp;
+[![facebook](https://github.com/versity/versitygw/blob/assets/assets/facebook.jpg)](https://www.facebook.com/versitysoftware) &nbsp;
+[![instagram](https://github.com/versity/versitygw/blob/assets/assets/instagram.jpg)](https://www.instagram.com/versitysoftware/) &nbsp;
+

auth/acl.go

@@ -0,0 +1,242 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package auth
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/versity/versitygw/s3err"
+)
+
+type ACL struct {
+	ACL      types.BucketCannedACL
+	Owner    string
+	Grantees []Grantee
+}
+
+type Grantee struct {
+	Permission types.Permission
+	Access     string
+}
+
+type GetBucketAclOutput struct {
+	Owner             *types.Owner
+	AccessControlList AccessControlList
+}
+
+type AccessControlList struct {
+	Grants []types.Grant
+}
+
+func ParseACL(data []byte) (ACL, error) {
+	if len(data) == 0 {
+		return ACL{}, nil
+	}
+
+	var acl ACL
+	if err := json.Unmarshal(data, &acl); err != nil {
+		return acl, fmt.Errorf("parse acl: %w", err)
+	}
+	return acl, nil
+}
+
+func ParseACLOutput(data []byte) (GetBucketAclOutput, error) {
+	var acl ACL
+	if err := json.Unmarshal(data, &acl); err != nil {
+		return GetBucketAclOutput{}, fmt.Errorf("parse acl: %w", err)
+	}
+
+	grants := []types.Grant{}
+
+	for _, elem := range acl.Grantees {
+		acs := elem.Access
+		grants = append(grants, types.Grant{Grantee: &types.Grantee{ID: &acs}, Permission: elem.Permission})
+	}
+
+	return GetBucketAclOutput{
+		Owner: &types.Owner{
+			ID: &acl.Owner,
+		},
+		AccessControlList: AccessControlList{
+			Grants: grants,
+		},
+	}, nil
+}
+
+func UpdateACL(input *s3.PutBucketAclInput, acl ACL, iam IAMService) error {
+	if acl.Owner != *input.AccessControlPolicy.Owner.ID {
+		return s3err.GetAPIError(s3err.ErrAccessDenied)
+	}
+
+	// if the ACL is specified, set the ACL, else replace the grantees
+	if input.ACL != "" {
+		acl.ACL = input.ACL
+		acl.Grantees = []Grantee{}
+		return nil
+	}
+
+	grantees := []Grantee{}
+
+	fullControlList, readList, readACPList, writeList, writeACPList := []string{}, []string{}, []string{}, []string{}, []string{}
+
+	if *input.GrantFullControl != "" {
+		fullControlList = splitUnique(*input.GrantFullControl, ",")
+		fmt.Println(fullControlList)
+		for _, str := range fullControlList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "FULL_CONTROL"})
+		}
+	}
+	if *input.GrantRead != "" {
+		readList = splitUnique(*input.GrantRead, ",")
+		for _, str := range readList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "READ"})
+		}
+	}
+	if *input.GrantReadACP != "" {
+		readACPList = splitUnique(*input.GrantReadACP, ",")
+		for _, str := range readACPList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "READ_ACP"})
+		}
+	}
+	if *input.GrantWrite != "" {
+		writeList = splitUnique(*input.GrantWrite, ",")
+		for _, str := range writeList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "WRITE"})
+		}
+	}
+	if *input.GrantWriteACP != "" {
+		writeACPList = splitUnique(*input.GrantWriteACP, ",")
+		for _, str := range writeACPList {
+			grantees = append(grantees, Grantee{Access: str, Permission: "WRITE_ACP"})
+		}
+	}
+
+	accs := append(append(append(append(fullControlList, readList...), writeACPList...), readACPList...), writeList...)
+
+	// Check if the specified accounts exist
+	accList, err := checkIfAccountsExist(accs, iam)
+	if err != nil {
+		return err
+	}
+	if len(accList) > 0 {
+		return fmt.Errorf("accounts does not exist: %s", strings.Join(accList, ", "))
+	}
+
+	acl.Grantees = grantees
+	acl.ACL = ""
+
+	return nil
+}
+
+func checkIfAccountsExist(accs []string, iam IAMService) ([]string, error) {
+	result := []string{}
+
+	for _, acc := range accs {
+		_, err := iam.GetUserAccount(acc)
+		if err != nil && err != ErrNoSuchUser {
+			return nil, fmt.Errorf("check user account: %w", err)
+		}
+		if err == nil {
+			result = append(result, acc)
+		}
+	}
+	return result, nil
+}
+
+func splitUnique(s, divider string) []string {
+	elements := strings.Split(s, divider)
+	uniqueElements := make(map[string]bool)
+	result := make([]string, 0, len(elements))
+
+	for _, element := range elements {
+		if _, ok := uniqueElements[element]; !ok {
+			result = append(result, element)
+			uniqueElements[element] = true
+		}
+	}
+
+	return result
+}
+
+func VerifyACL(acl ACL, bucket, access string, permission types.Permission, isRoot bool) error {
+	if isRoot {
+		return nil
+	}
+
+	if acl.Owner == access {
+		return nil
+	}
+
+	if acl.ACL != "" {
+		if (permission == "READ" || permission == "READ_ACP") && (acl.ACL != "public-read" && acl.ACL != "public-read-write") {
+			return s3err.GetAPIError(s3err.ErrAccessDenied)
+		}
+		if (permission == "WRITE" || permission == "WRITE_ACP") && acl.ACL != "public-read-write" {
+			return s3err.GetAPIError(s3err.ErrAccessDenied)
+		}
+
+		return nil
+	} else {
+		grantee := Grantee{Access: access, Permission: permission}
+		granteeFullCtrl := Grantee{Access: access, Permission: "FULL_CONTROL"}
+
+		isFound := false
+
+		for _, grt := range acl.Grantees {
+			if grt == grantee || grt == granteeFullCtrl {
+				isFound = true
+				break
+			}
+		}
+
+		if isFound {
+			return nil
+		}
+	}
+
+	return s3err.GetAPIError(s3err.ErrAccessDenied)
+}
+
+func IsAdmin(access string, isRoot bool) error {
+	var data IAMConfig
+
+	if isRoot {
+		return nil
+	}
+
+	file, err := os.ReadFile("users.json")
+	if err != nil {
+		return fmt.Errorf("unable to read config file: %w", err)
+	}
+
+	if err := json.Unmarshal(file, &data); err != nil {
+		return err
+	}
+
+	acc, ok := data.AccessAccounts[access]
+	if !ok {
+		return fmt.Errorf("user does not exist")
+	}
+
+	if acc.Role == "admin" {
+		return nil
+	}
+	return fmt.Errorf("only admin users have access to this resource")
+}

auth/iam.go

@@ -14,24 +14,21 @@
 
 package auth
 
-import "github.com/versity/versitygw/s3err"
+import (
+	"errors"
+)
 
-type IAMConfig struct {
-	AccessAccounts map[string]string
+// Account is a gateway IAM account
+type Account struct {
+	Secret string `json:"secret"`
+	Role   string `json:"role"`
 }
 
+// IAMService is the interface for all IAM service implementations
 type IAMService interface {
-	GetIAMConfig() (*IAMConfig, error)
+	CreateAccount(access string, account Account) error
+	GetUserAccount(access string) (Account, error)
+	DeleteUserAccount(access string) error
 }
 
-type IAMServiceUnsupported struct{}
-
-var _ IAMService = &IAMServiceUnsupported{}
-
-func New() IAMService {
-	return &IAMServiceUnsupported{}
-}
-
-func (IAMServiceUnsupported) GetIAMConfig() (*IAMConfig, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
-}
+var ErrNoSuchUser = errors.New("user not found")

auth/iam_internal.go

@@ -0,0 +1,178 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package auth
+
+import (
+	"encoding/json"
+	"fmt"
+	"hash/crc32"
+	"sync"
+)
+
+// IAMServiceInternal manages the internal IAM service
+type IAMServiceInternal struct {
+	storer Storer
+
+	mu     sync.RWMutex
+	accts  IAMConfig
+	serial uint32
+}
+
+// UpdateAcctFunc accepts the current data and returns the new data to be stored
+type UpdateAcctFunc func([]byte) ([]byte, error)
+
+// Storer is the interface to manage the peristent IAM data for the internal
+// IAM service
+type Storer interface {
+	InitIAM() error
+	GetIAM() ([]byte, error)
+	StoreIAM(UpdateAcctFunc) error
+}
+
+// IAMConfig stores all internal IAM accounts
+type IAMConfig struct {
+	AccessAccounts map[string]Account `json:"accessAccounts"`
+}
+
+var _ IAMService = &IAMServiceInternal{}
+
+// NewInternal creates a new instance for the Internal IAM service
+func NewInternal(s Storer) (*IAMServiceInternal, error) {
+	i := &IAMServiceInternal{
+		storer: s,
+	}
+
+	err := i.updateCache()
+	if err != nil {
+		return nil, fmt.Errorf("refresh iam cache: %w", err)
+	}
+
+	return i, nil
+}
+
+// CreateAccount creates a new IAM account. Returns an error if the account
+// already exists.
+func (s *IAMServiceInternal) CreateAccount(access string, account Account) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	return s.storer.StoreIAM(func(data []byte) ([]byte, error) {
+		var conf IAMConfig
+
+		if len(data) > 0 {
+			if err := json.Unmarshal(data, &conf); err != nil {
+				return nil, fmt.Errorf("failed to parse iam: %w", err)
+			}
+		} else {
+			conf.AccessAccounts = make(map[string]Account)
+		}
+
+		_, ok := conf.AccessAccounts[access]
+		if ok {
+			return nil, fmt.Errorf("account already exists")
+		}
+		conf.AccessAccounts[access] = account
+
+		b, err := json.Marshal(s.accts)
+		if err != nil {
+			return nil, fmt.Errorf("failed to serialize iam: %w", err)
+		}
+
+		return b, nil
+	})
+}
+
+// GetUserAccount retrieves account info for the requested user. Returns
+// ErrNoSuchUser if the account does not exist.
+func (s *IAMServiceInternal) GetUserAccount(access string) (Account, error) {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+
+	data, err := s.storer.GetIAM()
+	if err != nil {
+		return Account{}, fmt.Errorf("get iam data: %w", err)
+	}
+
+	serial := crc32.ChecksumIEEE(data)
+	if serial != s.serial {
+		s.mu.RUnlock()
+		err := s.updateCache()
+		s.mu.RLock()
+		if err != nil {
+			return Account{}, fmt.Errorf("refresh iam cache: %w", err)
+		}
+	}
+
+	acct, ok := s.accts.AccessAccounts[access]
+	if !ok {
+		return Account{}, ErrNoSuchUser
+	}
+
+	return acct, nil
+}
+
+// updateCache must be called with no locks held
+func (s *IAMServiceInternal) updateCache() error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	data, err := s.storer.GetIAM()
+	if err != nil {
+		return fmt.Errorf("get iam data: %w", err)
+	}
+
+	serial := crc32.ChecksumIEEE(data)
+
+	if len(data) > 0 {
+		if err := json.Unmarshal(data, &s.accts); err != nil {
+			return fmt.Errorf("failed to parse the config file: %w", err)
+		}
+	} else {
+		s.accts.AccessAccounts = make(map[string]Account)
+	}
+
+	s.serial = serial
+
+	return nil
+}
+
+// DeleteUserAccount deletes the specified user account. Does not check if
+// account exists.
+func (s *IAMServiceInternal) DeleteUserAccount(access string) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	return s.storer.StoreIAM(func(data []byte) ([]byte, error) {
+		if len(data) == 0 {
+			// empty config, do nothing
+			return data, nil
+		}
+
+		var conf IAMConfig
+
+		if err := json.Unmarshal(data, &conf); err != nil {
+			return nil, fmt.Errorf("failed to parse iam: %w", err)
+		}
+
+		delete(conf.AccessAccounts, access)
+
+		b, err := json.Marshal(s.accts)
+		if err != nil {
+			return nil, fmt.Errorf("failed to serialize iam: %w", err)
+		}
+
+		return b, nil
+	})
+}

backend/backend.go

@@ -21,6 +21,7 @@ import (
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/versity/versitygw/s3err"
+	"github.com/versity/versitygw/s3response"
 )
 
 //go:generate moq -out backend_moq_test.go . Backend
@@ -29,20 +30,21 @@ type Backend interface {
 	fmt.Stringer
 	Shutdown()
 
-	ListBuckets() (*s3.ListBucketsOutput, error)
+	ListBuckets() (s3response.ListAllMyBucketsResult, error)
 	HeadBucket(bucket string) (*s3.HeadBucketOutput, error)
-	GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error)
-	PutBucket(bucket string) error
-	PutBucketAcl(*s3.PutBucketAclInput) error
+	GetBucketAcl(bucket string) ([]byte, error)
+	PutBucket(bucket, owner string) error
+	PutBucketAcl(bucket string, data []byte) error
 	DeleteBucket(bucket string) error
 
 	CreateMultipartUpload(*s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error)
 	CompleteMultipartUpload(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error)
 	AbortMultipartUpload(*s3.AbortMultipartUploadInput) error
-	ListMultipartUploads(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error)
-	ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error)
+	ListMultipartUploads(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error)
+	ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error)
 	CopyPart(srcBucket, srcObject, DstBucket, uploadID, rangeHeader string, part int) (*types.CopyPartResult, error)
 	PutObjectPart(bucket, object, uploadID string, part int, length int64, r io.Reader) (etag string, err error)
+	UploadPartCopy(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
 	PutObject(*s3.PutObjectInput) (string, error)
 	HeadObject(bucket, object string) (*s3.HeadObjectOutput, error)
@@ -56,8 +58,6 @@ type Backend interface {
 	DeleteObjects(bucket string, objects *s3.DeleteObjectsInput) error
 	PutObjectAcl(*s3.PutObjectAclInput) error
 	RestoreObject(bucket, object string, restoreRequest *s3.RestoreObjectInput) error
-	UploadPart(bucket, object, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error)
-	UploadPartCopy(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
 	GetTags(bucket, object string) (map[string]string, error)
 	SetTags(bucket, object string, tags map[string]string) error
@@ -75,10 +75,10 @@ func (BackendUnsupported) Shutdown() {}
 func (BackendUnsupported) String() string {
 	return "Unsupported"
 }
-func (BackendUnsupported) ListBuckets() (*s3.ListBucketsOutput, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+func (BackendUnsupported) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
+	return s3response.ListAllMyBucketsResult{}, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) PutBucketAcl(*s3.PutBucketAclInput) error {
+func (BackendUnsupported) PutBucketAcl(bucket string, data []byte) error {
 	return s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) PutObjectAcl(*s3.PutObjectAclInput) error {
@@ -90,16 +90,13 @@ func (BackendUnsupported) RestoreObject(bucket, object string, restoreRequest *s
 func (BackendUnsupported) UploadPartCopy(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) UploadPart(bucket, object, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
-}
-func (BackendUnsupported) GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error) {
+func (BackendUnsupported) GetBucketAcl(bucket string) ([]byte, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) HeadBucket(bucket string) (*s3.HeadBucketOutput, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) PutBucket(bucket string) error {
+func (BackendUnsupported) PutBucket(bucket, owner string) error {
 	return s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) DeleteBucket(bucket string) error {
@@ -115,11 +112,11 @@ func (BackendUnsupported) CompleteMultipartUpload(bucket, object, uploadID strin
 func (BackendUnsupported) AbortMultipartUpload(input *s3.AbortMultipartUploadInput) error {
 	return s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) ListMultipartUploads(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+func (BackendUnsupported) ListMultipartUploads(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
+	return s3response.ListMultipartUploadsResponse{}, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
-func (BackendUnsupported) ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
-	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+func (BackendUnsupported) ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
+	return s3response.ListPartsResponse{}, s3err.GetAPIError(s3err.ErrNotImplemented)
 }
 func (BackendUnsupported) CopyPart(srcBucket, srcObject, DstBucket, uploadID, rangeHeader string, part int) (*types.CopyPartResult, error) {
 	return nil, s3err.GetAPIError(s3err.ErrNotImplemented)

backend/backend_moq_test.go

@@ -6,6 +6,7 @@ package backend
 import (
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/versity/versitygw/s3response"
 	"io"
 	"sync"
 )
@@ -44,7 +45,7 @@ var _ Backend = &BackendMock{}
 //			DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
 //				panic("mock out the DeleteObjects method")
 //			},
-//			GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
+//			GetBucketAclFunc: func(bucket string) ([]byte, error) {
 //				panic("mock out the GetBucketAcl method")
 //			},
 //			GetObjectFunc: func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
@@ -65,13 +66,13 @@ var _ Backend = &BackendMock{}
 //			HeadObjectFunc: func(bucket string, object string) (*s3.HeadObjectOutput, error) {
 //				panic("mock out the HeadObject method")
 //			},
-//			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
+//			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
 //				panic("mock out the ListBuckets method")
 //			},
-//			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
+//			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
 //				panic("mock out the ListMultipartUploads method")
 //			},
-//			ListObjectPartsFunc: func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
+//			ListObjectPartsFunc: func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
 //				panic("mock out the ListObjectParts method")
 //			},
 //			ListObjectsFunc: func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
@@ -80,10 +81,10 @@ var _ Backend = &BackendMock{}
 //			ListObjectsV2Func: func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
 //				panic("mock out the ListObjectsV2 method")
 //			},
-//			PutBucketFunc: func(bucket string) error {
+//			PutBucketFunc: func(bucket string, owner string) error {
 //				panic("mock out the PutBucket method")
 //			},
-//			PutBucketAclFunc: func(putBucketAclInput *s3.PutBucketAclInput) error {
+//			PutBucketAclFunc: func(bucket string, data []byte) error {
 //				panic("mock out the PutBucketAcl method")
 //			},
 //			PutObjectFunc: func(putObjectInput *s3.PutObjectInput) (string, error) {
@@ -110,9 +111,6 @@ var _ Backend = &BackendMock{}
 //			StringFunc: func() string {
 //				panic("mock out the String method")
 //			},
-//			UploadPartFunc: func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-//				panic("mock out the UploadPart method")
-//			},
 //			UploadPartCopyFunc: func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 //				panic("mock out the UploadPartCopy method")
 //			},
@@ -148,7 +146,7 @@ type BackendMock struct {
 	DeleteObjectsFunc func(bucket string, objects *s3.DeleteObjectsInput) error
 
 	// GetBucketAclFunc mocks the GetBucketAcl method.
-	GetBucketAclFunc func(bucket string) (*s3.GetBucketAclOutput, error)
+	GetBucketAclFunc func(bucket string) ([]byte, error)
 
 	// GetObjectFunc mocks the GetObject method.
 	GetObjectFunc func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error)
@@ -169,13 +167,13 @@ type BackendMock struct {
 	HeadObjectFunc func(bucket string, object string) (*s3.HeadObjectOutput, error)
 
 	// ListBucketsFunc mocks the ListBuckets method.
-	ListBucketsFunc func() (*s3.ListBucketsOutput, error)
+	ListBucketsFunc func() (s3response.ListAllMyBucketsResult, error)
 
 	// ListMultipartUploadsFunc mocks the ListMultipartUploads method.
-	ListMultipartUploadsFunc func(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error)
+	ListMultipartUploadsFunc func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error)
 
 	// ListObjectPartsFunc mocks the ListObjectParts method.
-	ListObjectPartsFunc func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error)
+	ListObjectPartsFunc func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error)
 
 	// ListObjectsFunc mocks the ListObjects method.
 	ListObjectsFunc func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsOutput, error)
@@ -184,10 +182,10 @@ type BackendMock struct {
 	ListObjectsV2Func func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error)
 
 	// PutBucketFunc mocks the PutBucket method.
-	PutBucketFunc func(bucket string) error
+	PutBucketFunc func(bucket string, owner string) error
 
 	// PutBucketAclFunc mocks the PutBucketAcl method.
-	PutBucketAclFunc func(putBucketAclInput *s3.PutBucketAclInput) error
+	PutBucketAclFunc func(bucket string, data []byte) error
 
 	// PutObjectFunc mocks the PutObject method.
 	PutObjectFunc func(putObjectInput *s3.PutObjectInput) (string, error)
@@ -213,9 +211,6 @@ type BackendMock struct {
 	// StringFunc mocks the String method.
 	StringFunc func() string
 
-	// UploadPartFunc mocks the UploadPart method.
-	UploadPartFunc func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error)
-
 	// UploadPartCopyFunc mocks the UploadPartCopy method.
 	UploadPartCopyFunc func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
@@ -389,11 +384,15 @@ type BackendMock struct {
 		PutBucket []struct {
 			// Bucket is the bucket argument value.
 			Bucket string
+			// Owner is the owner argument value.
+			Owner string
 		}
 		// PutBucketAcl holds details about calls to the PutBucketAcl method.
 		PutBucketAcl []struct {
-			// PutBucketAclInput is the putBucketAclInput argument value.
-			PutBucketAclInput *s3.PutBucketAclInput
+			// Bucket is the bucket argument value.
+			Bucket string
+			// Data is the data argument value.
+			Data []byte
 		}
 		// PutObject holds details about calls to the PutObject method.
 		PutObject []struct {
@@ -451,17 +450,6 @@ type BackendMock struct {
 		// String holds details about calls to the String method.
 		String []struct {
 		}
-		// UploadPart holds details about calls to the UploadPart method.
-		UploadPart []struct {
-			// Bucket is the bucket argument value.
-			Bucket string
-			// Object is the object argument value.
-			Object string
-			// UploadId is the uploadId argument value.
-			UploadId string
-			// Body is the Body argument value.
-			Body io.ReadSeeker
-		}
 		// UploadPartCopy holds details about calls to the UploadPartCopy method.
 		UploadPartCopy []struct {
 			// UploadPartCopyInput is the uploadPartCopyInput argument value.
@@ -498,7 +486,6 @@ type BackendMock struct {
 	lockSetTags                 sync.RWMutex
 	lockShutdown                sync.RWMutex
 	lockString                  sync.RWMutex
-	lockUploadPart              sync.RWMutex
 	lockUploadPartCopy          sync.RWMutex
 }
 
@@ -811,7 +798,7 @@ func (mock *BackendMock) DeleteObjectsCalls() []struct {
 }
 
 // GetBucketAcl calls GetBucketAclFunc.
-func (mock *BackendMock) GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error) {
+func (mock *BackendMock) GetBucketAcl(bucket string) ([]byte, error) {
 	if mock.GetBucketAclFunc == nil {
 		panic("BackendMock.GetBucketAclFunc: method is nil but Backend.GetBucketAcl was just called")
 	}
@@ -1067,7 +1054,7 @@ func (mock *BackendMock) HeadObjectCalls() []struct {
 }
 
 // ListBuckets calls ListBucketsFunc.
-func (mock *BackendMock) ListBuckets() (*s3.ListBucketsOutput, error) {
+func (mock *BackendMock) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
 	if mock.ListBucketsFunc == nil {
 		panic("BackendMock.ListBucketsFunc: method is nil but Backend.ListBuckets was just called")
 	}
@@ -1094,7 +1081,7 @@ func (mock *BackendMock) ListBucketsCalls() []struct {
 }
 
 // ListMultipartUploads calls ListMultipartUploadsFunc.
-func (mock *BackendMock) ListMultipartUploads(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
+func (mock *BackendMock) ListMultipartUploads(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
 	if mock.ListMultipartUploadsFunc == nil {
 		panic("BackendMock.ListMultipartUploadsFunc: method is nil but Backend.ListMultipartUploads was just called")
 	}
@@ -1126,7 +1113,7 @@ func (mock *BackendMock) ListMultipartUploadsCalls() []struct {
 }
 
 // ListObjectParts calls ListObjectPartsFunc.
-func (mock *BackendMock) ListObjectParts(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
+func (mock *BackendMock) ListObjectParts(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
 	if mock.ListObjectPartsFunc == nil {
 		panic("BackendMock.ListObjectPartsFunc: method is nil but Backend.ListObjectParts was just called")
 	}
@@ -1270,19 +1257,21 @@ func (mock *BackendMock) ListObjectsV2Calls() []struct {
 }
 
 // PutBucket calls PutBucketFunc.
-func (mock *BackendMock) PutBucket(bucket string) error {
+func (mock *BackendMock) PutBucket(bucket string, owner string) error {
 	if mock.PutBucketFunc == nil {
 		panic("BackendMock.PutBucketFunc: method is nil but Backend.PutBucket was just called")
 	}
 	callInfo := struct {
 		Bucket string
+		Owner  string
 	}{
 		Bucket: bucket,
+		Owner:  owner,
 	}
 	mock.lockPutBucket.Lock()
 	mock.calls.PutBucket = append(mock.calls.PutBucket, callInfo)
 	mock.lockPutBucket.Unlock()
-	return mock.PutBucketFunc(bucket)
+	return mock.PutBucketFunc(bucket, owner)
 }
 
 // PutBucketCalls gets all the calls that were made to PutBucket.
@@ -1291,9 +1280,11 @@ func (mock *BackendMock) PutBucket(bucket string) error {
 //	len(mockedBackend.PutBucketCalls())
 func (mock *BackendMock) PutBucketCalls() []struct {
 	Bucket string
+	Owner  string
 } {
 	var calls []struct {
 		Bucket string
+		Owner  string
 	}
 	mock.lockPutBucket.RLock()
 	calls = mock.calls.PutBucket
@@ -1302,19 +1293,21 @@ func (mock *BackendMock) PutBucketCalls() []struct {
 }
 
 // PutBucketAcl calls PutBucketAclFunc.
-func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) error {
+func (mock *BackendMock) PutBucketAcl(bucket string, data []byte) error {
 	if mock.PutBucketAclFunc == nil {
 		panic("BackendMock.PutBucketAclFunc: method is nil but Backend.PutBucketAcl was just called")
 	}
 	callInfo := struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}{
-		PutBucketAclInput: putBucketAclInput,
+		Bucket: bucket,
+		Data:   data,
 	}
 	mock.lockPutBucketAcl.Lock()
 	mock.calls.PutBucketAcl = append(mock.calls.PutBucketAcl, callInfo)
 	mock.lockPutBucketAcl.Unlock()
-	return mock.PutBucketAclFunc(putBucketAclInput)
+	return mock.PutBucketAclFunc(bucket, data)
 }
 
 // PutBucketAclCalls gets all the calls that were made to PutBucketAcl.
@@ -1322,10 +1315,12 @@ func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) e
 //
 //	len(mockedBackend.PutBucketAclCalls())
 func (mock *BackendMock) PutBucketAclCalls() []struct {
-	PutBucketAclInput *s3.PutBucketAclInput
+	Bucket string
+	Data   []byte
 } {
 	var calls []struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}
 	mock.lockPutBucketAcl.RLock()
 	calls = mock.calls.PutBucketAcl
@@ -1619,50 +1614,6 @@ func (mock *BackendMock) StringCalls() []struct {
 	return calls
 }
 
-// UploadPart calls UploadPartFunc.
-func (mock *BackendMock) UploadPart(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-	if mock.UploadPartFunc == nil {
-		panic("BackendMock.UploadPartFunc: method is nil but Backend.UploadPart was just called")
-	}
-	callInfo := struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}{
-		Bucket:   bucket,
-		Object:   object,
-		UploadId: uploadId,
-		Body:     Body,
-	}
-	mock.lockUploadPart.Lock()
-	mock.calls.UploadPart = append(mock.calls.UploadPart, callInfo)
-	mock.lockUploadPart.Unlock()
-	return mock.UploadPartFunc(bucket, object, uploadId, Body)
-}
-
-// UploadPartCalls gets all the calls that were made to UploadPart.
-// Check the length with:
-//
-//	len(mockedBackend.UploadPartCalls())
-func (mock *BackendMock) UploadPartCalls() []struct {
-	Bucket   string
-	Object   string
-	UploadId string
-	Body     io.ReadSeeker
-} {
-	var calls []struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}
-	mock.lockUploadPart.RLock()
-	calls = mock.calls.UploadPart
-	mock.lockUploadPart.RUnlock()
-	return calls
-}
-
 // UploadPartCopy calls UploadPartCopyFunc.
 func (mock *BackendMock) UploadPartCopy(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 	if mock.UploadPartCopyFunc == nil {

backend/backend_test.go

@@ -18,10 +18,9 @@ import (
 	"context"
 	"testing"
 
-	"github.com/aws/aws-sdk-go-v2/aws"
 	"github.com/aws/aws-sdk-go-v2/service/s3"
-	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/versity/versitygw/s3err"
+	"github.com/versity/versitygw/s3response"
 )
 
 func TestBackend_ListBuckets(t *testing.T) {
@@ -38,11 +37,13 @@ func TestBackend_ListBuckets(t *testing.T) {
 	tests = append(tests, test{
 		name: "list-Bucket",
 		c: &BackendMock{
-			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
-				return &s3.ListBucketsOutput{
-					Buckets: []types.Bucket{
-						{
-							Name: aws.String("t1"),
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{
+					Buckets: s3response.ListAllMyBucketsList{
+						Bucket: []s3response.ListAllMyBucketsEntry{
+							{
+								Name: "t1",
+							},
 						},
 					},
 				}, s3err.GetAPIError(0)
@@ -55,8 +56,8 @@ func TestBackend_ListBuckets(t *testing.T) {
 	}, test{
 		name: "list-Bucket-error",
 		c: &BackendMock{
-			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
-				return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{}, s3err.GetAPIError(s3err.ErrNotImplemented)
 			},
 		},
 		args: args{
@@ -120,7 +121,7 @@ func TestBackend_GetBucketAcl(t *testing.T) {
 	tests = append(tests, test{
 		name: "get bucket acl error",
 		c: &BackendMock{
-			GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
 				return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
 			},
 		},
@@ -140,8 +141,9 @@ func TestBackend_GetBucketAcl(t *testing.T) {
 }
 func TestBackend_PutBucket(t *testing.T) {
 	type args struct {
-		ctx        context.Context
-		bucketName string
+		ctx         context.Context
+		bucketName  string
+		bucketOwner string
 	}
 	type test struct {
 		name    string
@@ -153,31 +155,33 @@ func TestBackend_PutBucket(t *testing.T) {
 	tests = append(tests, test{
 		name: "put bucket ",
 		c: &BackendMock{
-			PutBucketFunc: func(bucket string) error {
+			PutBucketFunc: func(bucket, owner string) error {
 				return s3err.GetAPIError(0)
 			},
 		},
 		args: args{
-			ctx:        context.Background(),
-			bucketName: "b1",
+			ctx:         context.Background(),
+			bucketName:  "b1",
+			bucketOwner: "owner",
 		},
 		wantErr: false,
 	}, test{
 		name: "put bucket error",
 		c: &BackendMock{
-			PutBucketFunc: func(bucket string) error {
+			PutBucketFunc: func(bucket, owner string) error {
 				return s3err.GetAPIError(s3err.ErrNotImplemented)
 			},
 		},
 		args: args{
-			ctx:        context.Background(),
-			bucketName: "b2",
+			ctx:         context.Background(),
+			bucketName:  "b2",
+			bucketOwner: "owner",
 		},
 		wantErr: true,
 	})
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			if err := tt.c.PutBucket(tt.args.bucketName); (err.(s3err.APIError).Code != "") != tt.wantErr {
+			if err := tt.c.PutBucket(tt.args.bucketName, tt.args.bucketOwner); (err.(s3err.APIError).Code != "") != tt.wantErr {
 				t.Errorf("Backend.PutBucket() error = %v, wantErr %v", err, tt.wantErr)
 			}
 		})

backend/common.go

@@ -15,22 +15,31 @@
 package backend
 
 import (
+	"crypto/md5"
+	"encoding/hex"
 	"errors"
+	"fmt"
 	"io/fs"
 	"strconv"
 	"strings"
 	"time"
 
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/versity/versitygw/s3response"
+)
+
+var (
+	// RFC3339TimeFormat RFC3339 time format
+	RFC3339TimeFormat = "2006-01-02T15:04:05.999Z"
 )
 
 func IsValidBucketName(name string) bool { return true }
 
-type ByBucketName []types.Bucket
+type ByBucketName []s3response.ListAllMyBucketsEntry
 
 func (d ByBucketName) Len() int           { return len(d) }
 func (d ByBucketName) Swap(i, j int)      { d[i], d[j] = d[j], d[i] }
-func (d ByBucketName) Less(i, j int) bool { return *d[i].Name < *d[j].Name }
+func (d ByBucketName) Less(i, j int) bool { return d[i].Name < d[j].Name }
 
 type ByObjectName []types.Object
 
@@ -78,3 +87,25 @@ func ParseRange(file fs.FileInfo, acceptRange string) (int64, int64, error) {
 
 	return int64(startOffset), int64(endOffset - startOffset + 1), nil
 }
+
+func GetMultipartMD5(parts []types.Part) string {
+	var partsEtagBytes []byte
+	for _, part := range parts {
+		partsEtagBytes = append(partsEtagBytes, getEtagBytes(*part.ETag)...)
+	}
+	s3MD5 := fmt.Sprintf("%s-%d", md5String(partsEtagBytes), len(parts))
+	return s3MD5
+}
+
+func getEtagBytes(etag string) []byte {
+	decode, err := hex.DecodeString(strings.ReplaceAll(etag, string('"'), ""))
+	if err != nil {
+		return []byte(etag)
+	}
+	return decode
+}
+
+func md5String(data []byte) string {
+	sum := md5.Sum(data)
+	return hex.EncodeToString(sum[:])
+}

backend/posix/posix.go

@@ -28,35 +28,50 @@ import (
 	"sort"
 	"strconv"
 	"strings"
+	"sync"
 	"syscall"
+	"time"
 
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/google/uuid"
 	"github.com/pkg/xattr"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3err"
+	"github.com/versity/versitygw/s3response"
 )
 
 type Posix struct {
+	backend.BackendUnsupported
+
 	rootfd  *os.File
 	rootdir string
-	backend.BackendUnsupported
+
+	mu        sync.RWMutex
+	iamcache  []byte
+	iamvalid  bool
+	iamexpire time.Time
 }
 
 var _ backend.Backend = &Posix{}
 
+var (
+	cacheDuration = 5 * time.Minute
+)
+
 const (
 	metaTmpDir          = ".sgwtmp"
 	metaTmpMultipartDir = metaTmpDir + "/multipart"
 	onameAttr           = "user.objname"
 	tagHdr              = "X-Amz-Tagging"
-	dirObjKey           = "user.dirisobject"
-)
-
-var (
-	newObjUID = 0
-	newObjGID = 0
+	contentTypeHdr      = "content-type"
+	contentEncHdr       = "content-encoding"
+	emptyMD5            = "d41d8cd98f00b204e9800998ecf8427e"
+	iamFile             = "users.json"
+	iamBackupFile       = "users.json.backup"
+	aclkey              = "user.acl"
+	etagkey             = "user.etag"
 )
 
 func New(rootdir string) (*Posix, error) {
@@ -77,17 +92,18 @@ func (p *Posix) Shutdown() {
 	p.rootfd.Close()
 }
 
-func (p *Posix) Sring() string {
+func (p *Posix) String() string {
 	return "Posix Gateway"
 }
 
-func (p *Posix) ListBuckets() (*s3.ListBucketsOutput, error) {
+func (p *Posix) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
 	entries, err := os.ReadDir(".")
 	if err != nil {
-		return nil, fmt.Errorf("readdir buckets: %w", err)
+		return s3response.ListAllMyBucketsResult{},
+			fmt.Errorf("readdir buckets: %w", err)
 	}
 
-	var buckets []types.Bucket
+	var buckets []s3response.ListAllMyBucketsEntry
 	for _, entry := range entries {
 		if !entry.IsDir() {
 			// buckets must be a directory
@@ -100,16 +116,18 @@ func (p *Posix) ListBuckets() (*s3.ListBucketsOutput, error) {
 			continue
 		}
 
-		buckets = append(buckets, types.Bucket{
-			Name:         backend.GetStringPtr(entry.Name()),
-			CreationDate: backend.GetTimePtr(fi.ModTime()),
+		buckets = append(buckets, s3response.ListAllMyBucketsEntry{
+			Name:         entry.Name(),
+			CreationDate: fi.ModTime(),
 		})
 	}
 
 	sort.Sort(backend.ByBucketName(buckets))
 
-	return &s3.ListBucketsOutput{
-		Buckets: buckets,
+	return s3response.ListAllMyBucketsResult{
+		Buckets: s3response.ListAllMyBucketsList{
+			Bucket: buckets,
+		},
 	}, nil
 }
 
@@ -125,7 +143,7 @@ func (p *Posix) HeadBucket(bucket string) (*s3.HeadBucketOutput, error) {
 	return &s3.HeadBucketOutput{}, nil
 }
 
-func (p *Posix) PutBucket(bucket string) error {
+func (p *Posix) PutBucket(bucket string, owner string) error {
 	err := os.Mkdir(bucket, 0777)
 	if err != nil && os.IsExist(err) {
 		return s3err.GetAPIError(s3err.ErrBucketAlreadyExists)
@@ -134,6 +152,16 @@ func (p *Posix) PutBucket(bucket string) error {
 		return fmt.Errorf("mkdir bucket: %w", err)
 	}
 
+	acl := auth.ACL{ACL: "private", Owner: owner, Grantees: []auth.Grantee{}}
+	jsonACL, err := json.Marshal(acl)
+	if err != nil {
+		return fmt.Errorf("marshal acl: %w", err)
+	}
+
+	if err := xattr.Set(bucket, aclkey, jsonACL); err != nil {
+		return fmt.Errorf("set acl: %w", err)
+	}
+
 	return nil
 }
 
@@ -236,8 +264,10 @@ func (p *Posix) CompleteMultipartUpload(bucket, object, uploadID string, parts [
 	// check all parts ok
 	last := len(parts) - 1
 	partsize := int64(0)
+	var totalsize int64
 	for i, p := range parts {
-		fi, err := os.Lstat(filepath.Join(objdir, uploadID, fmt.Sprintf("%v", p.PartNumber)))
+		partPath := filepath.Join(objdir, uploadID, fmt.Sprintf("%v", p.PartNumber))
+		fi, err := os.Lstat(partPath)
 		if err != nil {
 			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
 		}
@@ -245,13 +275,21 @@ func (p *Posix) CompleteMultipartUpload(bucket, object, uploadID string, parts [
 		if i == 0 {
 			partsize = fi.Size()
 		}
+		totalsize += fi.Size()
 		// all parts except the last need to be the same size
 		if i < last && partsize != fi.Size() {
 			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
 		}
+
+		b, err := xattr.Get(partPath, etagkey)
+		etag := string(b)
+		if err != nil {
+			etag = ""
+		}
+		parts[i].ETag = &etag
 	}
 
-	f, err := openTmpFile(filepath.Join(bucket, metaTmpDir), bucket, object, 0)
+	f, err := openTmpFile(filepath.Join(bucket, metaTmpDir), bucket, object, totalsize)
 	if err != nil {
 		return nil, fmt.Errorf("open temp file: %w", err)
 	}
@@ -277,11 +315,8 @@ func (p *Posix) CompleteMultipartUpload(bucket, object, uploadID string, parts [
 	dir := filepath.Dir(objname)
 	if dir != "" {
 		if err = mkdirAll(dir, os.FileMode(0755), bucket, object); err != nil {
-			if err != nil && os.IsExist(err) {
-				return nil, s3err.GetAPIError(s3err.ErrObjectParentIsFile)
-			}
 			if err != nil {
-				return nil, fmt.Errorf("make object parent directories: %w", err)
+				return nil, s3err.GetAPIError(s3err.ErrExistingObjectIsDirectory)
 			}
 		}
 	}
@@ -300,24 +335,15 @@ func (p *Posix) CompleteMultipartUpload(bucket, object, uploadID string, parts [
 	}
 
 	// Calculate s3 compatible md5sum for complete multipart.
-	s3MD5 := getMultipartMD5(parts)
+	s3MD5 := backend.GetMultipartMD5(parts)
 
-	err = xattr.Set(objname, "user.etag", []byte(s3MD5))
+	err = xattr.Set(objname, etagkey, []byte(s3MD5))
 	if err != nil {
 		// cleanup object if returning error
 		os.Remove(objname)
 		return nil, fmt.Errorf("set etag attr: %w", err)
 	}
 
-	if newObjUID != 0 || newObjGID != 0 {
-		err = os.Chown(objname, newObjUID, newObjGID)
-		if err != nil {
-			// cleanup object if returning error
-			os.Remove(objname)
-			return nil, fmt.Errorf("set object uid/gid: %w", err)
-		}
-	}
-
 	// cleanup tmp dirs
 	os.RemoveAll(upiddir)
 	// use Remove for objdir in case there are still other uploads
@@ -365,22 +391,22 @@ func loadUserMetaData(path string, m map[string]string) (contentType, contentEnc
 		m[strings.TrimPrefix(e, "user.")] = string(b)
 	}
 
-	b, err := xattr.Get(path, "user.content-type")
+	b, err := xattr.Get(path, "user."+contentTypeHdr)
 	contentType = string(b)
 	if err != nil {
 		contentType = ""
 	}
 	if contentType != "" {
-		m["content-type"] = contentType
+		m[contentTypeHdr] = contentType
 	}
 
-	b, err = xattr.Get(path, "user.content-encoding")
+	b, err = xattr.Get(path, "user."+contentEncHdr)
 	contentEncoding = string(b)
 	if err != nil {
 		contentEncoding = ""
 	}
 	if contentEncoding != "" {
-		m["content-encoding"] = contentEncoding
+		m[contentEncHdr] = contentEncoding
 	}
 
 	return
@@ -396,8 +422,8 @@ func isValidMeta(val string) bool {
 	return false
 }
 
-// mkdirAll is similar to os.MkdirAll but it will also set uid/gid when
-// making new directories
+// mkdirAll is similar to os.MkdirAll but it will return ErrObjectParentIsFile
+// when appropriate
 func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
 	// Fast path: if we can tell whether path is a directory or file, stop with success or error.
 	dir, err := os.Stat(path)
@@ -405,7 +431,7 @@ func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
 		if dir.IsDir() {
 			return nil
 		}
-		return &os.PathError{Op: "mkdir", Path: path, Err: syscall.ENOTDIR}
+		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
 	}
 
 	// Slow path: make sure parent exists and then call Mkdir for path.
@@ -438,37 +464,9 @@ func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
 		}
 		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
 	}
-	if newObjUID != 0 || newObjGID != 0 {
-		err = os.Chown(path, newObjUID, newObjGID)
-		if err != nil {
-			return fmt.Errorf("set parent ownership: %w", err)
-		}
-	}
 	return nil
 }
 
-func getMultipartMD5(parts []types.Part) string {
-	var partsEtagBytes []byte
-	for _, part := range parts {
-		partsEtagBytes = append(partsEtagBytes, getEtagBytes(*part.ETag)...)
-	}
-	s3MD5 := fmt.Sprintf("%s-%d", md5String(partsEtagBytes), len(parts))
-	return s3MD5
-}
-
-func getEtagBytes(etag string) []byte {
-	decode, err := hex.DecodeString(strings.ReplaceAll(etag, string('"'), ""))
-	if err != nil {
-		return []byte(etag)
-	}
-	return decode
-}
-
-func md5String(data []byte) string {
-	sum := md5.Sum(data)
-	return hex.EncodeToString(sum[:])
-}
-
 func (p *Posix) AbortMultipartUpload(mpu *s3.AbortMultipartUploadInput) error {
 	bucket := *mpu.Bucket
 	object := *mpu.Key
@@ -499,24 +497,40 @@ func (p *Posix) AbortMultipartUpload(mpu *s3.AbortMultipartUploadInput) error {
 	return nil
 }
 
-func (p *Posix) ListMultipartUploads(mpu *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
+func (p *Posix) ListMultipartUploads(mpu *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
 	bucket := *mpu.Bucket
+	var delimiter string
+	if mpu.Delimiter != nil {
+		delimiter = *mpu.Delimiter
+	}
+	var prefix string
+	if mpu.Prefix != nil {
+		prefix = *mpu.Prefix
+	}
+
+	var lmu s3response.ListMultipartUploadsResponse
 
 	_, err := os.Stat(bucket)
 	if errors.Is(err, fs.ErrNotExist) {
-		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+		return lmu, s3err.GetAPIError(s3err.ErrNoSuchBucket)
 	}
 	if err != nil {
-		return nil, fmt.Errorf("stat bucket: %w", err)
+		return lmu, fmt.Errorf("stat bucket: %w", err)
 	}
 
 	// ignore readdir error and use the empty list returned
 	objs, _ := os.ReadDir(filepath.Join(bucket, metaTmpMultipartDir))
 
-	var uploads []types.MultipartUpload
+	var uploads []s3response.Upload
 
-	keyMarker := *mpu.KeyMarker
-	uploadIDMarker := *mpu.UploadIdMarker
+	var keyMarker string
+	if mpu.KeyMarker != nil {
+		keyMarker = *mpu.KeyMarker
+	}
+	var uploadIDMarker string
+	if mpu.UploadIdMarker != nil {
+		uploadIDMarker = *mpu.UploadIdMarker
+	}
 	var pastMarker bool
 	if keyMarker == "" && uploadIDMarker == "" {
 		pastMarker = true
@@ -532,7 +546,7 @@ func (p *Posix) ListMultipartUploads(mpu *s3.ListMultipartUploadsInput) (*s3.Lis
 			continue
 		}
 		objectName := string(b)
-		if !strings.HasPrefix(objectName, *mpu.Prefix) {
+		if mpu.Prefix != nil && !strings.HasPrefix(objectName, *mpu.Prefix) {
 			continue
 		}
 
@@ -558,64 +572,71 @@ func (p *Posix) ListMultipartUploads(mpu *s3.ListMultipartUploadsInput) (*s3.Lis
 			upiddir := filepath.Join(bucket, metaTmpMultipartDir, obj.Name(), upid.Name())
 			loadUserMetaData(upiddir, userMetaData)
 
+			fi, err := upid.Info()
+			if err != nil {
+				return lmu, fmt.Errorf("stat %q: %w", upid.Name(), err)
+			}
+
 			uploadID := upid.Name()
-			uploads = append(uploads, types.MultipartUpload{
-				Key:      &objectName,
-				UploadId: &uploadID,
+			uploads = append(uploads, s3response.Upload{
+				Key:       objectName,
+				UploadID:  uploadID,
+				Initiated: fi.ModTime().Format(backend.RFC3339TimeFormat),
 			})
 			if len(uploads) == int(mpu.MaxUploads) {
-				return &s3.ListMultipartUploadsOutput{
-					Bucket:             &bucket,
-					Delimiter:          mpu.Delimiter,
+				return s3response.ListMultipartUploadsResponse{
+					Bucket:             bucket,
+					Delimiter:          delimiter,
 					IsTruncated:        i != len(objs) || j != len(upids),
-					KeyMarker:          &keyMarker,
-					MaxUploads:         mpu.MaxUploads,
-					NextKeyMarker:      &objectName,
-					NextUploadIdMarker: &uploadID,
-					Prefix:             mpu.Prefix,
-					UploadIdMarker:     mpu.UploadIdMarker,
+					KeyMarker:          keyMarker,
+					MaxUploads:         int(mpu.MaxUploads),
+					NextKeyMarker:      objectName,
+					NextUploadIDMarker: uploadID,
+					Prefix:             prefix,
+					UploadIDMarker:     uploadIDMarker,
 					Uploads:            uploads,
 				}, nil
 			}
 		}
 	}
 
-	return &s3.ListMultipartUploadsOutput{
-		Bucket:         &bucket,
-		Delimiter:      mpu.Delimiter,
-		KeyMarker:      &keyMarker,
-		MaxUploads:     mpu.MaxUploads,
-		Prefix:         mpu.Prefix,
-		UploadIdMarker: mpu.UploadIdMarker,
+	return s3response.ListMultipartUploadsResponse{
+		Bucket:         bucket,
+		Delimiter:      delimiter,
+		KeyMarker:      keyMarker,
+		MaxUploads:     int(mpu.MaxUploads),
+		Prefix:         prefix,
+		UploadIDMarker: uploadIDMarker,
 		Uploads:        uploads,
 	}, nil
 }
 
-func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
+func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
+	var lpr s3response.ListPartsResponse
 	_, err := os.Stat(bucket)
 	if errors.Is(err, fs.ErrNotExist) {
-		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+		return lpr, s3err.GetAPIError(s3err.ErrNoSuchBucket)
 	}
 	if err != nil {
-		return nil, fmt.Errorf("stat bucket: %w", err)
+		return lpr, fmt.Errorf("stat bucket: %w", err)
 	}
 
 	sum, err := p.checkUploadIDExists(bucket, object, uploadID)
 	if err != nil {
-		return nil, err
+		return lpr, err
 	}
 
 	objdir := filepath.Join(bucket, metaTmpMultipartDir, fmt.Sprintf("%x", sum))
 
 	ents, err := os.ReadDir(filepath.Join(objdir, uploadID))
 	if errors.Is(err, fs.ErrNotExist) {
-		return nil, s3err.GetAPIError(s3err.ErrNoSuchUpload)
+		return lpr, s3err.GetAPIError(s3err.ErrNoSuchUpload)
 	}
 	if err != nil {
-		return nil, fmt.Errorf("readdir upload: %w", err)
+		return lpr, fmt.Errorf("readdir upload: %w", err)
 	}
 
-	var parts []types.Part
+	var parts []s3response.Part
 	for _, e := range ents {
 		pn, _ := strconv.Atoi(e.Name())
 		if pn <= partNumberMarker {
@@ -623,7 +644,7 @@ func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarke
 		}
 
 		partPath := filepath.Join(objdir, uploadID, e.Name())
-		b, err := xattr.Get(partPath, "user.etag")
+		b, err := xattr.Get(partPath, etagkey)
 		etag := string(b)
 		if err != nil {
 			etag = ""
@@ -634,10 +655,10 @@ func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarke
 			continue
 		}
 
-		parts = append(parts, types.Part{
-			PartNumber:   int32(pn),
-			ETag:         &etag,
-			LastModified: backend.GetTimePtr(fi.ModTime()),
+		parts = append(parts, s3response.Part{
+			PartNumber:   pn,
+			ETag:         etag,
+			LastModified: fi.ModTime().Format(backend.RFC3339TimeFormat),
 			Size:         fi.Size(),
 		})
 	}
@@ -646,12 +667,12 @@ func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarke
 		func(i int, j int) bool { return parts[i].PartNumber < parts[j].PartNumber })
 
 	oldLen := len(parts)
-	if len(parts) > maxParts {
+	if maxParts > 0 && len(parts) > maxParts {
 		parts = parts[:maxParts]
 	}
 	newLen := len(parts)
 
-	nextpart := int32(0)
+	nextpart := 0
 	if len(parts) != 0 {
 		nextpart = parts[len(parts)-1].PartNumber
 	}
@@ -660,15 +681,15 @@ func (p *Posix) ListObjectParts(bucket, object, uploadID string, partNumberMarke
 	upiddir := filepath.Join(objdir, uploadID)
 	loadUserMetaData(upiddir, userMetaData)
 
-	return &s3.ListPartsOutput{
-		Bucket:               &bucket,
+	return s3response.ListPartsResponse{
+		Bucket:               bucket,
 		IsTruncated:          oldLen != newLen,
-		Key:                  &object,
-		MaxParts:             int32(maxParts),
-		NextPartNumberMarker: backend.GetStringPtr(fmt.Sprintf("%v", nextpart)),
-		PartNumberMarker:     backend.GetStringPtr(fmt.Sprintf("%v", partNumberMarker)),
+		Key:                  object,
+		MaxParts:             maxParts,
+		NextPartNumberMarker: nextpart,
+		PartNumberMarker:     partNumberMarker,
 		Parts:                parts,
-		UploadId:             &uploadID,
+		UploadID:             uploadID,
 	}, nil
 }
 
@@ -709,8 +730,8 @@ func (p *Posix) PutObjectPart(bucket, object, uploadID string, part int, length
 	}
 
 	dataSum := hash.Sum(nil)
-	etag := hex.EncodeToString(dataSum[:])
-	xattr.Set(partPath, "user.etag", []byte(etag))
+	etag := hex.EncodeToString(dataSum)
+	xattr.Set(partPath, etagkey, []byte(etag))
 
 	return etag, nil
 }
@@ -737,13 +758,10 @@ func (p *Posix) PutObject(po *s3.PutObjectInput) (string, error) {
 			xattr.Set(name, "user."+k, []byte(v))
 		}
 
-		// set our attribute that this dir was specifically put
-		xattr.Set(name, dirObjKey, nil)
+		// set etag attribute to signify this dir was specifically put
+		xattr.Set(name, etagkey, []byte(emptyMD5))
 
-		// TODO: what etag should be returned here
-		// and we should set etag xattr to identify dir was
-		// specifically uploaded
-		return "", nil
+		return emptyMD5, nil
 	}
 
 	// object is file
@@ -764,7 +782,7 @@ func (p *Posix) PutObject(po *s3.PutObjectInput) (string, error) {
 	if dir != "" {
 		err = mkdirAll(dir, os.FileMode(0755), *po.Bucket, *po.Key)
 		if err != nil {
-			return "", fmt.Errorf("make object parent directories: %w", err)
+			return "", s3err.GetAPIError(s3err.ErrExistingObjectIsDirectory)
 		}
 	}
 
@@ -779,14 +797,7 @@ func (p *Posix) PutObject(po *s3.PutObjectInput) (string, error) {
 
 	dataSum := hash.Sum(nil)
 	etag := hex.EncodeToString(dataSum[:])
-	xattr.Set(name, "user.etag", []byte(etag))
-
-	if newObjUID != 0 || newObjGID != 0 {
-		err = os.Chown(name, newObjUID, newObjGID)
-		if err != nil {
-			return "", fmt.Errorf("set object uid/gid: %v", err)
-		}
-	}
+	xattr.Set(name, etagkey, []byte(etag))
 
 	return etag, nil
 }
@@ -823,11 +834,15 @@ func (p *Posix) removeParents(bucket, object string) error {
 		parent := filepath.Dir(objPath)
 
 		if filepath.Base(parent) == bucket {
+			// stop removing parents if we hit the bucket directory.
 			break
 		}
 
-		_, err := xattr.Get(parent, dirObjKey)
+		_, err := xattr.Get(parent, etagkey)
 		if err == nil {
+			// a directory with a valid etag means this was specifically
+			// uploaded with a put object, so stop here and leave this
+			// directory in place.
 			break
 		}
 
@@ -900,7 +915,7 @@ func (p *Posix) GetObject(bucket, object, acceptRange string, writer io.Writer)
 
 	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
 
-	b, err := xattr.Get(objPath, "user.etag")
+	b, err := xattr.Get(objPath, etagkey)
 	etag := string(b)
 	if err != nil {
 		etag = ""
@@ -944,7 +959,7 @@ func (p *Posix) HeadObject(bucket, object string) (*s3.HeadObjectOutput, error)
 	userMetaData := make(map[string]string)
 	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
 
-	b, err := xattr.Get(objPath, "user.etag")
+	b, err := xattr.Get(objPath, etagkey)
 	etag := string(b)
 	if err != nil {
 		etag = ""
@@ -1015,7 +1030,8 @@ func (p *Posix) ListObjects(bucket, prefix, marker, delim string, maxkeys int) (
 	}
 
 	fileSystem := os.DirFS(bucket)
-	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys)
+	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
+		fileToObj(bucket), []string{metaTmpDir})
 	if err != nil {
 		return nil, fmt.Errorf("walk %v: %w", bucket, err)
 	}
@@ -1033,6 +1049,67 @@ func (p *Posix) ListObjects(bucket, prefix, marker, delim string, maxkeys int) (
 	}, nil
 }
 
+func fileToObj(bucket string) backend.GetObjFunc {
+	return func(path string, d fs.DirEntry) (types.Object, error) {
+		if d.IsDir() {
+			// directory object only happens if directory empty
+			// check to see if this is a directory object by checking etag
+			etagBytes, err := xattr.Get(filepath.Join(bucket, path), etagkey)
+			if isNoAttr(err) || errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get etag: %w", err)
+			}
+			etag := string(etagBytes)
+
+			fi, err := d.Info()
+			if errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+			}
+
+			key := path + "/"
+
+			return types.Object{
+				ETag:         &etag,
+				Key:          &key,
+				LastModified: backend.GetTimePtr(fi.ModTime()),
+			}, nil
+		}
+
+		// file object, get object info and fill out object data
+		etagBytes, err := xattr.Get(filepath.Join(bucket, path), etagkey)
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil && !isNoAttr(err) {
+			return types.Object{}, fmt.Errorf("get etag: %w", err)
+		}
+		// note: isNoAttr(err) will return etagBytes = []byte{}
+		// so this will just set etag to "" if its not already set
+
+		etag := string(etagBytes)
+
+		fi, err := d.Info()
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil {
+			return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+		}
+
+		return types.Object{
+			ETag:         &etag,
+			Key:          &path,
+			LastModified: backend.GetTimePtr(fi.ModTime()),
+			Size:         fi.Size(),
+		}, nil
+	}
+}
+
 func (p *Posix) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
 	_, err := os.Stat(bucket)
 	if errors.Is(err, fs.ErrNotExist) {
@@ -1043,7 +1120,8 @@ func (p *Posix) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int)
 	}
 
 	fileSystem := os.DirFS(bucket)
-	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys)
+	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
+		fileToObj(bucket), []string{metaTmpDir})
 	if err != nil {
 		return nil, fmt.Errorf("walk %v: %w", bucket, err)
 	}
@@ -1061,6 +1139,41 @@ func (p *Posix) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int)
 	}, nil
 }
 
+func (p *Posix) PutBucketAcl(bucket string, data []byte) error {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return fmt.Errorf("stat bucket: %w", err)
+	}
+
+	if err := xattr.Set(bucket, aclkey, data); err != nil {
+		return fmt.Errorf("set acl: %w", err)
+	}
+
+	return nil
+}
+
+func (p *Posix) GetBucketAcl(bucket string) ([]byte, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	b, err := xattr.Get(bucket, aclkey)
+	if isNoAttr(err) {
+		return []byte{}, nil
+	}
+	if err != nil {
+		return nil, fmt.Errorf("get acl: %w", err)
+	}
+	return b, nil
+}
+
 func (p *Posix) GetTags(bucket, object string) (map[string]string, error) {
 	_, err := os.Stat(bucket)
 	if errors.Is(err, fs.ErrNotExist) {
@@ -1134,6 +1247,198 @@ func (p *Posix) RemoveTags(bucket, object string) error {
 	return p.SetTags(bucket, object, nil)
 }
 
+const (
+	iamMode = 0600
+)
+
+func (p *Posix) InitIAM() error {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+
+	_, err := os.ReadFile(iamFile)
+	if errors.Is(err, fs.ErrNotExist) {
+		b, err := json.Marshal(auth.IAMConfig{})
+		if err != nil {
+			return fmt.Errorf("marshal default iam: %w", err)
+		}
+		err = os.WriteFile(iamFile, b, iamMode)
+		if err != nil {
+			return fmt.Errorf("write default iam: %w", err)
+		}
+	}
+
+	return nil
+}
+
+func (p *Posix) GetIAM() ([]byte, error) {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+
+	if !p.iamvalid || !p.iamexpire.After(time.Now()) {
+		p.mu.RUnlock()
+		err := p.refreshIAM()
+		p.mu.RLock()
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return p.iamcache, nil
+}
+
+const (
+	backoff  = 100 * time.Millisecond
+	maxretry = 300
+)
+
+func (p *Posix) refreshIAM() error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	// We are going to be racing with other running gateways without any
+	// coordination. So we might find the file does not exist at times.
+	// For this case we need to retry for a while assuming the other gateway
+	// will eventually write the file. If it doesn't after the max retries,
+	// then we will return the error.
+
+	retries := 0
+
+	for {
+		b, err := os.ReadFile(iamFile)
+		if errors.Is(err, fs.ErrNotExist) {
+			// racing with someone else updating
+			// keep retrying after backoff
+			retries++
+			if retries < maxretry {
+				time.Sleep(backoff)
+				continue
+			}
+			return fmt.Errorf("read iam file: %w", err)
+		}
+		if err != nil {
+			return err
+		}
+
+		p.iamcache = b
+		p.iamvalid = true
+		p.iamexpire = time.Now().Add(cacheDuration)
+		break
+	}
+
+	return nil
+}
+
+func (p *Posix) StoreIAM(update auth.UpdateAcctFunc) error {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	// We are going to be racing with other running gateways without any
+	// coordination. So the strategy here is to read the current file data.
+	// If the file doesn't exist, then we assume someone else is currently
+	// updating the file. So we just need to keep retrying. We also need
+	// to make sure the data is consistent within a single update. So racing
+	// writes to a file would possibly leave this in some invalid state.
+	// We can get atomic updates with rename. If we read the data, update
+	// the data, write to a temp file, then rename the tempfile back to the
+	// data file. This should always result in a complete data image.
+
+	// There is at least one unsolved failure mode here.
+	// If a gateway removes the data file and then crashes, all other
+	// gateways will retry forever thinking that the original will eventually
+	// write the file.
+
+	retries := 0
+
+	for {
+		b, err := os.ReadFile(iamFile)
+		if errors.Is(err, fs.ErrNotExist) {
+			// racing with someone else updating
+			// keep retrying after backoff
+			retries++
+			if retries < maxretry {
+				time.Sleep(backoff)
+				continue
+			}
+
+			// we have been unsuccessful trying to read the iam file
+			// so this must be the case where something happened and
+			// the file did not get updated successfully, and probably
+			// isn't going to be. The recovery procedure would be to
+			// copy the backup file into place of the original.
+			return fmt.Errorf("no iam file, needs backup recovery")
+		}
+		if err != nil && !errors.Is(err, fs.ErrNotExist) {
+			return fmt.Errorf("read iam file: %w", err)
+		}
+
+		// reset retries on successful read
+		retries = 0
+
+		err = os.Remove(iamFile)
+		if errors.Is(err, fs.ErrNotExist) {
+			// racing with someone else updating
+			// keep retrying after backoff
+			time.Sleep(backoff)
+			continue
+		}
+		if err != nil && !errors.Is(err, fs.ErrNotExist) {
+			return fmt.Errorf("remove old iam file: %w", err)
+		}
+
+		// save copy of data
+		datacopy := make([]byte, len(b))
+		copy(datacopy, b)
+
+		// make a backup copy in case we crash before update
+		// this is after remove, so there is a small window something
+		// can go wrong, but the remove should barrier other gateways
+		// from trying to write backup at the same time. Only one
+		// gateway will successfully remove the file.
+		os.WriteFile(iamBackupFile, b, iamMode)
+
+		b, err = update(b)
+		if err != nil {
+			// update failed, try to write old data back out
+			os.WriteFile(iamFile, datacopy, iamMode)
+			return fmt.Errorf("update iam data: %w", err)
+		}
+
+		err = writeTempFile(b)
+		if err != nil {
+			// update failed, try to write old data back out
+			os.WriteFile(iamFile, datacopy, iamMode)
+			return err
+		}
+
+		p.iamcache = b
+		p.iamvalid = true
+		p.iamexpire = time.Now().Add(cacheDuration)
+		break
+	}
+
+	return nil
+}
+
+func writeTempFile(b []byte) error {
+	f, err := os.CreateTemp(".", iamFile)
+	if err != nil {
+		return fmt.Errorf("create temp file: %w", err)
+	}
+	defer os.Remove(f.Name())
+
+	_, err = f.Write(b)
+	if err != nil {
+		return fmt.Errorf("write temp file: %w", err)
+	}
+
+	err = os.Rename(f.Name(), iamFile)
+	if err != nil {
+		return fmt.Errorf("rename temp file: %w", err)
+	}
+
+	return nil
+}
+
 func isNoAttr(err error) bool {
 	if err == nil {
 		return false

backend/posix/posix_darwin.go

@@ -76,7 +76,7 @@ func (tmp *tmpfile) link() error {
 
 func (tmp *tmpfile) Write(b []byte) (int, error) {
 	if int64(len(b)) > tmp.size {
-		return 0, fmt.Errorf("write exceeds content length")
+		return 0, fmt.Errorf("write exceeds content length %v", tmp.size)
 	}
 
 	n, err := tmp.f.Write(b)

backend/posix/posix_linux.go

@@ -68,7 +68,7 @@ func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
 	// later to link file into namespace
 	f := os.NewFile(uintptr(fd), filepath.Join(procfddir, strconv.Itoa(fd)))
 
-	tmp := &tmpfile{f: f, isOTmp: true, size: size}
+	tmp := &tmpfile{f: f, bucket: bucket, objname: obj, isOTmp: true, size: size}
 	// falloc is best effort, its fine if this fails
 	if size > 0 {
 		tmp.falloc()
@@ -117,7 +117,8 @@ func (tmp *tmpfile) link() error {
 	err = unix.Linkat(int(procdir.Fd()), filepath.Base(tmp.f.Name()),
 		int(dir.Fd()), filepath.Base(objPath), unix.AT_SYMLINK_FOLLOW)
 	if err != nil {
-		return fmt.Errorf("link tmpfile: %w", err)
+		return fmt.Errorf("link tmpfile (%q in %q): %w",
+			filepath.Dir(objPath), filepath.Base(tmp.f.Name()), err)
 	}
 
 	err = tmp.f.Close()
@@ -150,7 +151,7 @@ func (tmp *tmpfile) fallbackLink() error {
 
 func (tmp *tmpfile) Write(b []byte) (int, error) {
 	if int64(len(b)) > tmp.size {
-		return 0, fmt.Errorf("write exceeds content length")
+		return 0, fmt.Errorf("write exceeds content length %v", tmp.size)
 	}
 
 	n, err := tmp.f.Write(b)

backend/scoutfs/scoutfs.go

@@ -15,12 +15,731 @@
 package scoutfs
 
 import (
+	"crypto/sha256"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/pkg/xattr"
+	"github.com/versity/scoutfs-go"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/backend/posix"
+	"github.com/versity/versitygw/s3err"
 )
 
 type ScoutFS struct {
 	*posix.Posix
+	rootfd  *os.File
+	rootdir string
+
+	// glaciermode enables the following behavior:
+	// GET object:  if file offline, return invalid object state
+	// HEAD object: if file offline, set obj storage class to GLACIER
+	//              if file offline and staging, x-amz-restore: ongoing-request="true"
+	//              if file offline and not staging, x-amz-restore: ongoing-request="false"
+	//              if file online, x-amz-restore: ongoing-request="false", expiry-date="Fri, 2 Dec 2050 00:00:00 GMT"
+	//              note: this expiry-date is not used but provided for client glacier compatibility
+	// ListObjects: if file offline, set obj storage class to GLACIER
+	// RestoreObject: add batch stage request to file
+	glaciermode bool
 }
 
-var _ backend.Backend = ScoutFS{}
+var _ backend.Backend = &ScoutFS{}
+
+const (
+	metaTmpDir          = ".sgwtmp"
+	metaTmpMultipartDir = metaTmpDir + "/multipart"
+	tagHdr              = "X-Amz-Tagging"
+	emptyMD5            = "d41d8cd98f00b204e9800998ecf8427e"
+	etagkey             = "user.etag"
+)
+
+var (
+	stageComplete      = "ongoing-request=\"false\", expiry-date=\"Fri, 2 Dec 2050 00:00:00 GMT\""
+	stageInProgress    = "true"
+	stageNotInProgress = "false"
+)
+
+const (
+	// ScoutFS special xattr types
+
+	systemPrefix = "scoutfs.hide."
+	onameAttr    = systemPrefix + "objname"
+	flagskey     = systemPrefix + "sam_flags"
+	stagecopykey = systemPrefix + "sam_stagereq"
+)
+
+const (
+	// ScoutAM Flags
+
+	// Staging - file requested stage
+	Staging uint64 = 1 << iota
+	// StageFail - all copies failed to stage
+	StageFail
+	// NoArchive - no archive copies of file should be made
+	NoArchive
+	// ExtCacheRequested means file policy requests Ext Cache
+	ExtCacheRequested
+	// ExtCacheDone means this file ext cache copy has been
+	// created already (and possibly pruned, so may not exist)
+	ExtCacheDone
+)
+
+// Option sets various options for scoutfs
+type Option func(s *ScoutFS)
+
+// WithGlacierEmulation sets glacier mode emulation
+func WithGlacierEmulation() Option {
+	return func(s *ScoutFS) { s.glaciermode = true }
+}
+
+func (s *ScoutFS) Shutdown() {
+	s.Posix.Shutdown()
+	s.rootfd.Close()
+	_ = s.rootdir
+}
+
+func (*ScoutFS) String() string {
+	return "ScoutFS Gateway"
+}
+
+// CompleteMultipartUpload scoutfs complete upload uses scoutfs move blocks
+// ioctl to not have to read and copy the part data to the final object. This
+// saves a read and write cycle for all mutlipart uploads.
+func (s *ScoutFS) CompleteMultipartUpload(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	sum, err := s.checkUploadIDExists(bucket, object, uploadID)
+	if err != nil {
+		return nil, err
+	}
+
+	objdir := filepath.Join(bucket, metaTmpMultipartDir, fmt.Sprintf("%x", sum))
+
+	// check all parts ok
+	last := len(parts) - 1
+	partsize := int64(0)
+	var totalsize int64
+	for i, p := range parts {
+		partPath := filepath.Join(objdir, uploadID, fmt.Sprintf("%v", p.PartNumber))
+		fi, err := os.Lstat(partPath)
+		if err != nil {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
+		}
+
+		if i == 0 {
+			partsize = fi.Size()
+		}
+		totalsize += fi.Size()
+		// all parts except the last need to be the same size
+		if i < last && partsize != fi.Size() {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
+		}
+		// non-last part sizes need to be multiples of 4k for move blocks
+		// TODO: fallback to no move blocks if not 4k aligned?
+		if i == 0 && i < last && fi.Size()%4096 != 0 {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidPart)
+		}
+
+		b, err := xattr.Get(partPath, "user.etag")
+		etag := string(b)
+		if err != nil {
+			etag = ""
+		}
+		parts[i].ETag = &etag
+	}
+
+	// use totalsize=0 because we wont be writing to the file, only moving
+	// extents around.  so we dont want to fallocate this.
+	f, err := openTmpFile(filepath.Join(bucket, metaTmpDir), bucket, object, 0)
+	if err != nil {
+		return nil, fmt.Errorf("open temp file: %w", err)
+	}
+	defer f.cleanup()
+
+	for _, p := range parts {
+		pf, err := os.Open(filepath.Join(objdir, uploadID, fmt.Sprintf("%v", p.PartNumber)))
+		if err != nil {
+			return nil, fmt.Errorf("open part %v: %v", p.PartNumber, err)
+		}
+
+		// scoutfs move data is a metadata only operation that moves the data
+		// extent references from the source, appeding to the destination.
+		// this needs to be 4k aligned.
+		err = scoutfs.MoveData(pf, f.f)
+		pf.Close()
+		if err != nil {
+			return nil, fmt.Errorf("move blocks part %v: %v", p.PartNumber, err)
+		}
+	}
+
+	userMetaData := make(map[string]string)
+	upiddir := filepath.Join(objdir, uploadID)
+	loadUserMetaData(upiddir, userMetaData)
+
+	objname := filepath.Join(bucket, object)
+	dir := filepath.Dir(objname)
+	if dir != "" {
+		if err = mkdirAll(dir, os.FileMode(0755), bucket, object); err != nil {
+			if err != nil {
+				return nil, s3err.GetAPIError(s3err.ErrExistingObjectIsDirectory)
+			}
+		}
+	}
+	err = f.link()
+	if err != nil {
+		return nil, fmt.Errorf("link object in namespace: %w", err)
+	}
+
+	for k, v := range userMetaData {
+		err = xattr.Set(objname, "user."+k, []byte(v))
+		if err != nil {
+			// cleanup object if returning error
+			os.Remove(objname)
+			return nil, fmt.Errorf("set user attr %q: %w", k, err)
+		}
+	}
+
+	// Calculate s3 compatible md5sum for complete multipart.
+	s3MD5 := backend.GetMultipartMD5(parts)
+
+	err = xattr.Set(objname, "user.etag", []byte(s3MD5))
+	if err != nil {
+		// cleanup object if returning error
+		os.Remove(objname)
+		return nil, fmt.Errorf("set etag attr: %w", err)
+	}
+
+	// cleanup tmp dirs
+	os.RemoveAll(upiddir)
+	// use Remove for objdir in case there are still other uploads
+	// for same object name outstanding
+	os.Remove(objdir)
+
+	return &s3.CompleteMultipartUploadOutput{
+		Bucket: &bucket,
+		ETag:   &s3MD5,
+		Key:    &object,
+	}, nil
+}
+
+func (s *ScoutFS) checkUploadIDExists(bucket, object, uploadID string) ([32]byte, error) {
+	sum := sha256.Sum256([]byte(object))
+	objdir := filepath.Join(bucket, metaTmpMultipartDir, fmt.Sprintf("%x", sum))
+
+	_, err := os.Stat(filepath.Join(objdir, uploadID))
+	if errors.Is(err, fs.ErrNotExist) {
+		return [32]byte{}, s3err.GetAPIError(s3err.ErrNoSuchUpload)
+	}
+	if err != nil {
+		return [32]byte{}, fmt.Errorf("stat upload: %w", err)
+	}
+	return sum, nil
+}
+
+func loadUserMetaData(path string, m map[string]string) (contentType, contentEncoding string) {
+	ents, err := xattr.List(path)
+	if err != nil || len(ents) == 0 {
+		return
+	}
+	for _, e := range ents {
+		if !isValidMeta(e) {
+			continue
+		}
+		b, err := xattr.Get(path, e)
+		if err == syscall.ENODATA {
+			m[strings.TrimPrefix(e, "user.")] = ""
+			continue
+		}
+		if err != nil {
+			continue
+		}
+		m[strings.TrimPrefix(e, "user.")] = string(b)
+	}
+
+	b, err := xattr.Get(path, "user.content-type")
+	contentType = string(b)
+	if err != nil {
+		contentType = ""
+	}
+	if contentType != "" {
+		m["content-type"] = contentType
+	}
+
+	b, err = xattr.Get(path, "user.content-encoding")
+	contentEncoding = string(b)
+	if err != nil {
+		contentEncoding = ""
+	}
+	if contentEncoding != "" {
+		m["content-encoding"] = contentEncoding
+	}
+
+	return
+}
+
+func isValidMeta(val string) bool {
+	if strings.HasPrefix(val, "user.X-Amz-Meta") {
+		return true
+	}
+	if strings.EqualFold(val, "user.Expires") {
+		return true
+	}
+	return false
+}
+
+// mkdirAll is similar to os.MkdirAll but it will return ErrObjectParentIsFile
+// when appropriate
+func mkdirAll(path string, perm os.FileMode, bucket, object string) error {
+	// Fast path: if we can tell whether path is a directory or file, stop with success or error.
+	dir, err := os.Stat(path)
+	if err == nil {
+		if dir.IsDir() {
+			return nil
+		}
+		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
+	}
+
+	// Slow path: make sure parent exists and then call Mkdir for path.
+	i := len(path)
+	for i > 0 && os.IsPathSeparator(path[i-1]) { // Skip trailing path separator.
+		i--
+	}
+
+	j := i
+	for j > 0 && !os.IsPathSeparator(path[j-1]) { // Scan backward over element.
+		j--
+	}
+
+	if j > 1 {
+		// Create parent.
+		err = mkdirAll(path[:j-1], perm, bucket, object)
+		if err != nil {
+			return err
+		}
+	}
+
+	// Parent now exists; invoke Mkdir and use its result.
+	err = os.Mkdir(path, perm)
+	if err != nil {
+		// Handle arguments like "foo/." by
+		// double-checking that directory doesn't exist.
+		dir, err1 := os.Lstat(path)
+		if err1 == nil && dir.IsDir() {
+			return nil
+		}
+		return s3err.GetAPIError(s3err.ErrObjectParentIsFile)
+	}
+	return nil
+}
+
+func (s *ScoutFS) HeadObject(bucket, object string) (*s3.HeadObjectOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	objPath := filepath.Join(bucket, object)
+	fi, err := os.Stat(objPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat object: %w", err)
+	}
+
+	userMetaData := make(map[string]string)
+	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
+
+	b, err := xattr.Get(objPath, etagkey)
+	etag := string(b)
+	if err != nil {
+		etag = ""
+	}
+
+	stclass := types.StorageClassStandard
+	requestOngoing := ""
+	if s.glaciermode {
+		requestOngoing = stageComplete
+
+		// Check if there are any offline exents associated with this file.
+		// If so, we will set storage class to glacier.
+		st, err := scoutfs.StatMore(objPath)
+		if errors.Is(err, fs.ErrNotExist) {
+			return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+		}
+		if err != nil {
+			return nil, fmt.Errorf("stat more: %w", err)
+		}
+		if st.Offline_blocks != 0 {
+			stclass = types.StorageClassGlacier
+			requestOngoing = stageNotInProgress
+
+			ok, err := isStaging(objPath)
+			if errors.Is(err, fs.ErrNotExist) {
+				return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+			}
+			if err != nil {
+				return nil, fmt.Errorf("check stage status: %w", err)
+			}
+			if ok {
+				requestOngoing = stageInProgress
+			}
+		}
+	}
+
+	return &s3.HeadObjectOutput{
+		ContentLength:   fi.Size(),
+		ContentType:     &contentType,
+		ContentEncoding: &contentEncoding,
+		ETag:            &etag,
+		LastModified:    backend.GetTimePtr(fi.ModTime()),
+		Metadata:        userMetaData,
+		StorageClass:    stclass,
+		Restore:         &requestOngoing,
+	}, nil
+}
+
+func (s *ScoutFS) GetObject(bucket, object, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	objPath := filepath.Join(bucket, object)
+	fi, err := os.Stat(objPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat object: %w", err)
+	}
+
+	startOffset, length, err := backend.ParseRange(fi, acceptRange)
+	if err != nil {
+		return nil, err
+	}
+
+	if startOffset+length > fi.Size() {
+		// TODO: is ErrInvalidRequest correct here?
+		return nil, s3err.GetAPIError(s3err.ErrInvalidRequest)
+	}
+
+	if s.glaciermode {
+		// Check if there are any offline exents associated with this file.
+		// If so, we will return the InvalidObjectState error.
+		st, err := scoutfs.StatMore(objPath)
+		if errors.Is(err, fs.ErrNotExist) {
+			return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+		}
+		if err != nil {
+			return nil, fmt.Errorf("stat more: %w", err)
+		}
+		if st.Offline_blocks != 0 {
+			return nil, s3err.GetAPIError(s3err.ErrInvalidObjectState)
+		}
+	}
+
+	f, err := os.Open(objPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("open object: %w", err)
+	}
+	defer f.Close()
+
+	rdr := io.NewSectionReader(f, startOffset, length)
+	_, err = io.Copy(writer, rdr)
+	if err != nil {
+		return nil, fmt.Errorf("copy data: %w", err)
+	}
+
+	userMetaData := make(map[string]string)
+
+	contentType, contentEncoding := loadUserMetaData(objPath, userMetaData)
+
+	b, err := xattr.Get(objPath, etagkey)
+	etag := string(b)
+	if err != nil {
+		etag = ""
+	}
+
+	tags, err := s.getXattrTags(bucket, object)
+	if err != nil {
+		return nil, fmt.Errorf("get object tags: %w", err)
+	}
+
+	return &s3.GetObjectOutput{
+		AcceptRanges:    &acceptRange,
+		ContentLength:   length,
+		ContentEncoding: &contentEncoding,
+		ContentType:     &contentType,
+		ETag:            &etag,
+		LastModified:    backend.GetTimePtr(fi.ModTime()),
+		Metadata:        userMetaData,
+		TagCount:        int32(len(tags)),
+		StorageClass:    types.StorageClassStandard,
+	}, nil
+}
+
+func (s *ScoutFS) getXattrTags(bucket, object string) (map[string]string, error) {
+	tags := make(map[string]string)
+	b, err := xattr.Get(filepath.Join(bucket, object), "user."+tagHdr)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if isNoAttr(err) {
+		return tags, nil
+	}
+	if err != nil {
+		return nil, fmt.Errorf("get tags: %w", err)
+	}
+
+	err = json.Unmarshal(b, &tags)
+	if err != nil {
+		return nil, fmt.Errorf("unmarshal tags: %w", err)
+	}
+
+	return tags, nil
+}
+
+func (s *ScoutFS) ListObjects(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	fileSystem := os.DirFS(bucket)
+	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
+		s.fileToObj(bucket), []string{metaTmpDir})
+	if err != nil {
+		return nil, fmt.Errorf("walk %v: %w", bucket, err)
+	}
+
+	return &s3.ListObjectsOutput{
+		CommonPrefixes: results.CommonPrefixes,
+		Contents:       results.Objects,
+		Delimiter:      &delim,
+		IsTruncated:    results.Truncated,
+		Marker:         &marker,
+		MaxKeys:        int32(maxkeys),
+		Name:           &bucket,
+		NextMarker:     &results.NextMarker,
+		Prefix:         &prefix,
+	}, nil
+}
+
+func (s *ScoutFS) ListObjectsV2(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("stat bucket: %w", err)
+	}
+
+	fileSystem := os.DirFS(bucket)
+	results, err := backend.Walk(fileSystem, prefix, delim, marker, maxkeys,
+		s.fileToObj(bucket), []string{metaTmpDir})
+	if err != nil {
+		return nil, fmt.Errorf("walk %v: %w", bucket, err)
+	}
+
+	return &s3.ListObjectsV2Output{
+		CommonPrefixes:        results.CommonPrefixes,
+		Contents:              results.Objects,
+		Delimiter:             &delim,
+		IsTruncated:           results.Truncated,
+		ContinuationToken:     &marker,
+		MaxKeys:               int32(maxkeys),
+		Name:                  &bucket,
+		NextContinuationToken: &results.NextMarker,
+		Prefix:                &prefix,
+	}, nil
+}
+
+func (s *ScoutFS) fileToObj(bucket string) backend.GetObjFunc {
+	return func(path string, d fs.DirEntry) (types.Object, error) {
+		objPath := filepath.Join(bucket, path)
+		if d.IsDir() {
+			// directory object only happens if directory empty
+			// check to see if this is a directory object by checking etag
+			etagBytes, err := xattr.Get(objPath, etagkey)
+			if isNoAttr(err) || errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get etag: %w", err)
+			}
+			etag := string(etagBytes)
+
+			fi, err := d.Info()
+			if errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+			}
+
+			key := path + "/"
+
+			return types.Object{
+				ETag:         &etag,
+				Key:          &key,
+				LastModified: backend.GetTimePtr(fi.ModTime()),
+			}, nil
+		}
+
+		// file object, get object info and fill out object data
+		etagBytes, err := xattr.Get(objPath, etagkey)
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil && !isNoAttr(err) {
+			return types.Object{}, fmt.Errorf("get etag: %w", err)
+		}
+		etag := string(etagBytes)
+
+		fi, err := d.Info()
+		if errors.Is(err, fs.ErrNotExist) {
+			return types.Object{}, backend.ErrSkipObj
+		}
+		if err != nil {
+			return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+		}
+
+		sc := types.ObjectStorageClassStandard
+		if s.glaciermode {
+			// Check if there are any offline exents associated with this file.
+			// If so, we will return the InvalidObjectState error.
+			st, err := scoutfs.StatMore(objPath)
+			if errors.Is(err, fs.ErrNotExist) {
+				return types.Object{}, backend.ErrSkipObj
+			}
+			if err != nil {
+				return types.Object{}, fmt.Errorf("stat more: %w", err)
+			}
+			if st.Offline_blocks != 0 {
+				sc = types.ObjectStorageClassGlacier
+			}
+		}
+
+		return types.Object{
+			ETag:         &etag,
+			Key:          &path,
+			LastModified: backend.GetTimePtr(fi.ModTime()),
+			Size:         fi.Size(),
+			StorageClass: sc,
+		}, nil
+	}
+}
+
+// RestoreObject will set stage request on file if offline and do nothing if
+// file is online
+func (s *ScoutFS) RestoreObject(bucket, object string, restoreRequest *s3.RestoreObjectInput) error {
+	_, err := os.Stat(bucket)
+	if errors.Is(err, fs.ErrNotExist) {
+		return s3err.GetAPIError(s3err.ErrNoSuchBucket)
+	}
+	if err != nil {
+		return fmt.Errorf("stat bucket: %w", err)
+	}
+
+	err = setStaging(filepath.Join(bucket, object))
+	if errors.Is(err, fs.ErrNotExist) {
+		return s3err.GetAPIError(s3err.ErrNoSuchKey)
+	}
+	if err != nil {
+		return fmt.Errorf("stage object: %w", err)
+	}
+
+	return nil
+}
+
+func setStaging(objname string) error {
+	b, err := xattr.Get(objname, flagskey)
+	if err != nil && !isNoAttr(err) {
+		return err
+	}
+
+	var oldflags uint64
+	if !isNoAttr(err) {
+		err = json.Unmarshal(b, &oldflags)
+		if err != nil {
+			return err
+		}
+	}
+
+	newflags := oldflags | Staging
+
+	if newflags == oldflags {
+		// no flags change, just return
+		return nil
+	}
+
+	return fSetNewGlobalFlags(objname, newflags)
+}
+
+func isStaging(objname string) (bool, error) {
+	b, err := xattr.Get(objname, flagskey)
+	if err != nil && !isNoAttr(err) {
+		return false, err
+	}
+
+	var flags uint64
+	if !isNoAttr(err) {
+		err = json.Unmarshal(b, &flags)
+		if err != nil {
+			return false, err
+		}
+	}
+
+	return flags&Staging == Staging, nil
+}
+
+func fSetNewGlobalFlags(objname string, flags uint64) error {
+	b, err := json.Marshal(&flags)
+	if err != nil {
+		return err
+	}
+
+	return xattr.Set(objname, flagskey, b)
+}
+
+func isNoAttr(err error) bool {
+	if err == nil {
+		return false
+	}
+	xerr, ok := err.(*xattr.Error)
+	if ok && xerr.Err == xattr.ENOATTR {
+		return true
+	}
+	if err == syscall.ENODATA {
+		return true
+	}
+	return false
+}

backend/scoutfs/scoutfs_darwin.go

@@ -0,0 +1,48 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scoutfs
+
+import (
+	"errors"
+	"fmt"
+	"os"
+)
+
+func New(rootdir string, opts ...Option) (*ScoutFS, error) {
+	return nil, fmt.Errorf("scoutfs only available on linux")
+}
+
+type tmpfile struct {
+	f *os.File
+}
+
+var (
+	errNotSupported = errors.New("not supported")
+)
+
+func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
+	return nil, errNotSupported
+}
+
+func (tmp *tmpfile) link() error {
+	return errNotSupported
+}
+
+func (tmp *tmpfile) Write(b []byte) (int, error) {
+	return 0, errNotSupported
+}
+
+func (tmp *tmpfile) cleanup() {
+}

backend/scoutfs/scoutfs_linux.go

@@ -0,0 +1,184 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scoutfs
+
+import (
+	"crypto/sha256"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strconv"
+	"syscall"
+
+	"golang.org/x/sys/unix"
+
+	"github.com/versity/versitygw/backend/posix"
+)
+
+func New(rootdir string, opts ...Option) (*ScoutFS, error) {
+	p, err := posix.New(rootdir)
+	if err != nil {
+		return nil, err
+	}
+
+	f, err := os.Open(rootdir)
+	if err != nil {
+		return nil, fmt.Errorf("open %v: %w", rootdir, err)
+	}
+
+	s := &ScoutFS{Posix: p, rootfd: f, rootdir: rootdir}
+	for _, opt := range opts {
+		opt(s)
+	}
+
+	return s, nil
+}
+
+const procfddir = "/proc/self/fd"
+
+type tmpfile struct {
+	f       *os.File
+	bucket  string
+	objname string
+	isOTmp  bool
+	size    int64
+}
+
+func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
+	// O_TMPFILE allows for a file handle to an unnamed file in the filesystem.
+	// This can help reduce contention within the namespace (parent directories),
+	// etc. And will auto cleanup the inode on close if we never link this
+	// file descriptor into the namespace.
+	// Not all filesystems support this, so fallback to CreateTemp for when
+	// this is not supported.
+	fd, err := unix.Open(dir, unix.O_RDWR|unix.O_TMPFILE|unix.O_CLOEXEC, 0666)
+	if err != nil {
+		// O_TMPFILE not supported, try fallback
+		err := os.MkdirAll(dir, 0700)
+		if err != nil {
+			return nil, fmt.Errorf("make temp dir: %w", err)
+		}
+		f, err := os.CreateTemp(dir,
+			fmt.Sprintf("%x.", sha256.Sum256([]byte(obj))))
+		if err != nil {
+			return nil, err
+		}
+		tmp := &tmpfile{f: f, bucket: bucket, objname: obj, size: size}
+		// falloc is best effort, its fine if this fails
+		if size > 0 {
+			tmp.falloc()
+		}
+		return tmp, nil
+	}
+
+	// for O_TMPFILE, filename is /proc/self/fd/<fd> to be used
+	// later to link file into namespace
+	f := os.NewFile(uintptr(fd), filepath.Join(procfddir, strconv.Itoa(fd)))
+
+	tmp := &tmpfile{f: f, bucket: bucket, objname: obj, isOTmp: true, size: size}
+	// falloc is best effort, its fine if this fails
+	if size > 0 {
+		tmp.falloc()
+	}
+	return tmp, nil
+}
+
+func (tmp *tmpfile) falloc() error {
+	err := syscall.Fallocate(int(tmp.f.Fd()), 0, 0, tmp.size)
+	if err != nil {
+		return fmt.Errorf("fallocate: %v", err)
+	}
+	return nil
+}
+
+func (tmp *tmpfile) link() error {
+	// We use Linkat/Rename as the atomic operation for object puts. The
+	// upload is written to a temp (or unnamed/O_TMPFILE) file to not conflict
+	// with any other simultaneous uploads. The final operation is to move the
+	// temp file into place for the object. This ensures the object semantics
+	// of last upload completed wins and is not some combination of writes
+	// from simultaneous uploads.
+	objPath := filepath.Join(tmp.bucket, tmp.objname)
+	err := os.Remove(objPath)
+	if err != nil && !errors.Is(err, fs.ErrNotExist) {
+		return fmt.Errorf("remove stale path: %w", err)
+	}
+
+	if !tmp.isOTmp {
+		// O_TMPFILE not suported, use fallback
+		return tmp.fallbackLink()
+	}
+
+	procdir, err := os.Open(procfddir)
+	if err != nil {
+		return fmt.Errorf("open proc dir: %w", err)
+	}
+	defer procdir.Close()
+
+	dir, err := os.Open(filepath.Dir(objPath))
+	if err != nil {
+		return fmt.Errorf("open parent dir: %w", err)
+	}
+	defer dir.Close()
+
+	err = unix.Linkat(int(procdir.Fd()), filepath.Base(tmp.f.Name()),
+		int(dir.Fd()), filepath.Base(objPath), unix.AT_SYMLINK_FOLLOW)
+	if err != nil {
+		return fmt.Errorf("link tmpfile: %w", err)
+	}
+
+	err = tmp.f.Close()
+	if err != nil {
+		return fmt.Errorf("close tmpfile: %w", err)
+	}
+
+	return nil
+}
+
+func (tmp *tmpfile) fallbackLink() error {
+	tempname := tmp.f.Name()
+	// cleanup in case anything goes wrong, if rename succeeds then
+	// this will no longer exist
+	defer os.Remove(tempname)
+
+	err := tmp.f.Close()
+	if err != nil {
+		return fmt.Errorf("close tmpfile: %w", err)
+	}
+
+	objPath := filepath.Join(tmp.bucket, tmp.objname)
+	err = os.Rename(tempname, objPath)
+	if err != nil {
+		return fmt.Errorf("rename tmpfile: %w", err)
+	}
+
+	return nil
+}
+
+func (tmp *tmpfile) Write(b []byte) (int, error) {
+	if int64(len(b)) > tmp.size {
+		return 0, fmt.Errorf("write exceeds content length %v", tmp.size)
+	}
+
+	n, err := tmp.f.Write(b)
+	tmp.size -= int64(n)
+	return n, err
+}
+
+func (tmp *tmpfile) cleanup() {
+	tmp.f.Close()
+}

backend/scoutfs/scoutfs_windows.go

@@ -0,0 +1,48 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scoutfs
+
+import (
+	"errors"
+	"fmt"
+	"os"
+)
+
+func New(rootdir string, opts ...Option) (*ScoutFS, error) {
+	return nil, fmt.Errorf("scoutfs only available on linux")
+}
+
+type tmpfile struct {
+	f *os.File
+}
+
+var (
+	errNotSupported = errors.New("not supported")
+)
+
+func openTmpFile(dir, bucket, obj string, size int64) (*tmpfile, error) {
+	return nil, errNotSupported
+}
+
+func (tmp *tmpfile) link() error {
+	return errNotSupported
+}
+
+func (tmp *tmpfile) Write(b []byte) (int, error) {
+	return 0, errNotSupported
+}
+
+func (tmp *tmpfile) cleanup() {
+}

backend/walk.go

@@ -15,6 +15,7 @@
 package backend
 
 import (
+	"errors"
 	"fmt"
 	"io/fs"
 	"os"
@@ -31,9 +32,13 @@ type WalkResults struct {
 	NextMarker     string
 }
 
+type GetObjFunc func(path string, d fs.DirEntry) (types.Object, error)
+
+var ErrSkipObj = errors.New("skip this object")
+
 // Walk walks the supplied fs.FS and returns results compatible with list
 // objects responses
-func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResults, error) {
+func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int, getObj GetObjFunc, skipdirs []string) (WalkResults, error) {
 	cpmap := make(map[string]struct{})
 	var objects []types.Object
 
@@ -63,16 +68,40 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 				return nil
 			}
 
-			// If prefix is defined and the directory does not match prefix,
-			// do not descend into the directory because nothing will
-			// match this prefix.  Make sure to append the / at the end of
-			// directories since this is implied as a directory path name.
-			if prefix != "" && !strings.HasPrefix(path+string(os.PathSeparator), prefix) {
+			if contains(d.Name(), skipdirs) {
 				return fs.SkipDir
 			}
 
-			// TODO: special case handling if directory is empty
-			// and was "PUT" explicitly
+			// If prefix is defined and the directory does not match prefix,
+			// do not descend into the directory because nothing will
+			// match this prefix. Make sure to append the / at the end of
+			// directories since this is implied as a directory path name.
+			// If path is a prefix of prefix, then path could still be
+			// building to match. So only skip if path isnt a prefix of prefix
+			// and prefix isnt a prefix of path.
+			if prefix != "" &&
+				!strings.HasPrefix(path+string(os.PathSeparator), prefix) &&
+				!strings.HasPrefix(prefix, path+string(os.PathSeparator)) {
+				return fs.SkipDir
+			}
+
+			// TODO: can we do better here rather than a second readdir
+			// per directory?
+			ents, err := fs.ReadDir(fileSystem, path)
+			if err != nil {
+				return fmt.Errorf("readdir %q: %w", path, err)
+			}
+			if len(ents) == 0 {
+				dirobj, err := getObj(path, d)
+				if err == ErrSkipObj {
+					return nil
+				}
+				if err != nil {
+					return fmt.Errorf("directory to object %q: %w", path, err)
+				}
+				objects = append(objects, dirobj)
+			}
+
 			return nil
 		}
 
@@ -83,28 +112,27 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 			pastMarker = true
 		}
 
-		// If object doesnt have prefix, dont include in results.
+		// If object doesn't have prefix, don't include in results.
 		if prefix != "" && !strings.HasPrefix(path, prefix) {
 			return nil
 		}
 
 		if delimiter == "" {
-			// If no delimeter specified, then all files with matching
+			// If no delimiter specified, then all files with matching
 			// prefix are included in results
-			fi, err := d.Info()
-			if err != nil {
-				return fmt.Errorf("get info for %v: %w", path, err)
+			obj, err := getObj(path, d)
+			if err == ErrSkipObj {
+				return nil
 			}
+			if err != nil {
+				return fmt.Errorf("file to object %q: %w", path, err)
+			}
+			objects = append(objects, obj)
 
-			objects = append(objects, types.Object{
-				ETag:         new(string),
-				Key:          &path,
-				LastModified: GetTimePtr(fi.ModTime()),
-				Size:         fi.Size(),
-			})
-			if (len(objects) + len(cpmap)) == max {
+			if max > 0 && (len(objects)+len(cpmap)) == max {
 				pastMax = true
 			}
+
 			return nil
 		}
 
@@ -115,7 +143,7 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 		//
 		// For example:
 		// prefix = A/
-		// delimeter = /
+		// delimiter = /
 		// and objects:
 		// A/file
 		// A/B/file
@@ -124,24 +152,22 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 		// objects: A/file
 		// common prefix: A/B/
 		//
-		// Note: No obects are included past the common prefix since
+		// Note: No objects are included past the common prefix since
 		// these are all rolled up into the common prefix.
-		// Note: The delimeter can be anything, so we have to operate on
-		// the full path without any assumptions on posix directory heirarchy
-		// here.  Usually the delimeter with be "/", but thats not required.
+		// Note: The delimiter can be anything, so we have to operate on
+		// the full path without any assumptions on posix directory hierarchy
+		// here.  Usually the delimiter will be "/", but thats not required.
 		suffix := strings.TrimPrefix(path, prefix)
 		before, _, found := strings.Cut(suffix, delimiter)
 		if !found {
-			fi, err := d.Info()
-			if err != nil {
-				return fmt.Errorf("get info for %v: %w", path, err)
+			obj, err := getObj(path, d)
+			if err == ErrSkipObj {
+				return nil
 			}
-			objects = append(objects, types.Object{
-				ETag:         new(string),
-				Key:          &path,
-				LastModified: GetTimePtr(fi.ModTime()),
-				Size:         fi.Size(),
-			})
+			if err != nil {
+				return fmt.Errorf("file to object %q: %w", path, err)
+			}
+			objects = append(objects, obj)
 			if (len(objects) + len(cpmap)) == max {
 				pastMax = true
 			}
@@ -150,7 +176,7 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 
 		// Common prefixes are a set, so should not have duplicates.
 		// These are abstractly a "directory", so need to include the
-		// delimeter at the end.
+		// delimiter at the end.
 		cpmap[prefix+before+delimiter] = struct{}{}
 		if (len(objects) + len(cpmap)) == max {
 			pastMax = true
@@ -162,15 +188,16 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 		return WalkResults{}, err
 	}
 
-	commonPrefixStrings := make([]string, 0, len(cpmap))
+	var commonPrefixStrings []string
 	for k := range cpmap {
 		commonPrefixStrings = append(commonPrefixStrings, k)
 	}
 	sort.Strings(commonPrefixStrings)
 	commonPrefixes := make([]types.CommonPrefix, 0, len(commonPrefixStrings))
 	for _, cp := range commonPrefixStrings {
+		pfx := cp
 		commonPrefixes = append(commonPrefixes, types.CommonPrefix{
-			Prefix: &cp,
+			Prefix: &pfx,
 		})
 	}
 
@@ -181,3 +208,12 @@ func Walk(fileSystem fs.FS, prefix, delimiter, marker string, max int) (WalkResu
 		NextMarker:     newMarker,
 	}, nil
 }
+
+func contains(a string, strs []string) bool {
+	for _, s := range strs {
+		if s == a {
+			return true
+		}
+	}
+	return false
+}

backend/walk_test.go

@@ -15,6 +15,9 @@
 package backend_test
 
 import (
+	"crypto/md5"
+	"encoding/hex"
+	"fmt"
 	"io/fs"
 	"testing"
 	"testing/fstest"
@@ -26,31 +29,84 @@ import (
 type walkTest struct {
 	fsys     fs.FS
 	expected backend.WalkResults
+	getobj   backend.GetObjFunc
+}
+
+func getObj(path string, d fs.DirEntry) (types.Object, error) {
+	if d.IsDir() {
+		etag := getMD5(path)
+
+		fi, err := d.Info()
+		if err != nil {
+			return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+		}
+
+		return types.Object{
+			ETag:         &etag,
+			Key:          &path,
+			LastModified: backend.GetTimePtr(fi.ModTime()),
+		}, nil
+	}
+
+	etag := getMD5(path)
+
+	fi, err := d.Info()
+	if err != nil {
+		return types.Object{}, fmt.Errorf("get fileinfo: %w", err)
+	}
+
+	return types.Object{
+		ETag:         &etag,
+		Key:          &path,
+		LastModified: backend.GetTimePtr(fi.ModTime()),
+		Size:         fi.Size(),
+	}, nil
+}
+
+func getMD5(text string) string {
+	hash := md5.Sum([]byte(text))
+	return hex.EncodeToString(hash[:])
 }
 
 func TestWalk(t *testing.T) {
-	tests := []walkTest{{
-		// test case from
-		// https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html
-		fsys: fstest.MapFS{
-			"sample.jpg":                       {},
-			"photos/2006/January/sample.jpg":   {},
-			"photos/2006/February/sample2.jpg": {},
-			"photos/2006/February/sample3.jpg": {},
-			"photos/2006/February/sample4.jpg": {},
+	tests := []walkTest{
+		{
+			// test case from
+			// https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html
+			fsys: fstest.MapFS{
+				"sample.jpg":                       {},
+				"photos/2006/January/sample.jpg":   {},
+				"photos/2006/February/sample2.jpg": {},
+				"photos/2006/February/sample3.jpg": {},
+				"photos/2006/February/sample4.jpg": {},
+			},
+			expected: backend.WalkResults{
+				CommonPrefixes: []types.CommonPrefix{{
+					Prefix: backend.GetStringPtr("photos/"),
+				}},
+				Objects: []types.Object{{
+					Key: backend.GetStringPtr("sample.jpg"),
+				}},
+			},
+			getobj: getObj,
 		},
-		expected: backend.WalkResults{
-			CommonPrefixes: []types.CommonPrefix{{
-				Prefix: backend.GetStringPtr("photos/"),
-			}},
-			Objects: []types.Object{{
-				Key: backend.GetStringPtr("sample.jpg"),
-			}},
+		{
+			// test case single dir/single file
+			fsys: fstest.MapFS{
+				"test/file": {},
+			},
+			expected: backend.WalkResults{
+				CommonPrefixes: []types.CommonPrefix{{
+					Prefix: backend.GetStringPtr("test/"),
+				}},
+				Objects: []types.Object{},
+			},
+			getobj: getObj,
 		},
-	}}
+	}
 
 	for _, tt := range tests {
-		res, err := backend.Walk(tt.fsys, "", "/", "", 1000)
+		res, err := backend.Walk(tt.fsys, "", "/", "", 1000, tt.getobj, []string{})
 		if err != nil {
 			t.Fatalf("walk: %v", err)
 		}
@@ -67,13 +123,16 @@ func compareResults(got, wanted backend.WalkResults, t *testing.T) {
 	}
 
 	if !compareObjects(got.Objects, wanted.Objects) {
-		t.Errorf("unexpected common prefix, got %v wanted %v",
+		t.Errorf("unexpected object, got %v wanted %v",
 			printObjects(got.Objects),
 			printObjects(wanted.Objects))
 	}
 }
 
 func compareCommonPrefix(a, b []types.CommonPrefix) bool {
+	if len(a) == 0 && len(b) == 0 {
+		return true
+	}
 	if len(a) != len(b) {
 		return false
 	}
@@ -108,6 +167,9 @@ func printCommonPrefixes(list []types.CommonPrefix) string {
 }
 
 func compareObjects(a, b []types.Object) bool {
+	if len(a) == 0 && len(b) == 0 {
+		return true
+	}
 	if len(a) != len(b) {
 		return false
 	}

cmd/versitygw/admin.go

@@ -0,0 +1,198 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package main
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"time"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
+	"github.com/urfave/cli/v2"
+)
+
+var (
+	adminAccess string
+	adminSecret string
+	adminRegion string
+)
+
+func adminCommand() *cli.Command {
+	return &cli.Command{
+		Name:  "admin",
+		Usage: "admin CLI tool",
+		Description: `admin CLI tool for interacting with admin api.
+		Here is the available api list:
+		create-user
+		`,
+		Subcommands: []*cli.Command{
+			{
+				Name:   "create-user",
+				Usage:  "Create a new user",
+				Action: createUser,
+				Flags: []cli.Flag{
+					&cli.StringFlag{
+						Name:     "access",
+						Usage:    "access value for the new user",
+						Required: true,
+						Aliases:  []string{"a"},
+					},
+					&cli.StringFlag{
+						Name:     "secret",
+						Usage:    "secret value for the new user",
+						Required: true,
+						Aliases:  []string{"s"},
+					},
+					&cli.StringFlag{
+						Name:     "role",
+						Usage:    "role for the new user",
+						Required: true,
+						Aliases:  []string{"r"},
+					},
+					&cli.StringFlag{
+						Name:    "region",
+						Usage:   "s3 region string for the user",
+						Value:   "us-east-1",
+						Aliases: []string{"rg"},
+					},
+				},
+			},
+			{
+				Name:   "delete-user",
+				Usage:  "Delete a user",
+				Action: deleteUser,
+				Flags: []cli.Flag{
+					&cli.StringFlag{
+						Name:     "access",
+						Usage:    "access value for the user to be deleted",
+						Required: true,
+						Aliases:  []string{"a"},
+					},
+				},
+			},
+		},
+		Flags: []cli.Flag{
+			// TODO: create a configuration file for this
+			&cli.StringFlag{
+				Name:        "adminAccess",
+				Usage:       "admin access account",
+				EnvVars:     []string{"ADMIN_ACCESS_KEY_ID", "ADMIN_ACCESS_KEY"},
+				Aliases:     []string{"aa"},
+				Destination: &adminAccess,
+			},
+			&cli.StringFlag{
+				Name:        "adminSecret",
+				Usage:       "admin secret access key",
+				EnvVars:     []string{"ADMIN_SECRET_ACCESS_KEY", "ADMIN_SECRET_KEY"},
+				Aliases:     []string{"as"},
+				Destination: &adminSecret,
+			},
+			&cli.StringFlag{
+				Name:        "adminRegion",
+				Usage:       "s3 region string",
+				Value:       "us-east-1",
+				Destination: &adminRegion,
+				Aliases:     []string{"ar"},
+			},
+		},
+	}
+}
+
+func createUser(ctx *cli.Context) error {
+	access, secret, role, region := ctx.String("access"), ctx.String("secret"), ctx.String("role"), ctx.String("region")
+	if access == "" || secret == "" || region == "" {
+		return fmt.Errorf("invalid input parameters for the new user")
+	}
+	if role != "admin" && role != "user" {
+		return fmt.Errorf("invalid input parameter for role")
+	}
+
+	req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("http://localhost:7070/create-user?access=%v&secret=%v&role=%v&region=%v", access, secret, role, region), nil)
+	if err != nil {
+		return fmt.Errorf("failed to send the request: %w", err)
+	}
+
+	signer := v4.NewSigner()
+
+	hashedPayload := sha256.Sum256([]byte{})
+	hexPayload := hex.EncodeToString(hashedPayload[:])
+
+	req.Header.Set("X-Amz-Content-Sha256", hexPayload)
+
+	signErr := signer.SignHTTP(req.Context(), aws.Credentials{AccessKeyID: adminAccess, SecretAccessKey: adminSecret}, req, hexPayload, "s3", adminRegion, time.Now())
+	if signErr != nil {
+		return fmt.Errorf("failed to sign the request: %w", err)
+	}
+
+	client := http.Client{}
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return fmt.Errorf("failed to send the request: %w", err)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+
+	fmt.Printf("%s", body)
+
+	return nil
+}
+
+func deleteUser(ctx *cli.Context) error {
+	access := ctx.String("access")
+	if access == "" {
+		return fmt.Errorf("invalid input parameter for the new user")
+	}
+
+	req, err := http.NewRequest(http.MethodDelete, fmt.Sprintf("http://localhost:7070/delete-user?access=%v", access), nil)
+	if err != nil {
+		return fmt.Errorf("failed to send the request: %w", err)
+	}
+
+	signer := v4.NewSigner()
+
+	hashedPayload := sha256.Sum256([]byte{})
+	hexPayload := hex.EncodeToString(hashedPayload[:])
+
+	req.Header.Set("X-Amz-Content-Sha256", hexPayload)
+
+	signErr := signer.SignHTTP(req.Context(), aws.Credentials{AccessKeyID: adminAccess, SecretAccessKey: adminSecret}, req, hexPayload, "s3", adminRegion, time.Now())
+	if signErr != nil {
+		return fmt.Errorf("failed to sign the request: %w", err)
+	}
+
+	client := http.Client{}
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return fmt.Errorf("failed to send the request: %w", err)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+
+	fmt.Printf("%s", body)
+
+	return nil
+}

cmd/versitygw/main.go

@@ -22,16 +22,16 @@ import (
 
 	"github.com/gofiber/fiber/v2"
 	"github.com/urfave/cli/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api"
 	"github.com/versity/versitygw/s3api/middlewares"
 )
 
 var (
 	port              string
-	adminAccess       string
-	adminSecret       string
+	rootUserAccess    string
+	rootUserSecret    string
 	region            string
 	certFile, keyFile string
 	debug             bool
@@ -51,6 +51,9 @@ func main() {
 
 	app.Commands = []*cli.Command{
 		posixCommand(),
+		scoutfsCommand(),
+		adminCommand(),
+		testCommand(),
 	}
 
 	if err := app.Run(os.Args); err != nil {
@@ -94,21 +97,24 @@ func initFlags() []cli.Flag {
 		},
 		&cli.StringFlag{
 			Name:        "access",
-			Usage:       "admin access account",
-			Destination: &adminAccess,
-			EnvVars:     []string{"ADMIN_ACCESS_KEY_ID", "ADMIN_ACCESS_KEY"},
+			Usage:       "root user access key",
+			EnvVars:     []string{"ROOT_ACCESS_KEY_ID", "ROOT_ACCESS_KEY"},
+			Aliases:     []string{"a"},
+			Destination: &rootUserAccess,
 		},
 		&cli.StringFlag{
 			Name:        "secret",
-			Usage:       "admin secret access key",
-			Destination: &adminSecret,
-			EnvVars:     []string{"ADMIN_SECRET_ACCESS_KEY", "ADMIN_SECRET_KEY"},
+			Usage:       "root user secret access key",
+			EnvVars:     []string{"ROOT_SECRET_ACCESS_KEY", "ROOT_SECRET_KEY"},
+			Aliases:     []string{"s"},
+			Destination: &rootUserSecret,
 		},
 		&cli.StringFlag{
 			Name:        "region",
 			Usage:       "s3 region string",
 			Value:       "us-east-1",
 			Destination: &region,
+			Aliases:     []string{"r"},
 		},
 		&cli.StringFlag{
 			Name:        "cert",
@@ -128,10 +134,11 @@ func initFlags() []cli.Flag {
 	}
 }
 
-func runGateway(be backend.Backend) error {
+func runGateway(be backend.Backend, s auth.Storer) error {
 	app := fiber.New(fiber.Config{
 		AppName:      "versitygw",
 		ServerHeader: "VERSITYGW",
+		BodyLimit:    5 * 1024 * 1024 * 1024,
 	})
 
 	var opts []s3api.Option
@@ -155,12 +162,20 @@ func runGateway(be backend.Backend) error {
 		opts = append(opts, s3api.WithDebug())
 	}
 
-	srv, err := s3api.New(app, be, port,
-		middlewares.AdminConfig{
-			AdminAccess: adminAccess,
-			AdminSecret: adminSecret,
-			Region:      region,
-		}, auth.IAMServiceUnsupported{}, opts...)
+	err := s.InitIAM()
+	if err != nil {
+		return fmt.Errorf("init iam: %w", err)
+	}
+
+	iam, err := auth.NewInternal(s)
+	if err != nil {
+		return fmt.Errorf("setup internal iam service: %w", err)
+	}
+
+	srv, err := s3api.New(app, be, middlewares.RootUserConfig{
+		Access: rootUserAccess,
+		Secret: rootUserSecret,
+	}, port, region, iam, opts...)
 	if err != nil {
 		return fmt.Errorf("init gateway: %v", err)
 	}

cmd/versitygw/posix.go

@@ -49,5 +49,5 @@ func runPosix(ctx *cli.Context) error {
 		return fmt.Errorf("init posix: %v", err)
 	}
 
-	return runGateway(be)
+	return runGateway(be, be)
 }

cmd/versitygw/scoutfs.go

@@ -0,0 +1,73 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package main
+
+import (
+	"fmt"
+
+	"github.com/urfave/cli/v2"
+	"github.com/versity/versitygw/backend/scoutfs"
+)
+
+var (
+	glacier bool
+)
+
+func scoutfsCommand() *cli.Command {
+	return &cli.Command{
+		Name:  "scoutfs",
+		Usage: "scoutfs filesystem storage backend",
+		Description: `Support for ScoutFS.
+The top level directory for the gateway must be provided. All sub directories
+of the top level directory are treated as buckets, and all files/directories
+below the "bucket directory" are treated as the objects. The object name is
+split on "/" separator to translate to posix storage.
+For example:
+top level: /mnt/fs/gwroot
+bucket: mybucket
+object: a/b/c/myobject
+will be translated into the file /mnt/fs/gwroot/mybucket/a/b/c/myobject
+
+ScoutFS contains optimizations for multipart uploads using extent
+move interfaces as well as support for tiered filesystems.`,
+		Action: runScoutfs,
+		Flags: []cli.Flag{
+			&cli.BoolFlag{
+				Name:        "glacier",
+				Usage:       "enable glacier emulation mode",
+				Aliases:     []string{"g"},
+				Destination: &glacier,
+			},
+		},
+	}
+}
+
+func runScoutfs(ctx *cli.Context) error {
+	if ctx.NArg() == 0 {
+		return fmt.Errorf("no directory provided for operation")
+	}
+
+	var opts []scoutfs.Option
+	if glacier {
+		opts = append(opts, scoutfs.WithGlacierEmulation())
+	}
+
+	be, err := scoutfs.New(ctx.Args().Get(0), opts...)
+	if err != nil {
+		return fmt.Errorf("init scoutfs: %v", err)
+	}
+
+	return runGateway(be, be)
+}

cmd/versitygw/test.go

@@ -0,0 +1,180 @@
+package main
+
+import (
+	"fmt"
+
+	"github.com/urfave/cli/v2"
+	"github.com/versity/versitygw/integration"
+)
+
+var (
+	awsID     string
+	awsSecret string
+	endpoint  string
+)
+
+func testCommand() *cli.Command {
+	return &cli.Command{
+		Name:  "test",
+		Usage: "Client side testing command for the gateway",
+		Description: `The testing CLI is used to test group of versitygw actions.
+		It also includes some performance and stress testing`,
+		Subcommands: initTestCommands(),
+		Flags:       initTestFlags(),
+	}
+}
+
+func initTestFlags() []cli.Flag {
+	return []cli.Flag{
+		&cli.StringFlag{
+			Name:        "access",
+			Usage:       "aws user access key",
+			EnvVars:     []string{"AWS_ACCESS_KEY_ID", "AWS_ACCESS_KEY"},
+			Aliases:     []string{"a"},
+			Destination: &awsID,
+		},
+		&cli.StringFlag{
+			Name:        "secret",
+			Usage:       "aws user secret access key",
+			EnvVars:     []string{"AWS_SECRET_ACCESS_KEY", "AWS_SECRET_KEY"},
+			Aliases:     []string{"s"},
+			Destination: &awsSecret,
+		},
+		&cli.StringFlag{
+			Name:        "endpoint",
+			Usage:       "s3 server endpoint",
+			Destination: &endpoint,
+			Aliases:     []string{"e"},
+		},
+		&cli.BoolFlag{
+			Name:        "debug",
+			Usage:       "enable debug mode",
+			Aliases:     []string{"d"},
+			Destination: &debug,
+		},
+	}
+}
+
+func initTestCommands() []*cli.Command {
+	return []*cli.Command{
+		{
+			Name:  "make-bucket",
+			Usage: "Test bucket creation.",
+			Description: `Calls s3 gateway create-bucket action to create a new bucket,
+		then calls delete-bucket action to delete the bucket.`,
+			Action: getAction(integration.TestMakeBucket),
+		},
+		{
+			Name:  "put-get-object",
+			Usage: "Test put & get object.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it,
+			gets the object from the bucket, deletes both the object and bucket.`,
+			Action: getAction(integration.TestPutGetObject),
+		},
+		{
+			Name:  "put-get-mp-object",
+			Usage: "Test put & get multipart object.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it with multipart upload,
+			gets the object from the bucket, deletes both the object and bucket.`,
+			Action: getAction(integration.TestPutGetMPObject),
+		},
+		{
+			Name:  "put-dir-object",
+			Usage: "Test put directory object.",
+			Description: `Creates a bucket with s3 gateway action, puts a directory object in it,
+			lists the bucket's objects, deletes both the objects and bucket.`,
+			Action: getAction(integration.TestPutDirObject),
+		},
+		{
+			Name:  "list-objects",
+			Usage: "Test list-objects action.",
+			Description: `Creates a bucket with s3 gateway action, puts 2 directory objects in it,
+			lists the bucket's objects, deletes both the objects and bucket.`,
+			Action: getAction(integration.TestListObject),
+		},
+		{
+			Name:  "abort-mp",
+			Usage: "Tests abort-multipart-upload action.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			lists the multipart upload, aborts the multipart upload, lists the multipart upload again,
+			deletes both the objects and bucket.`,
+			Action: getAction(integration.TestListAbortMultiPartObject),
+		},
+		{
+			Name:  "list-parts",
+			Usage: "Tests list-parts action.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			lists the upload parts, deletes both the objects and bucket.`,
+			Action: getAction(integration.TestListMultiParts),
+		},
+		{
+			Name:  "incorrect-mp",
+			Usage: "Tests incorrect multipart case.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			uploads different parts, completes the multipart upload with incorrect part numbers,
+			calls the head-object action, compares the content length, removes both the object and bucket`,
+			Action: getAction(integration.TestIncorrectMultiParts),
+		},
+		{
+			Name:  "incomplete-mp",
+			Usage: "Tests incomplete multi parts.",
+			Description: `Creates a bucket with s3 gateway action, creates a multipart upload,
+			upload a part, lists the parts, checks if the uploaded part is in the list, 
+			removes both the object and the bucket`,
+			Action: getAction(integration.TestIncompleteMultiParts),
+		},
+		{
+			Name:  "incomplete-put-object",
+			Usage: "Tests incomplete put objects case.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it,
+			gets the object with head-object action, expects the object to be got, 
+			removes both the object and bucket`,
+			Action: getAction(integration.TestIncompletePutObject),
+		},
+		{
+			Name:  "get-range",
+			Usage: "Tests get object by range.",
+			Description: `Creates a bucket with s3 gateway action, puts an object in it,
+			gets the object by specifying the object range, compares the range with the original one,
+			removes both the object and the bucket`,
+			Action: getAction(integration.TestRangeGet),
+		},
+		{
+			Name:  "invalid-mp",
+			Usage: "Tests invalid multi part case.",
+			Description: `Creates a bucket with s3 gateway action, creates a multi part upload,
+			uploads an invalid part, gets the object with head-object action, expects to get error,
+			removes both the object and bucket`,
+			Action: getAction(integration.TestInvalidMultiParts),
+		},
+		{
+			Name:        "full-flow",
+			Usage:       "Tests the full flow of gateway.",
+			Description: `Runs all the available tests to test the full flow of the gateway.`,
+			Action:      getAction(integration.TestFullFlow),
+		},
+	}
+}
+
+type testFunc func(*integration.S3Conf)
+
+func getAction(tf testFunc) func(*cli.Context) error {
+	return func(ctx *cli.Context) error {
+		opts := []integration.Option{
+			integration.WithAccess(awsID),
+			integration.WithSecret(awsSecret),
+			integration.WithRegion(region),
+			integration.WithEndpoint(endpoint),
+		}
+		if debug {
+			opts = append(opts, integration.WithDebug())
+		}
+
+		s := integration.NewS3Conf(opts...)
+		tf(s)
+
+		fmt.Println()
+		fmt.Println("RAN:", integration.RunCount, "PASS:", integration.PassCount, "FAIL:", integration.FailCount)
+		return nil
+	}
+}

go.mod

@@ -3,29 +3,42 @@ module github.com/versity/versitygw
 go 1.20
 
 require (
-	github.com/aws/aws-sdk-go-v2 v1.18.0
-	github.com/aws/aws-sdk-go-v2/service/s3 v1.33.1
+	github.com/aws/aws-sdk-go-v2 v1.18.1
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0
 	github.com/aws/smithy-go v1.13.5
 	github.com/gofiber/fiber/v2 v2.46.0
 	github.com/google/uuid v1.3.0
 	github.com/pkg/xattr v0.4.9
-	github.com/urfave/cli/v2 v2.25.4
+	github.com/urfave/cli/v2 v2.25.6
 	github.com/valyala/fasthttp v1.47.0
-	golang.org/x/sys v0.8.0
+	github.com/versity/scoutfs-go v0.0.0-20230606232754-0474b14343b9
+	golang.org/x/sys v0.9.0
+)
+
+require (
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.19.2 // indirect
+	github.com/jmespath/go-jmespath v0.4.0 // indirect
 )
 
 require (
 	github.com/andybalholm/brotli v1.0.5 // indirect
 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25 // indirect
+	github.com/aws/aws-sdk-go-v2/config v1.18.27
+	github.com/aws/aws-sdk-go-v2/credentials v1.13.26
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26 // indirect
 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.28 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
-	github.com/klauspost/compress v1.16.5 // indirect
+	github.com/klauspost/compress v1.16.6 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.19 // indirect
 	github.com/mattn/go-runewidth v0.0.14 // indirect

go.sum

@@ -1,25 +1,43 @@
 github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
 github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
-github.com/aws/aws-sdk-go-v2 v1.18.0 h1:882kkTpSFhdgYRKVZ/VCgf7sd0ru57p2JCxz4/oN5RY=
-github.com/aws/aws-sdk-go-v2 v1.18.0/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
+github.com/aws/aws-sdk-go-v2 v1.18.1 h1:+tefE750oAb7ZQGzla6bLkOwfcQCEtC5y2RqoqCeqKo=
+github.com/aws/aws-sdk-go-v2 v1.18.1/go.mod h1:uzbQtefpm44goOPmdKyAlXSNcwlRgF3ePWVW6EtJvvw=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 h1:dK82zF6kkPeCo8J1e+tGx4JdvDIQzj7ygIoLg8WMuGs=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10/go.mod h1:VeTZetY5KRJLuD/7fkQXMU6Mw7H5m/KP2J5Iy9osMno=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33 h1:kG5eQilShqmJbv11XL1VpyDbaEJzWxd4zRiCG30GSn4=
-github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33/go.mod h1:7i0PF1ME/2eUPFcjkVIwq+DOygHEoK92t5cDqNgYbIw=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27 h1:vFQlirhuM8lLlpI7imKOMsjdQLuN9CPi+k44F/OFVsk=
-github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27/go.mod h1:UrHnn3QV/d0pBZ6QBAEQcqFLf8FAzLmoUfPVIueOvoM=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25 h1:AzwRi5OKKwo4QNqPf7TjeO+tK8AyOK3GVSwmRPo7/Cs=
-github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25/go.mod h1:SUbB4wcbSEyCvqBxv/O/IBf93RbEze7U7OnoTlpPB+g=
+github.com/aws/aws-sdk-go-v2/config v1.18.27 h1:Az9uLwmssTE6OGTpsFqOnaGpLnKDqNYOJzWuC6UAYzA=
+github.com/aws/aws-sdk-go-v2/config v1.18.27/go.mod h1:0My+YgmkGxeqjXZb5BYme5pc4drjTnM+x1GJ3zv42Nw=
+github.com/aws/aws-sdk-go-v2/credentials v1.13.26 h1:qmU+yhKmOCyujmuPY7tf5MxR/RKyZrOPO3V4DobiTUk=
+github.com/aws/aws-sdk-go-v2/credentials v1.13.26/go.mod h1:GoXt2YC8jHUBbA4jr+W3JiemnIbkXOfxSXcisUsZ3os=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 h1:LxK/bitrAr4lnh9LnIS6i7zWbCOdMsfzKFBI6LUCS0I=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4/go.mod h1:E1hLXN/BL2e6YizK1zFlYd8vsfi2GTjbjBazinMmeaM=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70 h1:4bh28MeeXoBFTjb0JjQ5sVatzlf5xA1DziV8mZed9v4=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70/go.mod h1:9yI5NXzqy2yOiMytv6QLZHvlyHLwYxO9iIq+bZIbrFg=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 h1:A5UqQEmPaCFpedKouS4v+dHCTUo2sKqhoKO9U5kxyWo=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34/go.mod h1:wZpTEecJe0Btj3IYnDx/VlUzor9wm3fJHyvLpQF0VwY=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 h1:srIVS45eQuewqz6fKKu6ZGXaq6FuFg5NzgQBAM6g8Y4=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28/go.mod h1:7VRpKQQedkfIEXb4k52I7swUnZP0wohVajJMRn3vsUw=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35 h1:LWA+3kDM8ly001vJ1X1waCuLJdtTl48gwkPKWy9sosI=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35/go.mod h1:0Eg1YjxE0Bhn56lx+SHJwCzhW+2JGtizsrx+lCqrfm0=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26 h1:wscW+pnn3J1OYnanMnza5ZVYXLX4cKk5rAvUAl4Qu+c=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.26/go.mod h1:MtYiox5gvyB+OyP0Mr0Sm/yzbEAIPL9eijj/ouHAPw0=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11 h1:y2+VQzC6Zh2ojtV2LoC0MNwHWc6qXv/j2vrQtlftkdA=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11/go.mod h1:iV4q2hsqtNECrfmlXyord9u4zyuFEJX9eLgLpSPzWA8=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.28 h1:vGWm5vTpMr39tEZfQeDiDAMgk+5qsnvRny3FjLpnH5w=
-github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.28/go.mod h1:spfrICMD6wCAhjhzHuy6DOZZ+LAIY10UxhUmLzpJTTs=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27 h1:0iKliEXAcCa2qVtRs7Ot5hItA2MsufrphbRFlz1Owxo=
-github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27/go.mod h1:EOwBD4J4S5qYszS5/3DpkejfuK+Z5/1uzICfPaZLtqw=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.2 h1:NbWkRxEEIRSCqxhsHQuMiTH7yo+JZW1gp8v3elSVMTQ=
-github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.2/go.mod h1:4tfW5l4IAB32VWCDEBxCRtR9T4BWy4I4kr1spr8NgZM=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.33.1 h1:O+9nAy9Bb6bJFTpeNFtd9UfHbgxO1o4ZDAM9rQp5NsY=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.33.1/go.mod h1:J9kLNzEiHSeGMyN7238EjJmBpCniVzFda75Gxl/NqB8=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29 h1:zZSLP3v3riMOP14H7b4XP0uyfREDQOYv2cqIrvTXDNQ=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.29/go.mod h1:z7EjRjVwZ6pWcWdI2H64dKttvzaP99jRIj5hphW0M5U=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 h1:bkRyG4a929RCnpVSTvLM2j/T4ls015ZhhYApbmYs15s=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28/go.mod h1:jj7znCIg05jXlaGBlFMGP8+7UN3VtCkRBG2spnmRQkU=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 h1:dBL3StFxHtpBzJJ/mNEsjXVgfO+7jR0dAIEwLqMapEA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3/go.mod h1:f1QyiAsvIv4B49DmCqrhlXqyaR+0IxMmyX+1P+AnzOM=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.34.1 h1:rYYwwsGqbwvGgQHjBkqgDt8MynXk+I8xgS0IEj5gOT0=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.34.1/go.mod h1:aVbf0sko/TsLWHx30c/uVu7c62+0EAJ3vbxaJga0xCw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0 h1:ya7fmrN2fE7s1P2gaPbNg5MTkERVWfsH8ToP1YC4Z9o=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0/go.mod h1:aVbf0sko/TsLWHx30c/uVu7c62+0EAJ3vbxaJga0xCw=
+github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 h1:nneMBM2p79PGWBQovYO/6Xnc2ryRMw3InnDJq1FHkSY=
+github.com/aws/aws-sdk-go-v2/service/sso v1.12.12/go.mod h1:HuCOxYsF21eKrerARYO6HapNeh9GBNq7fius2AcwodY=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12 h1:2qTR7IFk7/0IN/adSFhYu9Xthr0zVFTgBrmPldILn80=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12/go.mod h1:E4VrHCPzmVB/KFXtqBGKb3c8zpbNBgKe3fisDNLAW5w=
+github.com/aws/aws-sdk-go-v2/service/sts v1.19.2 h1:XFJ2Z6sNUUcAz9poj+245DMkrHE4h2j5I9/xD50RHfE=
+github.com/aws/aws-sdk-go-v2/service/sts v1.19.2/go.mod h1:dp0yLPsLBOi++WTxzCjA/oZqi6NPIhoR+uF7GeMU9eg=
 github.com/aws/smithy-go v1.13.5 h1:hgz0X/DX0dGqTYpGALqXJoRKRj5oQ7150i5FdTePzO8=
 github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
 github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
@@ -31,10 +49,11 @@ github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
 github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
 github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
 github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
 github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
-github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI=
-github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
+github.com/klauspost/compress v1.16.6 h1:91SKEy4K37vkp255cJ8QesJhjyRO0hn9i9G0GoUwLsk=
+github.com/klauspost/compress v1.16.6/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
 github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
 github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
 github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
@@ -62,14 +81,16 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/tinylib/msgp v1.1.6/go.mod h1:75BAfg2hauQhs3qedfdDZmWAPcFMAvJE5b9rGOMufyw=
 github.com/tinylib/msgp v1.1.8 h1:FCXC1xanKO4I8plpHGH2P7koL/RzZs12l/+r7vakfm0=
 github.com/tinylib/msgp v1.1.8/go.mod h1:qkpG+2ldGg4xRFmx+jfTvZPxfGFhi64BcnL9vkCm/Tw=
-github.com/urfave/cli/v2 v2.25.4 h1:HyYwPrTO3im9rYhUff/ZNs78eolxt0nJ4LN+9yJKSH4=
-github.com/urfave/cli/v2 v2.25.4/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
+github.com/urfave/cli/v2 v2.25.6 h1:yuSkgDSZfH3L1CjF2/5fNNg2KbM47pY2EvjBq4ESQnU=
+github.com/urfave/cli/v2 v2.25.6/go.mod h1:8qnjx1vcq5s2/wpsqoZFndg2CE5tNFyrTvS6SinrnYQ=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/fasthttp v1.47.0 h1:y7moDoxYzMooFpT5aHgNgVOQDrS3qlkfiP9mDtGGK9c=
 github.com/valyala/fasthttp v1.47.0/go.mod h1:k2zXd82h/7UZc3VOdJ2WaUqt1uZ/XpXAfE9i+HBC3lA=
 github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8=
 github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
+github.com/versity/scoutfs-go v0.0.0-20230606232754-0474b14343b9 h1:ZfmQR01Kk6/kQh6+zlqfBYszVY02fzf9xYrchOY4NFM=
+github.com/versity/scoutfs-go v0.0.0-20230606232754-0474b14343b9/go.mod h1:gJsq73k+4685y+rbDIpPY8i/5GbsiwP6JFoFyUDB1fQ=
 github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU=
 github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673/go.mod h1:N3UwUGtsrSj3ccvlPHLoLsHnpR27oXr4ZE984MbSER8=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -102,8 +123,8 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
-golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
+golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=

integration/output.go

@@ -0,0 +1,31 @@
+package integration
+
+import "fmt"
+
+var (
+	colorReset = "\033[0m"
+	colorRed   = "\033[31m"
+	colorGreen = "\033[32m"
+	colorCyan  = "\033[36m"
+)
+
+var (
+	RunCount  = 0
+	PassCount = 0
+	FailCount = 0
+)
+
+func runF(format string, a ...interface{}) {
+	RunCount++
+	fmt.Printf(colorCyan+"RUN  "+colorReset+format+"\n", a...)
+}
+
+func failF(format string, a ...interface{}) {
+	FailCount++
+	fmt.Printf(colorRed+"FAIL "+colorReset+format+"\n", a...)
+}
+
+func passF(format string, a ...interface{}) {
+	PassCount++
+	fmt.Printf(colorGreen+"PASS "+colorReset+format+"\n", a...)
+}

integration/reader.go

@@ -0,0 +1,54 @@
+package integration
+
+import (
+	"crypto/rand"
+	"crypto/sha256"
+	"hash"
+	"io"
+)
+
+type RReader struct {
+	buf      []byte
+	dataleft int
+	hash     hash.Hash
+}
+
+func NewDataReader(totalsize, bufsize int) *RReader {
+	b := make([]byte, bufsize)
+	rand.Read(b)
+	return &RReader{
+		buf:      b,
+		dataleft: totalsize,
+		hash:     sha256.New(),
+	}
+}
+
+func (r *RReader) Read(p []byte) (int, error) {
+	n := min(len(p), len(r.buf), r.dataleft)
+	r.dataleft -= n
+	err := error(nil)
+	if n == 0 {
+		err = io.EOF
+	}
+	r.hash.Write(r.buf[:n])
+	return copy(p, r.buf[:n]), err
+}
+
+func (r *RReader) Sum() []byte {
+	return r.hash.Sum(nil)
+}
+
+func min(values ...int) int {
+	if len(values) == 0 {
+		return 0
+	}
+
+	min := values[0]
+	for _, v := range values {
+		if v < min {
+			min = v
+		}
+	}
+
+	return min
+}

integration/s3conf.go

@@ -0,0 +1,125 @@
+package integration
+
+import (
+	"context"
+	"log"
+	"net/http"
+	"os"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
+	"github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/credentials"
+	"github.com/aws/smithy-go/middleware"
+)
+
+type S3Conf struct {
+	awsID           string
+	awsSecret       string
+	awsRegion       string
+	endpoint        string
+	checksumDisable bool
+	pathStyle       bool
+	PartSize        int64
+	Concurrency     int
+	debug           bool
+}
+
+func NewS3Conf(opts ...Option) *S3Conf {
+	s := &S3Conf{
+		PartSize:    64 * 1024 * 1024, // 64B default chunksize
+		Concurrency: 1,                // 1 default concurrency
+	}
+
+	for _, opt := range opts {
+		opt(s)
+	}
+	return s
+}
+
+type Option func(*S3Conf)
+
+func WithAccess(ak string) Option {
+	return func(s *S3Conf) { s.awsID = ak }
+}
+func WithSecret(sk string) Option {
+	return func(s *S3Conf) { s.awsSecret = sk }
+}
+func WithRegion(r string) Option {
+	return func(s *S3Conf) { s.awsRegion = r }
+}
+func WithEndpoint(e string) Option {
+	return func(s *S3Conf) { s.endpoint = e }
+}
+func WithDisableChecksum() Option {
+	return func(s *S3Conf) { s.checksumDisable = true }
+}
+func WithPathStyle() Option {
+	return func(s *S3Conf) { s.pathStyle = true }
+}
+func WithPartSize(p int64) Option {
+	return func(s *S3Conf) { s.PartSize = p }
+}
+func WithConcurrency(c int) Option {
+	return func(s *S3Conf) { s.Concurrency = c }
+}
+func WithDebug() Option {
+	return func(s *S3Conf) { s.debug = true }
+}
+
+func (c *S3Conf) getCreds() credentials.StaticCredentialsProvider {
+	// TODO support token/IAM
+	if c.awsSecret == "" {
+		c.awsSecret = os.Getenv("AWS_SECRET_ACCESS_KEY")
+	}
+	if c.awsSecret == "" {
+		log.Fatal("no AWS_SECRET_ACCESS_KEY found")
+	}
+
+	return credentials.NewStaticCredentialsProvider(c.awsID, c.awsSecret, "")
+}
+
+func (c *S3Conf) ResolveEndpoint(service, region string, options ...interface{}) (aws.Endpoint, error) {
+	return aws.Endpoint{
+		PartitionID:       "aws",
+		URL:               c.endpoint,
+		SigningRegion:     c.awsRegion,
+		HostnameImmutable: true,
+	}, nil
+}
+
+func (c *S3Conf) Config() aws.Config {
+	creds := c.getCreds()
+
+	tr := &http.Transport{}
+	client := &http.Client{Transport: tr}
+
+	opts := []func(*config.LoadOptions) error{
+		config.WithRegion(c.awsRegion),
+		config.WithCredentialsProvider(creds),
+		config.WithHTTPClient(client),
+	}
+
+	if c.endpoint != "" && c.endpoint != "aws" {
+		opts = append(opts,
+			config.WithEndpointResolverWithOptions(c))
+	}
+
+	if c.checksumDisable {
+		opts = append(opts,
+			config.WithAPIOptions([]func(*middleware.Stack) error{v4.SwapComputePayloadSHA256ForUnsignedPayloadMiddleware}))
+	}
+
+	if c.debug {
+		opts = append(opts,
+			config.WithClientLogMode(aws.LogSigning|aws.LogRetries|aws.LogRequest|aws.LogResponse|aws.LogRequestEventMessage|aws.LogResponseEventMessage))
+	}
+
+	cfg, err := config.LoadDefaultConfig(
+		context.TODO(), opts...)
+	if err != nil {
+		log.Fatalln("error:", err)
+	}
+
+	return cfg
+}

s3api/controllers/admin.go

@@ -0,0 +1,61 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package controllers
+
+import (
+	"fmt"
+
+	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
+)
+
+type AdminController struct {
+	IAMService auth.IAMService
+}
+
+func (c AdminController) CreateUser(ctx *fiber.Ctx) error {
+	access, secret, role := ctx.Query("access"), ctx.Query("secret"), ctx.Query("role")
+	requesterRole := ctx.Locals("role")
+
+	if requesterRole != "admin" {
+		return fmt.Errorf("access denied: only admin users have access to this resource")
+	}
+
+	user := auth.Account{Secret: secret, Role: role}
+
+	err := c.IAMService.CreateAccount(access, user)
+	if err != nil {
+		return fmt.Errorf("failed to create a user: %w", err)
+	}
+
+	ctx.SendString("The user has been created successfully")
+	return nil
+}
+
+func (c AdminController) DeleteUser(ctx *fiber.Ctx) error {
+	access := ctx.Query("access")
+	requesterRole := ctx.Locals("role")
+	if requesterRole != "admin" {
+		return fmt.Errorf("access denied: only admin users have access to this resource")
+	}
+
+	err := c.IAMService.DeleteUserAccount(access)
+	if err != nil {
+		return err
+	}
+
+	ctx.SendString("The user has been created successfully")
+	return nil
+}

s3api/controllers/backend_moq_test.go

@@ -7,6 +7,7 @@ import (
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/versity/versitygw/backend"
+	"github.com/versity/versitygw/s3response"
 	"io"
 	"sync"
 )
@@ -45,7 +46,7 @@ var _ backend.Backend = &BackendMock{}
 //			DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
 //				panic("mock out the DeleteObjects method")
 //			},
-//			GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
+//			GetBucketAclFunc: func(bucket string) ([]byte, error) {
 //				panic("mock out the GetBucketAcl method")
 //			},
 //			GetObjectFunc: func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
@@ -66,13 +67,13 @@ var _ backend.Backend = &BackendMock{}
 //			HeadObjectFunc: func(bucket string, object string) (*s3.HeadObjectOutput, error) {
 //				panic("mock out the HeadObject method")
 //			},
-//			ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
+//			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
 //				panic("mock out the ListBuckets method")
 //			},
-//			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
+//			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
 //				panic("mock out the ListMultipartUploads method")
 //			},
-//			ListObjectPartsFunc: func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
+//			ListObjectPartsFunc: func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
 //				panic("mock out the ListObjectParts method")
 //			},
 //			ListObjectsFunc: func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
@@ -81,10 +82,10 @@ var _ backend.Backend = &BackendMock{}
 //			ListObjectsV2Func: func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
 //				panic("mock out the ListObjectsV2 method")
 //			},
-//			PutBucketFunc: func(bucket string) error {
+//			PutBucketFunc: func(bucket string, owner string) error {
 //				panic("mock out the PutBucket method")
 //			},
-//			PutBucketAclFunc: func(putBucketAclInput *s3.PutBucketAclInput) error {
+//			PutBucketAclFunc: func(bucket string, data []byte) error {
 //				panic("mock out the PutBucketAcl method")
 //			},
 //			PutObjectFunc: func(putObjectInput *s3.PutObjectInput) (string, error) {
@@ -111,9 +112,6 @@ var _ backend.Backend = &BackendMock{}
 //			StringFunc: func() string {
 //				panic("mock out the String method")
 //			},
-//			UploadPartFunc: func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-//				panic("mock out the UploadPart method")
-//			},
 //			UploadPartCopyFunc: func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 //				panic("mock out the UploadPartCopy method")
 //			},
@@ -149,7 +147,7 @@ type BackendMock struct {
 	DeleteObjectsFunc func(bucket string, objects *s3.DeleteObjectsInput) error
 
 	// GetBucketAclFunc mocks the GetBucketAcl method.
-	GetBucketAclFunc func(bucket string) (*s3.GetBucketAclOutput, error)
+	GetBucketAclFunc func(bucket string) ([]byte, error)
 
 	// GetObjectFunc mocks the GetObject method.
 	GetObjectFunc func(bucket string, object string, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error)
@@ -170,13 +168,13 @@ type BackendMock struct {
 	HeadObjectFunc func(bucket string, object string) (*s3.HeadObjectOutput, error)
 
 	// ListBucketsFunc mocks the ListBuckets method.
-	ListBucketsFunc func() (*s3.ListBucketsOutput, error)
+	ListBucketsFunc func() (s3response.ListAllMyBucketsResult, error)
 
 	// ListMultipartUploadsFunc mocks the ListMultipartUploads method.
-	ListMultipartUploadsFunc func(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error)
+	ListMultipartUploadsFunc func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error)
 
 	// ListObjectPartsFunc mocks the ListObjectParts method.
-	ListObjectPartsFunc func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error)
+	ListObjectPartsFunc func(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error)
 
 	// ListObjectsFunc mocks the ListObjects method.
 	ListObjectsFunc func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsOutput, error)
@@ -185,10 +183,10 @@ type BackendMock struct {
 	ListObjectsV2Func func(bucket string, prefix string, marker string, delim string, maxkeys int) (*s3.ListObjectsV2Output, error)
 
 	// PutBucketFunc mocks the PutBucket method.
-	PutBucketFunc func(bucket string) error
+	PutBucketFunc func(bucket string, owner string) error
 
 	// PutBucketAclFunc mocks the PutBucketAcl method.
-	PutBucketAclFunc func(putBucketAclInput *s3.PutBucketAclInput) error
+	PutBucketAclFunc func(bucket string, data []byte) error
 
 	// PutObjectFunc mocks the PutObject method.
 	PutObjectFunc func(putObjectInput *s3.PutObjectInput) (string, error)
@@ -214,9 +212,6 @@ type BackendMock struct {
 	// StringFunc mocks the String method.
 	StringFunc func() string
 
-	// UploadPartFunc mocks the UploadPart method.
-	UploadPartFunc func(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error)
-
 	// UploadPartCopyFunc mocks the UploadPartCopy method.
 	UploadPartCopyFunc func(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)
 
@@ -390,11 +385,15 @@ type BackendMock struct {
 		PutBucket []struct {
 			// Bucket is the bucket argument value.
 			Bucket string
+			// Owner is the owner argument value.
+			Owner string
 		}
 		// PutBucketAcl holds details about calls to the PutBucketAcl method.
 		PutBucketAcl []struct {
-			// PutBucketAclInput is the putBucketAclInput argument value.
-			PutBucketAclInput *s3.PutBucketAclInput
+			// Bucket is the bucket argument value.
+			Bucket string
+			// Data is the data argument value.
+			Data []byte
 		}
 		// PutObject holds details about calls to the PutObject method.
 		PutObject []struct {
@@ -452,17 +451,6 @@ type BackendMock struct {
 		// String holds details about calls to the String method.
 		String []struct {
 		}
-		// UploadPart holds details about calls to the UploadPart method.
-		UploadPart []struct {
-			// Bucket is the bucket argument value.
-			Bucket string
-			// Object is the object argument value.
-			Object string
-			// UploadId is the uploadId argument value.
-			UploadId string
-			// Body is the Body argument value.
-			Body io.ReadSeeker
-		}
 		// UploadPartCopy holds details about calls to the UploadPartCopy method.
 		UploadPartCopy []struct {
 			// UploadPartCopyInput is the uploadPartCopyInput argument value.
@@ -499,7 +487,6 @@ type BackendMock struct {
 	lockSetTags                 sync.RWMutex
 	lockShutdown                sync.RWMutex
 	lockString                  sync.RWMutex
-	lockUploadPart              sync.RWMutex
 	lockUploadPartCopy          sync.RWMutex
 }
 
@@ -812,7 +799,7 @@ func (mock *BackendMock) DeleteObjectsCalls() []struct {
 }
 
 // GetBucketAcl calls GetBucketAclFunc.
-func (mock *BackendMock) GetBucketAcl(bucket string) (*s3.GetBucketAclOutput, error) {
+func (mock *BackendMock) GetBucketAcl(bucket string) ([]byte, error) {
 	if mock.GetBucketAclFunc == nil {
 		panic("BackendMock.GetBucketAclFunc: method is nil but Backend.GetBucketAcl was just called")
 	}
@@ -1068,7 +1055,7 @@ func (mock *BackendMock) HeadObjectCalls() []struct {
 }
 
 // ListBuckets calls ListBucketsFunc.
-func (mock *BackendMock) ListBuckets() (*s3.ListBucketsOutput, error) {
+func (mock *BackendMock) ListBuckets() (s3response.ListAllMyBucketsResult, error) {
 	if mock.ListBucketsFunc == nil {
 		panic("BackendMock.ListBucketsFunc: method is nil but Backend.ListBuckets was just called")
 	}
@@ -1095,7 +1082,7 @@ func (mock *BackendMock) ListBucketsCalls() []struct {
 }
 
 // ListMultipartUploads calls ListMultipartUploadsFunc.
-func (mock *BackendMock) ListMultipartUploads(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
+func (mock *BackendMock) ListMultipartUploads(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
 	if mock.ListMultipartUploadsFunc == nil {
 		panic("BackendMock.ListMultipartUploadsFunc: method is nil but Backend.ListMultipartUploads was just called")
 	}
@@ -1127,7 +1114,7 @@ func (mock *BackendMock) ListMultipartUploadsCalls() []struct {
 }
 
 // ListObjectParts calls ListObjectPartsFunc.
-func (mock *BackendMock) ListObjectParts(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
+func (mock *BackendMock) ListObjectParts(bucket string, object string, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
 	if mock.ListObjectPartsFunc == nil {
 		panic("BackendMock.ListObjectPartsFunc: method is nil but Backend.ListObjectParts was just called")
 	}
@@ -1271,19 +1258,21 @@ func (mock *BackendMock) ListObjectsV2Calls() []struct {
 }
 
 // PutBucket calls PutBucketFunc.
-func (mock *BackendMock) PutBucket(bucket string) error {
+func (mock *BackendMock) PutBucket(bucket string, owner string) error {
 	if mock.PutBucketFunc == nil {
 		panic("BackendMock.PutBucketFunc: method is nil but Backend.PutBucket was just called")
 	}
 	callInfo := struct {
 		Bucket string
+		Owner  string
 	}{
 		Bucket: bucket,
+		Owner:  owner,
 	}
 	mock.lockPutBucket.Lock()
 	mock.calls.PutBucket = append(mock.calls.PutBucket, callInfo)
 	mock.lockPutBucket.Unlock()
-	return mock.PutBucketFunc(bucket)
+	return mock.PutBucketFunc(bucket, owner)
 }
 
 // PutBucketCalls gets all the calls that were made to PutBucket.
@@ -1292,9 +1281,11 @@ func (mock *BackendMock) PutBucket(bucket string) error {
 //	len(mockedBackend.PutBucketCalls())
 func (mock *BackendMock) PutBucketCalls() []struct {
 	Bucket string
+	Owner  string
 } {
 	var calls []struct {
 		Bucket string
+		Owner  string
 	}
 	mock.lockPutBucket.RLock()
 	calls = mock.calls.PutBucket
@@ -1303,19 +1294,21 @@ func (mock *BackendMock) PutBucketCalls() []struct {
 }
 
 // PutBucketAcl calls PutBucketAclFunc.
-func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) error {
+func (mock *BackendMock) PutBucketAcl(bucket string, data []byte) error {
 	if mock.PutBucketAclFunc == nil {
 		panic("BackendMock.PutBucketAclFunc: method is nil but Backend.PutBucketAcl was just called")
 	}
 	callInfo := struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}{
-		PutBucketAclInput: putBucketAclInput,
+		Bucket: bucket,
+		Data:   data,
 	}
 	mock.lockPutBucketAcl.Lock()
 	mock.calls.PutBucketAcl = append(mock.calls.PutBucketAcl, callInfo)
 	mock.lockPutBucketAcl.Unlock()
-	return mock.PutBucketAclFunc(putBucketAclInput)
+	return mock.PutBucketAclFunc(bucket, data)
 }
 
 // PutBucketAclCalls gets all the calls that were made to PutBucketAcl.
@@ -1323,10 +1316,12 @@ func (mock *BackendMock) PutBucketAcl(putBucketAclInput *s3.PutBucketAclInput) e
 //
 //	len(mockedBackend.PutBucketAclCalls())
 func (mock *BackendMock) PutBucketAclCalls() []struct {
-	PutBucketAclInput *s3.PutBucketAclInput
+	Bucket string
+	Data   []byte
 } {
 	var calls []struct {
-		PutBucketAclInput *s3.PutBucketAclInput
+		Bucket string
+		Data   []byte
 	}
 	mock.lockPutBucketAcl.RLock()
 	calls = mock.calls.PutBucketAcl
@@ -1620,50 +1615,6 @@ func (mock *BackendMock) StringCalls() []struct {
 	return calls
 }
 
-// UploadPart calls UploadPartFunc.
-func (mock *BackendMock) UploadPart(bucket string, object string, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-	if mock.UploadPartFunc == nil {
-		panic("BackendMock.UploadPartFunc: method is nil but Backend.UploadPart was just called")
-	}
-	callInfo := struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}{
-		Bucket:   bucket,
-		Object:   object,
-		UploadId: uploadId,
-		Body:     Body,
-	}
-	mock.lockUploadPart.Lock()
-	mock.calls.UploadPart = append(mock.calls.UploadPart, callInfo)
-	mock.lockUploadPart.Unlock()
-	return mock.UploadPartFunc(bucket, object, uploadId, Body)
-}
-
-// UploadPartCalls gets all the calls that were made to UploadPart.
-// Check the length with:
-//
-//	len(mockedBackend.UploadPartCalls())
-func (mock *BackendMock) UploadPartCalls() []struct {
-	Bucket   string
-	Object   string
-	UploadId string
-	Body     io.ReadSeeker
-} {
-	var calls []struct {
-		Bucket   string
-		Object   string
-		UploadId string
-		Body     io.ReadSeeker
-	}
-	mock.lockUploadPart.RLock()
-	calls = mock.calls.UploadPart
-	mock.lockUploadPart.RUnlock()
-	return calls
-}
-
 // UploadPartCopy calls UploadPartCopyFunc.
 func (mock *BackendMock) UploadPartCopy(uploadPartCopyInput *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
 	if mock.UploadPartCopyFunc == nil {

s3api/controllers/base.go

@@ -20,22 +20,24 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"log"
 	"net/http"
 	"strconv"
 	"strings"
-	"time"
 
 	"github.com/aws/aws-sdk-go-v2/aws"
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3api/utils"
 	"github.com/versity/versitygw/s3err"
 )
 
 type S3ApiController struct {
-	be backend.Backend
+	be  backend.Backend
+	iam auth.IAMService
 }
 
 func New(be backend.Backend) S3ApiController {
@@ -43,77 +45,187 @@ func New(be backend.Backend) S3ApiController {
 }
 
 func (c S3ApiController) ListBuckets(ctx *fiber.Ctx) error {
+	access, isRoot := ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+	if err := auth.IsAdmin(access, isRoot); err != nil {
+		return SendXMLResponse(ctx, nil, err)
+	}
 	res, err := c.be.ListBuckets()
-	return Responce(ctx, res, err)
+	return SendXMLResponse(ctx, res, err)
 }
 
 func (c S3ApiController) GetActions(ctx *fiber.Ctx) error {
-	bucket, key, keyEnd, uploadId, maxPartsStr, partNumberMarkerStr, acceptRange := ctx.Params("bucket"), ctx.Params("key"), ctx.Params("*1"), ctx.Query("uploadId"), ctx.Query("max-parts"), ctx.Query("part-number-marker"), ctx.Get("Range")
+	bucket := ctx.Params("bucket")
+	key := ctx.Params("key")
+	keyEnd := ctx.Params("*1")
+	uploadId := ctx.Query("uploadId")
+	maxParts := ctx.QueryInt("max-parts", 0)
+	partNumberMarker := ctx.QueryInt("part-number-marker", 0)
+	acceptRange := ctx.Get("Range")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	if uploadId != "" {
-		maxParts, err := strconv.Atoi(maxPartsStr)
-		if err != nil && maxPartsStr != "" {
-			return errors.New("wrong api call")
+		if maxParts < 0 || (maxParts == 0 && ctx.Query("max-parts") != "") {
+			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidMaxParts))
+		}
+		if partNumberMarker < 0 || (partNumberMarker == 0 && ctx.Query("part-number-marker") != "") {
+			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidPartNumberMarker))
 		}
 
-		partNumberMarker, err := strconv.Atoi(partNumberMarkerStr)
-		if err != nil && partNumberMarkerStr != "" {
-			return errors.New("wrong api call")
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
 		}
 
-		res, err := c.be.ListObjectParts(bucket, "", uploadId, partNumberMarker, maxParts)
-		return Responce(ctx, res, err)
+		res, err := c.be.ListObjectParts(bucket, key, uploadId, partNumberMarker, maxParts)
+		return SendXMLResponse(ctx, res, err)
 	}
 
 	if ctx.Request().URI().QueryArgs().Has("acl") {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ_ACP", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.GetObjectAcl(bucket, key)
-		return Responce(ctx, res, err)
+		return SendXMLResponse(ctx, res, err)
 	}
 
 	if attrs := ctx.Get("X-Amz-Object-Attributes"); attrs != "" {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.GetObjectAttributes(bucket, key, strings.Split(attrs, ","))
-		return Responce(ctx, res, err)
+		return SendXMLResponse(ctx, res, err)
 	}
 
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ_ACP", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	ctx.Locals("logResBody", false)
 	res, err := c.be.GetObject(bucket, key, acceptRange, ctx.Response().BodyWriter())
 	if err != nil {
-		return Responce(ctx, res, err)
+		return SendResponse(ctx, err)
 	}
-	return nil
+	if res == nil {
+		return SendResponse(ctx, fmt.Errorf("get object nil response"))
+	}
+
+	utils.SetMetaHeaders(ctx, res.Metadata)
+	var lastmod string
+	if res.LastModified != nil {
+		lastmod = res.LastModified.Format(timefmt)
+	}
+	utils.SetResponseHeaders(ctx, []utils.CustomHeader{
+		{
+			Key:   "Content-Length",
+			Value: fmt.Sprint(res.ContentLength),
+		},
+		{
+			Key:   "Content-Type",
+			Value: getstring(res.ContentType),
+		},
+		{
+			Key:   "Content-Encoding",
+			Value: getstring(res.ContentEncoding),
+		},
+		{
+			Key:   "ETag",
+			Value: getstring(res.ETag),
+		},
+		{
+			Key:   "Last-Modified",
+			Value: lastmod,
+		},
+		{
+			Key:   "x-amz-storage-class",
+			Value: string(res.StorageClass),
+		},
+	})
+	return SendResponse(ctx, err)
+}
+
+func getstring(s *string) string {
+	if s == nil {
+		return ""
+	}
+	return *s
 }
 
 func (c S3ApiController) ListActions(ctx *fiber.Ctx) error {
+	bucket := ctx.Params("bucket")
+	prefix := ctx.Query("prefix")
+	marker := ctx.Query("continuation-token")
+	delimiter := ctx.Query("delimiter")
+	maxkeys := ctx.QueryInt("max-keys")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	if ctx.Request().URI().QueryArgs().Has("acl") {
-		res, err := c.be.GetBucketAcl(ctx.Params("bucket"))
-		return Responce(ctx, res, err)
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ_ACP", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+
+		res, err := auth.ParseACLOutput(data)
+		return SendXMLResponse(ctx, res, err)
 	}
 
 	if ctx.Request().URI().QueryArgs().Has("uploads") {
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 		res, err := c.be.ListMultipartUploads(&s3.ListMultipartUploadsInput{Bucket: aws.String(ctx.Params("bucket"))})
-		return Responce(ctx, res, err)
+		return SendXMLResponse(ctx, res, err)
 	}
 
 	if ctx.QueryInt("list-type") == 2 {
-		res, err := c.be.ListObjectsV2(ctx.Params("bucket"), "", "", "", 1)
-		return Responce(ctx, res, err)
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+		res, err := c.be.ListObjectsV2(bucket, prefix, marker, delimiter, maxkeys)
+		return SendXMLResponse(ctx, res, err)
 	}
 
-	res, err := c.be.ListObjects(ctx.Params("bucket"), "", "", "", 1)
-	return Responce(ctx, res, err)
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+		return SendXMLResponse(ctx, nil, err)
+	}
+
+	res, err := c.be.ListObjects(bucket, prefix, marker, delimiter, maxkeys)
+	return SendXMLResponse(ctx, res, err)
 }
 
 func (c S3ApiController) PutBucketActions(ctx *fiber.Ctx) error {
-	bucket, acl, grantFullControl, grantRead, grantReadACP, granWrite, grantWriteACP :=
+	bucket, acl, grantFullControl, grantRead, grantReadACP, granWrite, grantWriteACP, access, isRoot :=
 		ctx.Params("bucket"),
 		ctx.Get("X-Amz-Acl"),
 		ctx.Get("X-Amz-Grant-Full-Control"),
 		ctx.Get("X-Amz-Grant-Read"),
 		ctx.Get("X-Amz-Grant-Read-Acp"),
 		ctx.Get("X-Amz-Grant-Write"),
-		ctx.Get("X-Amz-Grant-Write-Acp")
+		ctx.Get("X-Amz-Grant-Write-Acp"),
+		ctx.Locals("access").(string),
+		ctx.Locals("isRoot").(bool)
 
 	grants := grantFullControl + grantRead + grantReadACP + granWrite + grantWriteACP
 
@@ -121,86 +233,116 @@ func (c S3ApiController) PutBucketActions(ctx *fiber.Ctx) error {
 		if grants != "" && acl != "" {
 			return errors.New("wrong api call")
 		}
-		err := c.be.PutBucketAcl(&s3.PutBucketAclInput{
-			Bucket:           &bucket,
-			ACL:              types.BucketCannedACL(acl),
-			GrantFullControl: &grantFullControl,
-			GrantRead:        &grantRead,
-			GrantReadACP:     &grantReadACP,
-			GrantWrite:       &granWrite,
-			GrantWriteACP:    &grantWriteACP,
-		})
 
-		return Responce[any](ctx, nil, err)
+		if acl != "" && acl != "private" && acl != "public-read" && acl != "public-read-write" {
+			return errors.New("wrong api call")
+		}
+
+		data, err := c.be.GetBucketAcl(bucket)
+		if err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		parsedAcl, err := auth.ParseACL(data)
+		if err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE_ACP", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		input := &s3.PutBucketAclInput{
+			Bucket:              &bucket,
+			ACL:                 types.BucketCannedACL(acl),
+			GrantFullControl:    &grantFullControl,
+			GrantRead:           &grantRead,
+			GrantReadACP:        &grantReadACP,
+			GrantWrite:          &granWrite,
+			GrantWriteACP:       &grantWriteACP,
+			AccessControlPolicy: &types.AccessControlPolicy{Owner: &types.Owner{ID: &access}},
+		}
+
+		err = auth.UpdateACL(input, parsedAcl, c.iam)
+		return SendResponse(ctx, err)
 	}
 
-	err := c.be.PutBucket(bucket)
-	return Responce[any](ctx, nil, err)
+	err := c.be.PutBucket(bucket, access)
+	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
-	dstBucket, dstKeyStart, dstKeyEnd, uploadId, partNumberStr := ctx.Params("bucket"), ctx.Params("key"), ctx.Params("*1"), ctx.Query("uploadId"), ctx.Query("partNumber")
-	copySource, copySrcIfMatch, copySrcIfNoneMatch,
-		copySrcModifSince, copySrcUnmodifSince, acl,
-		grantFullControl, grantRead, grantReadACP,
-		granWrite, grantWriteACP, contentLengthStr :=
-		// Copy source headers
-		ctx.Get("X-Amz-Copy-Source"),
-		ctx.Get("X-Amz-Copy-Source-If-Match"),
-		ctx.Get("X-Amz-Copy-Source-If-None-Match"),
-		ctx.Get("X-Amz-Copy-Source-If-Modified-Since"),
-		ctx.Get("X-Amz-Copy-Source-If-Unmodified-Since"),
-		// Permission headers
-		ctx.Get("X-Amz-Acl"),
-		ctx.Get("X-Amz-Grant-Full-Control"),
-		ctx.Get("X-Amz-Grant-Read"),
-		ctx.Get("X-Amz-Grant-Read-Acp"),
-		ctx.Get("X-Amz-Grant-Write"),
-		ctx.Get("X-Amz-Grant-Write-Acp"),
-		// Other headers
-		ctx.Get("Content-Length")
+	bucket := ctx.Params("bucket")
+	keyStart := ctx.Params("key")
+	keyEnd := ctx.Params("*1")
+	uploadId := ctx.Query("uploadId")
+	partNumberStr := ctx.Query("partNumber")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
+
+	// Copy source headers
+	copySource := ctx.Get("X-Amz-Copy-Source")
+	copySrcIfMatch := ctx.Get("X-Amz-Copy-Source-If-Match")
+	copySrcIfNoneMatch := ctx.Get("X-Amz-Copy-Source-If-None-Match")
+	copySrcModifSince := ctx.Get("X-Amz-Copy-Source-If-Modified-Since")
+	copySrcUnmodifSince := ctx.Get("X-Amz-Copy-Source-If-Unmodified-Since")
+
+	// Permission headers
+	acl := ctx.Get("X-Amz-Acl")
+	grantFullControl := ctx.Get("X-Amz-Grant-Full-Control")
+	grantRead := ctx.Get("X-Amz-Grant-Read")
+	grantReadACP := ctx.Get("X-Amz-Grant-Read-Acp")
+	granWrite := ctx.Get("X-Amz-Grant-Write")
+	grantWriteACP := ctx.Get("X-Amz-Grant-Write-Acp")
+
+	// Other headers
+	contentLengthStr := ctx.Get("Content-Length")
 
 	grants := grantFullControl + grantRead + grantReadACP + granWrite + grantWriteACP
 
-	if dstKeyEnd != "" {
-		dstKeyStart = strings.Join([]string{dstKeyStart, dstKeyEnd}, "/")
+	if keyEnd != "" {
+		keyStart = strings.Join([]string{keyStart, keyEnd}, "/")
+	}
+	path := ctx.Path()
+	if path[len(path)-1:] == "/" && keyStart[len(keyStart)-1:] != "/" {
+		keyStart = keyStart + "/"
 	}
 
-	if partNumberStr != "" {
-		copySrcModifSinceDate, err := time.Parse(time.RFC3339, copySrcModifSince)
-		if err != nil && copySrcModifSince != "" {
-			return errors.New("wrong api call")
-		}
-
-		copySrcUnmodifSinceDate, err := time.Parse(time.RFC3339, copySrcUnmodifSince)
-		if err != nil && copySrcUnmodifSince != "" {
-			return errors.New("wrong api call")
-		}
-
-		partNumber, err := strconv.ParseInt(partNumberStr, 10, 64)
+	var contentLength int64
+	if contentLengthStr != "" {
+		var err error
+		contentLength, err = strconv.ParseInt(contentLengthStr, 10, 64)
 		if err != nil {
-			return errors.New("wrong api call")
+			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidRequest))
 		}
-
-		res, err := c.be.UploadPartCopy(&s3.UploadPartCopyInput{
-			Bucket:                      &dstBucket,
-			Key:                         &dstKeyStart,
-			PartNumber:                  int32(partNumber),
-			UploadId:                    &uploadId,
-			CopySource:                  &copySource,
-			CopySourceIfMatch:           &copySrcIfMatch,
-			CopySourceIfNoneMatch:       &copySrcIfNoneMatch,
-			CopySourceIfModifiedSince:   &copySrcModifSinceDate,
-			CopySourceIfUnmodifiedSince: &copySrcUnmodifSinceDate,
-		})
-
-		return Responce(ctx, res, err)
 	}
 
-	if uploadId != "" {
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if uploadId != "" && partNumberStr != "" {
+		partNumber := ctx.QueryInt("partNumber", -1)
+		if partNumber < 1 {
+			return SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidPart))
+		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		body := io.ReadSeeker(bytes.NewReader([]byte(ctx.Body())))
-		res, err := c.be.UploadPart(dstBucket, dstKeyStart, uploadId, body)
-		return Responce(ctx, res, err)
+		ctx.Locals("logReqBody", false)
+		etag, err := c.be.PutObjectPart(bucket, keyStart, uploadId,
+			partNumber, contentLength, body)
+		ctx.Response().Header.Set("Etag", etag)
+		return SendResponse(ctx, err)
 	}
 
 	if grants != "" || acl != "" {
@@ -208,9 +350,13 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 			return errors.New("wrong api call")
 		}
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE_ACP", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		err := c.be.PutObjectAcl(&s3.PutObjectAclInput{
-			Bucket:           &dstBucket,
-			Key:              &dstKeyStart,
+			Bucket:           &bucket,
+			Key:              &keyStart,
 			ACL:              types.ObjectCannedACL(acl),
 			GrantFullControl: &grantFullControl,
 			GrantRead:        &grantRead,
@@ -218,59 +364,117 @@ func (c S3ApiController) PutActions(ctx *fiber.Ctx) error {
 			GrantWrite:       &granWrite,
 			GrantWriteACP:    &grantWriteACP,
 		})
-		return Responce[any](ctx, nil, err)
+		return SendResponse(ctx, err)
 	}
 
 	if copySource != "" {
+		_, _, _, _ = copySrcIfMatch, copySrcIfNoneMatch,
+			copySrcModifSince, copySrcUnmodifSince
 		copySourceSplit := strings.Split(copySource, "/")
 		srcBucket, srcObject := copySourceSplit[0], copySourceSplit[1:]
 
-		res, err := c.be.CopyObject(srcBucket, strings.Join(srcObject, "/"), dstBucket, dstKeyStart)
-		return Responce(ctx, res, err)
-	}
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
 
-	contentLength, err := strconv.ParseInt(contentLengthStr, 10, 64)
-	if err != nil {
-		return errors.New("wrong api call")
+		res, err := c.be.CopyObject(srcBucket, strings.Join(srcObject, "/"), bucket, keyStart)
+		return SendXMLResponse(ctx, res, err)
 	}
 
 	metadata := utils.GetUserMetaData(&ctx.Request().Header)
 
-	res, err := c.be.PutObject(&s3.PutObjectInput{
-		Bucket:        &dstBucket,
-		Key:           &dstKeyStart,
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	ctx.Locals("logReqBody", false)
+	etag, err := c.be.PutObject(&s3.PutObjectInput{
+		Bucket:        &bucket,
+		Key:           &keyStart,
 		ContentLength: contentLength,
 		Metadata:      metadata,
 		Body:          bytes.NewReader(ctx.Request().Body()),
 	})
-	return Responce(ctx, res, err)
+	ctx.Response().Header.Set("ETag", etag)
+	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) DeleteBucket(ctx *fiber.Ctx) error {
-	err := c.be.DeleteBucket(ctx.Params("bucket"))
-	return Responce[any](ctx, nil, err)
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	err = c.be.DeleteBucket(bucket)
+	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) DeleteObjects(ctx *fiber.Ctx) error {
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
 	var dObj types.Delete
+
 	if err := xml.Unmarshal(ctx.Body(), &dObj); err != nil {
 		return errors.New("wrong api call")
 	}
 
-	err := c.be.DeleteObjects(ctx.Params("bucket"), &s3.DeleteObjectsInput{Delete: &dObj})
-	return Responce[any](ctx, nil, err)
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	err = c.be.DeleteObjects(bucket, &s3.DeleteObjectsInput{Delete: &dObj})
+	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) DeleteActions(ctx *fiber.Ctx) error {
-	bucket, key, keyEnd, uploadId := ctx.Params("bucket"), ctx.Params("key"), ctx.Params("*1"), ctx.Query("uploadId")
+	bucket := ctx.Params("bucket")
+	key := ctx.Params("key")
+	keyEnd := ctx.Params("*1")
+	uploadId := ctx.Query("uploadId")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
 
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	if uploadId != "" {
 		expectedBucketOwner, requestPayer := ctx.Get("X-Amz-Expected-Bucket-Owner"), ctx.Get("X-Amz-Request-Payer")
 
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
 		err := c.be.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
 			UploadId:            &uploadId,
 			Bucket:              &bucket,
@@ -278,30 +482,78 @@ func (c S3ApiController) DeleteActions(ctx *fiber.Ctx) error {
 			ExpectedBucketOwner: &expectedBucketOwner,
 			RequestPayer:        types.RequestPayer(requestPayer),
 		})
-		return Responce[any](ctx, nil, err)
+		return SendResponse(ctx, err)
 	}
 
-	err := c.be.DeleteObject(bucket, key)
-	return Responce[any](ctx, nil, err)
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	err = c.be.DeleteObject(bucket, key)
+	return SendResponse(ctx, err)
 }
 
 func (c S3ApiController) HeadBucket(ctx *fiber.Ctx) error {
-	res, err := c.be.HeadBucket(ctx.Params("bucket"))
-	return Responce(ctx, res, err)
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	_, err = c.be.HeadBucket(bucket)
+	// TODO: set bucket response headers
+	return SendResponse(ctx, err)
 }
 
+const (
+	timefmt = "Mon, 02 Jan 2006 15:04:05 GMT"
+)
+
 func (c S3ApiController) HeadObject(ctx *fiber.Ctx) error {
-	bucket, key, keyEnd := ctx.Params("bucket"), ctx.Params("key"), ctx.Params("*1")
+	bucket, access, isRoot := ctx.Params("bucket"), ctx.Locals("access").(string), ctx.Locals("isRoot").(bool)
+	key := ctx.Params("key")
+	keyEnd := ctx.Params("*1")
 	if keyEnd != "" {
 		key = strings.Join([]string{key, keyEnd}, "/")
 	}
 
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "READ", isRoot); err != nil {
+		return SendResponse(ctx, err)
+	}
+
 	res, err := c.be.HeadObject(bucket, key)
 	if err != nil {
-		return ErrorResponse(ctx, err)
+		return SendResponse(ctx, err)
+	}
+	if res == nil {
+		return SendResponse(ctx, fmt.Errorf("head object nil response"))
 	}
 
 	utils.SetMetaHeaders(ctx, res.Metadata)
+	var lastmod string
+	if res.LastModified != nil {
+		lastmod = res.LastModified.Format(timefmt)
+	}
 	utils.SetResponseHeaders(ctx, []utils.CustomHeader{
 		{
 			Key:   "Content-Length",
@@ -309,84 +561,145 @@ func (c S3ApiController) HeadObject(ctx *fiber.Ctx) error {
 		},
 		{
 			Key:   "Content-Type",
-			Value: *res.ContentType,
+			Value: getstring(res.ContentType),
 		},
 		{
 			Key:   "Content-Encoding",
-			Value: *res.ContentEncoding,
+			Value: getstring(res.ContentEncoding),
 		},
 		{
 			Key:   "ETag",
-			Value: *res.ETag,
+			Value: getstring(res.ETag),
 		},
 		{
 			Key:   "Last-Modified",
-			Value: res.LastModified.Format("20060102T150405Z"),
+			Value: lastmod,
+		},
+		{
+			Key:   "x-amz-storage-class",
+			Value: string(res.StorageClass),
+		},
+		{
+			Key:   "x-amz-restore",
+			Value: getstring(res.Restore),
 		},
 	})
 
+	return SendResponse(ctx, nil)
+}
+
+func (c S3ApiController) CreateActions(ctx *fiber.Ctx) error {
+	bucket := ctx.Params("bucket")
+	key := ctx.Params("key")
+	keyEnd := ctx.Params("*1")
+	uploadId := ctx.Query("uploadId")
+	access := ctx.Locals("access").(string)
+	isRoot := ctx.Locals("isRoot").(bool)
+
+	if keyEnd != "" {
+		key = strings.Join([]string{key, keyEnd}, "/")
+	}
+
+	data, err := c.be.GetBucketAcl(bucket)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	parsedAcl, err := auth.ParseACL(data)
+	if err != nil {
+		return SendResponse(ctx, err)
+	}
+
+	var restoreRequest s3.RestoreObjectInput
+	if ctx.Request().URI().QueryArgs().Has("restore") {
+		xmlErr := xml.Unmarshal(ctx.Body(), &restoreRequest)
+		if xmlErr != nil {
+			return errors.New("wrong api call")
+		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendResponse(ctx, err)
+		}
+
+		err := c.be.RestoreObject(bucket, key, &restoreRequest)
+		return SendResponse(ctx, err)
+	}
+
+	if uploadId != "" {
+		data := struct {
+			Parts []types.Part `xml:"Part"`
+		}{}
+
+		if err := xml.Unmarshal(ctx.Body(), &data); err != nil {
+			return errors.New("wrong api call")
+		}
+
+		if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+			return SendXMLResponse(ctx, nil, err)
+		}
+
+		res, err := c.be.CompleteMultipartUpload(bucket, key, uploadId, data.Parts)
+		return SendXMLResponse(ctx, res, err)
+	}
+
+	if err := auth.VerifyACL(parsedAcl, bucket, access, "WRITE", isRoot); err != nil {
+		return SendXMLResponse(ctx, nil, err)
+	}
+
+	res, err := c.be.CreateMultipartUpload(&s3.CreateMultipartUploadInput{Bucket: &bucket, Key: &key})
+	return SendXMLResponse(ctx, res, err)
+}
+
+func SendResponse(ctx *fiber.Ctx, err error) error {
+	if err != nil {
+		serr, ok := err.(s3err.APIError)
+		if ok {
+			ctx.Status(serr.HTTPStatusCode)
+			return ctx.Send(s3err.GetAPIErrorResponse(serr, "", "", ""))
+		}
+
+		log.Printf("Internal Error, %v", err)
+		ctx.Status(http.StatusInternalServerError)
+		return ctx.Send(s3err.GetAPIErrorResponse(
+			s3err.GetAPIError(s3err.ErrInternalError), "", "", ""))
+	}
+
+	utils.LogCtxDetails(ctx, []byte{})
+
 	// https://github.com/gofiber/fiber/issues/2080
 	// ctx.SendStatus() sets incorrect content length on HEAD request
 	ctx.Status(http.StatusOK)
 	return nil
 }
 
-func (c S3ApiController) CreateActions(ctx *fiber.Ctx) error {
-	bucket, key, keyEnd, uploadId := ctx.Params("bucket"), ctx.Params("key"), ctx.Params("*1"), ctx.Query("uploadId")
-	var restoreRequest s3.RestoreObjectInput
-
-	if keyEnd != "" {
-		key = strings.Join([]string{key, keyEnd}, "/")
-	}
-
-	if ctx.Request().URI().QueryArgs().Has("restore") {
-		xmlErr := xml.Unmarshal(ctx.Body(), &restoreRequest)
-		if xmlErr != nil {
-			return errors.New("wrong api call")
-		}
-		err := c.be.RestoreObject(bucket, key, &restoreRequest)
-		return Responce[any](ctx, nil, err)
-	}
-
-	if uploadId != "" {
-		var parts []types.Part
-
-		if err := xml.Unmarshal(ctx.Body(), &parts); err != nil {
-			return errors.New("wrong api call")
-		}
-
-		res, err := c.be.CompleteMultipartUpload(bucket, "", uploadId, parts)
-		return Responce(ctx, res, err)
-	}
-	res, err := c.be.CreateMultipartUpload(&s3.CreateMultipartUploadInput{Bucket: &bucket, Key: &key})
-	return Responce(ctx, res, err)
-}
-
-func Responce[R comparable](ctx *fiber.Ctx, resp R, err error) error {
+func SendXMLResponse(ctx *fiber.Ctx, resp any, err error) error {
 	if err != nil {
 		serr, ok := err.(s3err.APIError)
 		if ok {
 			ctx.Status(serr.HTTPStatusCode)
 			return ctx.Send(s3err.GetAPIErrorResponse(serr, "", "", ""))
 		}
+
+		log.Printf("Internal Error, %v", err)
+		ctx.Status(http.StatusInternalServerError)
+
 		return ctx.Send(s3err.GetAPIErrorResponse(
 			s3err.GetAPIError(s3err.ErrInternalError), "", "", ""))
 	}
 
 	var b []byte
-	if b, err = xml.Marshal(resp); err != nil {
-		return err
+
+	if resp != nil {
+		if b, err = xml.Marshal(resp); err != nil {
+			return err
+		}
+
+		if len(b) > 0 {
+			ctx.Response().Header.SetContentType(fiber.MIMEApplicationXML)
+		}
 	}
 
+	utils.LogCtxDetails(ctx, b)
+
 	return ctx.Send(b)
 }
-
-func ErrorResponse(ctx *fiber.Ctx, err error) error {
-	serr, ok := err.(s3err.APIError)
-	if ok {
-		ctx.Status(serr.HTTPStatusCode)
-		return ctx.Send(s3err.GetAPIErrorResponse(serr, "", "", ""))
-	}
-	return ctx.Send(s3err.GetAPIErrorResponse(
-		s3err.GetAPIError(s3err.ErrInternalError), "", "", ""))
-}

s3api/controllers/base_test.go

@@ -15,6 +15,7 @@
 package controllers
 
 import (
+	"encoding/json"
 	"io"
 	"net/http"
 	"net/http/httptest"
@@ -26,11 +27,25 @@ import (
 	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 	"github.com/gofiber/fiber/v2"
-	"github.com/valyala/fasthttp"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3err"
+	"github.com/versity/versitygw/s3response"
 )
 
+var (
+	acl     auth.ACL
+	acldata []byte
+)
+
+func init() {
+	var err error
+	acldata, err = json.Marshal(acl)
+	if err != nil {
+		panic(err)
+	}
+}
+
 func TestNew(t *testing.T) {
 	type args struct {
 		be backend.Backend
@@ -64,58 +79,86 @@ func TestNew(t *testing.T) {
 
 func TestS3ApiController_ListBuckets(t *testing.T) {
 	type args struct {
-		ctx *fiber.Ctx
+		req *http.Request
 	}
 
 	app := fiber.New()
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{}, nil
+			},
+		},
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
+	app.Get("/", s3ApiController.ListBuckets)
+
+	// Error case
+	appErr := fiber.New()
+	s3ApiControllerErr := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListBucketsFunc: func() (s3response.ListAllMyBucketsResult, error) {
+				return s3response.ListAllMyBucketsResult{}, s3err.GetAPIError(s3err.ErrMethodNotAllowed)
+			},
+		},
+	}
+
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
+	appErr.Get("/", s3ApiControllerErr.ListBuckets)
 
 	tests := []struct {
 		name       string
-		c          S3ApiController
 		args       args
+		app        *fiber.App
 		wantErr    bool
 		statusCode int
 	}{
 		{
-			name: "List-bucket-not-implemented",
-			c: S3ApiController{
-				be: backend.BackendUnsupported{},
-			},
+			name: "List-bucket-method-not-allowed",
 			args: args{
-				ctx: app.AcquireCtx(&fasthttp.RequestCtx{}),
+				req: httptest.NewRequest(http.MethodGet, "/", nil),
 			},
+			app:        appErr,
 			wantErr:    false,
-			statusCode: 501,
+			statusCode: 405,
 		},
 		{
 			name: "list-bucket-success",
-			c: S3ApiController{
-				be: &BackendMock{
-					ListBucketsFunc: func() (*s3.ListBucketsOutput, error) {
-						return &s3.ListBucketsOutput{}, nil
-					},
-				},
-			},
 			args: args{
-				ctx: app.AcquireCtx(&fasthttp.RequestCtx{}),
+				req: httptest.NewRequest(http.MethodGet, "/", nil),
 			},
+			app:        app,
 			wantErr:    false,
 			statusCode: 200,
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			err := tt.c.ListBuckets(tt.args.ctx)
+			resp, err := tt.app.Test(tt.args.req)
 
 			if (err != nil) != tt.wantErr {
 				t.Errorf("S3ApiController.ListBuckets() error = %v, wantErr %v", err, tt.wantErr)
-				return
 			}
 
-			statusCode := tt.args.ctx.Response().StatusCode()
-
-			if statusCode != tt.statusCode {
-				t.Errorf("S3ApiController.ListBuckets() code = %v, wantErr %v", statusCode, tt.wantErr)
+			if resp.StatusCode != tt.statusCode {
+				t.Errorf("S3ApiController.ListBuckets() statusCode = %v, wantStatusCode = %v", resp.StatusCode, tt.statusCode)
 			}
 		})
 	}
@@ -127,20 +170,31 @@ func TestS3ApiController_GetActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		ListObjectPartsFunc: func(bucket, object, uploadID string, partNumberMarker int, maxParts int) (*s3.ListPartsOutput, error) {
-			return &s3.ListPartsOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListObjectPartsFunc: func(bucket, object, uploadID string, partNumberMarker int, maxParts int) (s3response.ListPartsResponse, error) {
+				return s3response.ListPartsResponse{}, nil
+			},
+			GetObjectAclFunc: func(bucket, object string) (*s3.GetObjectAclOutput, error) {
+				return &s3.GetObjectAclOutput{}, nil
+			},
+			GetObjectAttributesFunc: func(bucket, object string, attributes []string) (*s3.GetObjectAttributesOutput, error) {
+				return &s3.GetObjectAttributesOutput{}, nil
+			},
+			GetObjectFunc: func(bucket, object, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
+				return &s3.GetObjectOutput{Metadata: nil}, nil
+			},
 		},
-		GetObjectAclFunc: func(bucket, object string) (*s3.GetObjectAclOutput, error) {
-			return &s3.GetObjectAclOutput{}, nil
-		},
-		GetObjectAttributesFunc: func(bucket, object string, attributes []string) (*s3.GetObjectAttributesOutput, error) {
-			return &s3.GetObjectAttributesOutput{}, nil
-		},
-		GetObjectFunc: func(bucket, object, acceptRange string, writer io.Writer) (*s3.GetObjectOutput, error) {
-			return &s3.GetObjectOutput{Metadata: nil}, nil
-		},
-	}}
+	}
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Get("/:bucket/:key/*", s3ApiController.GetActions)
 
 	// GetObjectACL
@@ -169,16 +223,16 @@ func TestS3ApiController_GetActions(t *testing.T) {
 				req: httptest.NewRequest(http.MethodGet, "/my-bucket/key?uploadId=hello&max-parts=InvalidMaxParts", nil),
 			},
 			wantErr:    false,
-			statusCode: 500,
+			statusCode: 400,
 		},
 		{
-			name: "Get-actions-invalid-part-number",
+			name: "Get-actions-invalid-part-number-marker",
 			app:  app,
 			args: args{
 				req: httptest.NewRequest(http.MethodGet, "/my-bucket/key?uploadId=hello&max-parts=200&part-number-marker=InvalidPartNumber", nil),
 			},
 			wantErr:    false,
-			statusCode: 500,
+			statusCode: 400,
 		},
 		{
 			name: "Get-actions-list-object-parts-success",
@@ -229,29 +283,50 @@ func TestS3ApiController_ListActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		GetBucketAclFunc: func(bucket string) (*s3.GetBucketAclOutput, error) {
-			return &s3.GetBucketAclOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (s3response.ListMultipartUploadsResponse, error) {
+				return s3response.ListMultipartUploadsResponse{}, nil
+			},
+			ListObjectsV2Func: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
+				return &s3.ListObjectsV2Output{}, nil
+			},
+			ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
+				return &s3.ListObjectsOutput{}, nil
+			},
 		},
-		ListMultipartUploadsFunc: func(output *s3.ListMultipartUploadsInput) (*s3.ListMultipartUploadsOutput, error) {
-			return &s3.ListMultipartUploadsOutput{}, nil
-		},
-		ListObjectsV2Func: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsV2Output, error) {
-			return &s3.ListObjectsV2Output{}, nil
-		},
-		ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
-			return &s3.ListObjectsOutput{}, nil
-		},
-	}}
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
+
 	app.Get("/:bucket", s3ApiController.ListActions)
 
 	//Error case
-	s3ApiControllerError := S3ApiController{be: &BackendMock{
-		ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
-			return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+	s3ApiControllerError := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			ListObjectsFunc: func(bucket, prefix, marker, delim string, maxkeys int) (*s3.ListObjectsOutput, error) {
+				return nil, s3err.GetAPIError(s3err.ErrNotImplemented)
+			},
 		},
-	}}
+	}
 	appError := fiber.New()
+	appError.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appError.Get("/:bucket", s3ApiControllerError.ListActions)
 
 	tests := []struct {
@@ -328,14 +403,26 @@ func TestS3ApiController_PutBucketActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		PutBucketAclFunc: func(*s3.PutBucketAclInput) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			PutBucketAclFunc: func(string, []byte) error {
+				return nil
+			},
+			PutBucketFunc: func(bucket, owner string) error {
+				return nil
+			},
 		},
-		PutBucketFunc: func(bucket string) error {
-			return nil
-		},
-	}}
+	}
+	// Mock ctx.Locals
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Put("/:bucket", s3ApiController.PutBucketActions)
 
 	// Error case
@@ -401,23 +488,31 @@ func TestS3ApiController_PutActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		UploadPartCopyFunc: func(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
-			return &s3.UploadPartCopyOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			UploadPartCopyFunc: func(*s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {
+				return &s3.UploadPartCopyOutput{}, nil
+			},
+			PutObjectAclFunc: func(*s3.PutObjectAclInput) error {
+				return nil
+			},
+			CopyObjectFunc: func(srcBucket, srcObject, DstBucket, dstObject string) (*s3.CopyObjectOutput, error) {
+				return &s3.CopyObjectOutput{}, nil
+			},
+			PutObjectFunc: func(*s3.PutObjectInput) (string, error) {
+				return "Hey", nil
+			},
 		},
-		UploadPartFunc: func(bucket, object, uploadId string, Body io.ReadSeeker) (*s3.UploadPartOutput, error) {
-			return &s3.UploadPartOutput{}, nil
-		},
-		PutObjectAclFunc: func(*s3.PutObjectAclInput) error {
-			return nil
-		},
-		CopyObjectFunc: func(srcBucket, srcObject, DstBucket, dstObject string) (*s3.CopyObjectOutput, error) {
-			return &s3.CopyObjectOutput{}, nil
-		},
-		PutObjectFunc: func(*s3.PutObjectInput) (string, error) {
-			return "Hey", nil
-		},
-	}}
+	}
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Put("/:bucket/:key/*", s3ApiController.PutActions)
 
 	//PutObjectAcl error
@@ -441,13 +536,13 @@ func TestS3ApiController_PutActions(t *testing.T) {
 		statusCode int
 	}{
 		{
-			name: "Upload-copy-part-error-case",
+			name: "Upload-put-part-error-case",
 			app:  app,
 			args: args{
-				req: httptest.NewRequest(http.MethodPut, "/my-bucket/my-key?partNumber=invalid", nil),
+				req: httptest.NewRequest(http.MethodPut, "/my-bucket/my-key?uploadId=abc&partNumber=invalid", nil),
 			},
 			wantErr:    false,
-			statusCode: 500,
+			statusCode: 400,
 		},
 		{
 			name: "Upload-copy-part-success",
@@ -517,11 +612,13 @@ func TestS3ApiController_PutActions(t *testing.T) {
 		resp, err := tt.app.Test(tt.args.req)
 
 		if (err != nil) != tt.wantErr {
-			t.Errorf("S3ApiController.GetActions() error = %v, wantErr %v", err, tt.wantErr)
+			t.Errorf("S3ApiController.GetActions() %v error = %v, wantErr %v",
+				tt.name, err, tt.wantErr)
 		}
 
 		if resp.StatusCode != tt.statusCode {
-			t.Errorf("S3ApiController.GetActions() statusCode = %v, wantStatusCode = %v", resp.StatusCode, tt.statusCode)
+			t.Errorf("S3ApiController.GetActions() %v statusCode = %v, wantStatusCode = %v",
+				tt.name, resp.StatusCode, tt.statusCode)
 		}
 	}
 }
@@ -532,23 +629,46 @@ func TestS3ApiController_DeleteBucket(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		DeleteBucketFunc: func(bucket string) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteBucketFunc: func(bucket string) error {
+				return nil
+			},
 		},
-	}}
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 
 	app.Delete("/:bucket", s3ApiController.DeleteBucket)
 
 	// error case
 	appErr := fiber.New()
 
-	s3ApiControllerErr := S3ApiController{be: &BackendMock{
-		DeleteBucketFunc: func(bucket string) error {
-			return s3err.GetAPIError(48)
+	s3ApiControllerErr := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteBucketFunc: func(bucket string) error {
+				return s3err.GetAPIError(48)
+			},
 		},
-	}}
+	}
 
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appErr.Delete("/:bucket", s3ApiControllerErr.DeleteBucket)
 
 	tests := []struct {
@@ -596,12 +716,23 @@ func TestS3ApiController_DeleteObjects(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteObjectsFunc: func(bucket string, objects *s3.DeleteObjectsInput) error {
+				return nil
+			},
 		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Post("/:bucket", s3ApiController.DeleteObjects)
 
 	// Valid request body
@@ -655,26 +786,46 @@ func TestS3ApiController_DeleteActions(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		DeleteObjectFunc: func(bucket, object string) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			DeleteObjectFunc: func(bucket, object string) error {
+				return nil
+			},
+			AbortMultipartUploadFunc: func(*s3.AbortMultipartUploadInput) error {
+				return nil
+			},
 		},
-		AbortMultipartUploadFunc: func(*s3.AbortMultipartUploadInput) error {
-			return nil
-		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Delete("/:bucket/:key/*", s3ApiController.DeleteActions)
 
 	//Error case
 	appErr := fiber.New()
 
 	s3ApiControllerErr := S3ApiController{be: &BackendMock{
+		GetBucketAclFunc: func(bucket string) ([]byte, error) {
+			return acldata, nil
+		},
 		DeleteObjectFunc: func(bucket, object string) error {
 			return s3err.GetAPIError(7)
 		},
 	}}
 
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appErr.Delete("/:bucket", s3ApiControllerErr.DeleteBucket)
 
 	tests := []struct {
@@ -731,22 +882,45 @@ func TestS3ApiController_HeadBucket(t *testing.T) {
 	}
 
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		HeadBucketFunc: func(bucket string) (*s3.HeadBucketOutput, error) {
-			return &s3.HeadBucketOutput{}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			HeadBucketFunc: func(bucket string) (*s3.HeadBucketOutput, error) {
+				return &s3.HeadBucketOutput{}, nil
+			},
 		},
-	}}
+	}
+
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 
 	app.Head("/:bucket", s3ApiController.HeadBucket)
 
-	//Error case
+	// Error case
 	appErr := fiber.New()
 
 	s3ApiControllerErr := S3ApiController{be: &BackendMock{
+		GetBucketAclFunc: func(bucket string) ([]byte, error) {
+			return acldata, nil
+		},
 		HeadBucketFunc: func(bucket string) (*s3.HeadBucketOutput, error) {
 			return nil, s3err.GetAPIError(3)
 		},
-	}}
+	},
+	}
+
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 
 	appErr.Head("/:bucket", s3ApiControllerErr.HeadBucket)
 
@@ -802,29 +976,51 @@ func TestS3ApiController_HeadObject(t *testing.T) {
 	eTag := "Valid etag"
 	lastModifie := time.Now()
 
-	s3ApiController := S3ApiController{be: &BackendMock{
-		HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
-			return &s3.HeadObjectOutput{
-				ContentEncoding: &contentEncoding,
-				ContentLength:   64,
-				ContentType:     &contentType,
-				LastModified:    &lastModifie,
-				ETag:            &eTag,
-			}, nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
+				return &s3.HeadObjectOutput{
+					ContentEncoding: &contentEncoding,
+					ContentLength:   64,
+					ContentType:     &contentType,
+					LastModified:    &lastModifie,
+					ETag:            &eTag,
+				}, nil
+			},
 		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Head("/:bucket/:key/*", s3ApiController.HeadObject)
 
 	//Error case
 	appErr := fiber.New()
 
-	s3ApiControllerErr := S3ApiController{be: &BackendMock{
-		HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
-			return nil, s3err.GetAPIError(42)
+	s3ApiControllerErr := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			HeadObjectFunc: func(bucket, object string) (*s3.HeadObjectOutput, error) {
+				return nil, s3err.GetAPIError(42)
+			},
 		},
-	}}
+	}
 
+	appErr.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	appErr.Head("/:bucket/:key/*", s3ApiControllerErr.HeadObject)
 
 	tests := []struct {
@@ -871,18 +1067,29 @@ func TestS3ApiController_CreateActions(t *testing.T) {
 		req *http.Request
 	}
 	app := fiber.New()
-	s3ApiController := S3ApiController{be: &BackendMock{
-		RestoreObjectFunc: func(bucket, object string, restoreRequest *s3.RestoreObjectInput) error {
-			return nil
+	s3ApiController := S3ApiController{
+		be: &BackendMock{
+			GetBucketAclFunc: func(bucket string) ([]byte, error) {
+				return acldata, nil
+			},
+			RestoreObjectFunc: func(bucket, object string, restoreRequest *s3.RestoreObjectInput) error {
+				return nil
+			},
+			CompleteMultipartUploadFunc: func(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error) {
+				return &s3.CompleteMultipartUploadOutput{}, nil
+			},
+			CreateMultipartUploadFunc: func(*s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error) {
+				return &s3.CreateMultipartUploadOutput{}, nil
+			},
 		},
-		CompleteMultipartUploadFunc: func(bucket, object, uploadID string, parts []types.Part) (*s3.CompleteMultipartUploadOutput, error) {
-			return &s3.CompleteMultipartUploadOutput{}, nil
-		},
-		CreateMultipartUploadFunc: func(*s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error) {
-			return &s3.CreateMultipartUploadOutput{}, nil
-		},
-	}}
+	}
 
+	app.Use(func(ctx *fiber.Ctx) error {
+		ctx.Locals("access", "valid access")
+		ctx.Locals("isRoot", true)
+		ctx.Locals("isDebug", false)
+		return ctx.Next()
+	})
 	app.Post("/:bucket/:key/*", s3ApiController.CreateActions)
 
 	tests := []struct {
@@ -951,7 +1158,7 @@ func TestS3ApiController_CreateActions(t *testing.T) {
 	}
 }
 
-func Test_responce(t *testing.T) {
+func Test_XMLresponse(t *testing.T) {
 	type args struct {
 		ctx  *fiber.Ctx
 		resp any
@@ -959,6 +1166,15 @@ func Test_responce(t *testing.T) {
 	}
 	app := fiber.New()
 
+	var ctx fiber.Ctx
+	// Mocking the fiber ctx
+	app.Get("/:bucket/:key", func(c *fiber.Ctx) error {
+		ctx = *c
+		return nil
+	})
+
+	app.Test(httptest.NewRequest(http.MethodGet, "/my-bucket/my-key", nil))
+
 	tests := []struct {
 		name       string
 		args       args
@@ -968,7 +1184,7 @@ func Test_responce(t *testing.T) {
 		{
 			name: "Internal-server-error",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: nil,
 				err:  s3err.GetAPIError(16),
 			},
@@ -978,7 +1194,7 @@ func Test_responce(t *testing.T) {
 		{
 			name: "Error-not-implemented",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: nil,
 				err:  s3err.GetAPIError(50),
 			},
@@ -988,7 +1204,7 @@ func Test_responce(t *testing.T) {
 		{
 			name: "Invalid-request-body",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: make(chan int),
 				err:  nil,
 			},
@@ -998,7 +1214,7 @@ func Test_responce(t *testing.T) {
 		{
 			name: "Successful-response",
 			args: args{
-				ctx:  app.AcquireCtx(&fasthttp.RequestCtx{}),
+				ctx:  &ctx,
 				resp: "Valid response",
 				err:  nil,
 			},
@@ -1008,14 +1224,84 @@ func Test_responce(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			if err := Responce(tt.args.ctx, tt.args.resp, tt.args.err); (err != nil) != tt.wantErr {
-				t.Errorf("responce() error = %v, wantErr %v", err, tt.wantErr)
+			if err := SendXMLResponse(tt.args.ctx, tt.args.resp, tt.args.err); (err != nil) != tt.wantErr {
+				t.Errorf("response() %v error = %v, wantErr %v", tt.name, err, tt.wantErr)
 			}
 
 			statusCode := tt.args.ctx.Response().StatusCode()
 
 			if statusCode != tt.statusCode {
-				t.Errorf("responce() code = %v, wantErr %v", statusCode, tt.wantErr)
+				t.Errorf("response() %v code = %v, wantErr %v", tt.name, statusCode, tt.wantErr)
+			}
+
+			tt.args.ctx.Status(http.StatusOK)
+		})
+	}
+}
+
+func Test_response(t *testing.T) {
+	type args struct {
+		ctx  *fiber.Ctx
+		resp any
+		err  error
+	}
+	app := fiber.New()
+	var ctx fiber.Ctx
+	// Mocking the fiber ctx
+	app.Get("/:bucket/:key", func(c *fiber.Ctx) error {
+		ctx = *c
+		return nil
+	})
+
+	app.Test(httptest.NewRequest(http.MethodGet, "/my-bucket/my-key", nil))
+
+	tests := []struct {
+		name       string
+		args       args
+		wantErr    bool
+		statusCode int
+	}{
+		{
+			name: "Internal-server-error",
+			args: args{
+				ctx:  &ctx,
+				resp: nil,
+				err:  s3err.GetAPIError(16),
+			},
+			wantErr:    false,
+			statusCode: 500,
+		},
+		{
+			name: "Error-not-implemented",
+			args: args{
+				ctx:  &ctx,
+				resp: nil,
+				err:  s3err.GetAPIError(50),
+			},
+			wantErr:    false,
+			statusCode: 501,
+		},
+		{
+			name: "Successful-response",
+			args: args{
+				ctx:  &ctx,
+				resp: "Valid response",
+				err:  nil,
+			},
+			wantErr:    false,
+			statusCode: 200,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if err := SendResponse(tt.args.ctx, tt.args.err); (err != nil) != tt.wantErr {
+				t.Errorf("response() %v error = %v, wantErr %v", tt.name, err, tt.wantErr)
+			}
+
+			statusCode := tt.args.ctx.Response().StatusCode()
+
+			if statusCode != tt.statusCode {
+				t.Errorf("response() %v code = %v, wantErr %v", tt.name, statusCode, tt.wantErr)
 			}
 		})
 	}

s3api/middlewares/authentication.go

@@ -25,7 +25,7 @@ import (
 	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
 	"github.com/aws/smithy-go/logging"
 	"github.com/gofiber/fiber/v2"
-	"github.com/versity/versitygw/backend/auth"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/s3api/controllers"
 	"github.com/versity/versitygw/s3api/utils"
 	"github.com/versity/versitygw/s3err"
@@ -35,63 +35,62 @@ const (
 	iso8601Format = "20060102T150405Z"
 )
 
-type AdminConfig struct {
-	AdminAccess string
-	AdminSecret string
-	Region      string
+type RootUserConfig struct {
+	Access string
+	Secret string
 }
 
-func VerifyV4Signature(config AdminConfig, iam auth.IAMService, debug bool) fiber.Handler {
-	acct := accounts{
-		admin: config,
-		iam:   iam,
-	}
+func VerifyV4Signature(root RootUserConfig, iam auth.IAMService, region string, debug bool) fiber.Handler {
+	acct := accounts{root: root, iam: iam}
 
 	return func(ctx *fiber.Ctx) error {
 		authorization := ctx.Get("Authorization")
 		if authorization == "" {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrAuthHeaderEmpty))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrAuthHeaderEmpty))
 		}
 
 		// Check the signature version
 		authParts := strings.Split(authorization, " ")
 		if len(authParts) < 4 {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrMissingFields))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrMissingFields))
 		}
 		if authParts[0] != "AWS4-HMAC-SHA256" {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrSignatureVersionNotSupported))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrSignatureVersionNotSupported))
 		}
 
 		credKv := strings.Split(authParts[1], "=")
 		if len(credKv) != 2 {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrCredMalformed))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrCredMalformed))
 		}
 		creds := strings.Split(credKv[1], "/")
 		if len(creds) < 4 {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrCredMalformed))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrCredMalformed))
 		}
 
-		signHdrKv := strings.Split(authParts[2], "=")
+		signHdrKv := strings.Split(authParts[2][:len(authParts[2])-1], "=")
 		if len(signHdrKv) != 2 {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrCredMalformed))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrCredMalformed))
 		}
 		signedHdrs := strings.Split(signHdrKv[1], ";")
 
-		secret, ok := acct.getAcctSecret(creds[0])
-		if !ok {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrInvalidAccessKeyID))
+		account, err := acct.getAccount(creds[0])
+		if err == auth.ErrNoSuchUser {
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidAccessKeyID))
+		}
+		if err != nil {
+			return controllers.SendResponse(ctx, err)
 		}
 
 		// Check X-Amz-Date header
 		date := ctx.Get("X-Amz-Date")
 		if date == "" {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrMissingDateHeader))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrMissingDateHeader))
 		}
 
 		// Parse the date and check the date validity
 		tdate, err := time.Parse(iso8601Format, date)
 		if err != nil {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrMalformedDate))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrMalformedDate))
 		}
 
 		// Calculate the hash of the request payload
@@ -102,60 +101,61 @@ func VerifyV4Signature(config AdminConfig, iam auth.IAMService, debug bool) fibe
 
 		// Compare the calculated hash with the hash provided
 		if hashPayloadHeader != hexPayload {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrContentSHA256Mismatch))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrContentSHA256Mismatch))
 		}
 
 		// Create a new http request instance from fasthttp request
 		req, err := utils.CreateHttpRequestFromCtx(ctx, signedHdrs)
 		if err != nil {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrInternalError))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrInternalError))
 		}
 
 		signer := v4.NewSigner()
 
 		signErr := signer.SignHTTP(req.Context(), aws.Credentials{
 			AccessKeyID:     creds[0],
-			SecretAccessKey: secret,
-		}, req, hexPayload, creds[3], config.Region, tdate, func(options *v4.SignerOptions) {
+			SecretAccessKey: account.Secret,
+		}, req, hexPayload, creds[3], region, tdate, func(options *v4.SignerOptions) {
 			if debug {
 				options.LogSigning = true
 				options.Logger = logging.NewStandardLogger(os.Stderr)
 			}
 		})
 		if signErr != nil {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrInternalError))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrInternalError))
 		}
 
 		parts := strings.Split(req.Header.Get("Authorization"), " ")
 		if len(parts) < 4 {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrMissingFields))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrMissingFields))
 		}
 		calculatedSign := strings.Split(parts[3], "=")[1]
 		expectedSign := strings.Split(authParts[3], "=")[1]
 
 		if expectedSign != calculatedSign {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrSignatureDoesNotMatch))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrSignatureDoesNotMatch))
 		}
 
+		ctx.Locals("role", account.Role)
+		ctx.Locals("access", creds[0])
+		ctx.Locals("isRoot", creds[0] == root.Access)
+
 		return ctx.Next()
 	}
 }
 
 type accounts struct {
-	admin AdminConfig
-	iam   auth.IAMService
+	root RootUserConfig
+	iam  auth.IAMService
 }
 
-func (a accounts) getAcctSecret(access string) (string, bool) {
-	if a.admin.AdminAccess == access {
-		return a.admin.AdminSecret, true
+func (a accounts) getAccount(access string) (auth.Account, error) {
+	if access == a.root.Access {
+		return auth.Account{
+			Secret: a.root.Secret,
+			Role:   "admin",
+		}, nil
 	}
 
-	conf, err := a.iam.GetIAMConfig()
-	if err != nil {
-		return "", false
-	}
-
-	secret, ok := conf.AccessAccounts[access]
-	return secret, ok
+	return a.iam.GetUserAccount(access)
 }

s3api/middlewares/logger.go

@@ -0,0 +1,44 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package middlewares
+
+import (
+	"fmt"
+	"log"
+
+	"github.com/gofiber/fiber/v2"
+)
+
+func RequestLogger(isDebug bool) fiber.Handler {
+	return func(ctx *fiber.Ctx) error {
+		ctx.Locals("isDebug", isDebug)
+		if isDebug {
+			log.Println("Request headers: ")
+			ctx.Request().Header.VisitAll(func(key, val []byte) {
+				log.Printf("%s: %s", key, val)
+			})
+
+			if ctx.Request().URI().QueryArgs().Len() != 0 {
+				fmt.Println()
+				log.Println("Request query arguments: ")
+				ctx.Request().URI().QueryArgs().VisitAll(func(key, val []byte) {
+					log.Printf("%s: %s", key, val)
+				})
+			}
+		}
+
+		return ctx.Next()
+	}
+}

s3api/middlewares/md5.go

@@ -34,7 +34,7 @@ func VerifyMD5Body() fiber.Handler {
 		calculatedSum := base64.StdEncoding.EncodeToString(sum[:])
 
 		if incomingSum != calculatedSum {
-			return controllers.Responce[any](ctx, nil, s3err.GetAPIError(s3err.ErrInvalidDigest))
+			return controllers.SendResponse(ctx, s3err.GetAPIError(s3err.ErrInvalidDigest))
 		}
 
 		return ctx.Next()

s3api/router.go

@@ -16,14 +16,22 @@ package s3api
 
 import (
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 	"github.com/versity/versitygw/s3api/controllers"
 )
 
 type S3ApiRouter struct{}
 
-func (sa *S3ApiRouter) Init(app *fiber.App, be backend.Backend) {
+func (sa *S3ApiRouter) Init(app *fiber.App, be backend.Backend, iam auth.IAMService) {
 	s3ApiController := controllers.New(be)
+	adminController := controllers.AdminController{IAMService: iam}
+
+	// TODO: think of better routing system
+	app.Post("/create-user", adminController.CreateUser)
+
+	// Admin Delete api
+	app.Delete("/delete-user", adminController.DeleteUser)
 	// ListBuckets action
 	app.Get("/", s3ApiController.ListBuckets)
 

s3api/router_test.go

@@ -18,6 +18,7 @@ import (
 	"testing"
 
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
 )
 
@@ -25,6 +26,7 @@ func TestS3ApiRouter_Init(t *testing.T) {
 	type args struct {
 		app *fiber.App
 		be  backend.Backend
+		iam auth.IAMService
 	}
 	tests := []struct {
 		name string
@@ -37,12 +39,13 @@ func TestS3ApiRouter_Init(t *testing.T) {
 			args: args{
 				app: fiber.New(),
 				be:  backend.BackendUnsupported{},
+				iam: &auth.IAMServiceInternal{},
 			},
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			tt.sa.Init(tt.args.app, tt.args.be)
+			tt.sa.Init(tt.args.app, tt.args.be, tt.args.iam)
 		})
 	}
 }

s3api/server.go

@@ -19,8 +19,8 @@ import (
 
 	"github.com/gofiber/fiber/v2"
 	"github.com/gofiber/fiber/v2/middleware/logger"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api/middlewares"
 )
 
@@ -33,7 +33,7 @@ type S3ApiServer struct {
 	debug   bool
 }
 
-func New(app *fiber.App, be backend.Backend, port string, adminUser middlewares.AdminConfig, iam auth.IAMService, opts ...Option) (*S3ApiServer, error) {
+func New(app *fiber.App, be backend.Backend, root middlewares.RootUserConfig, port, region string, iam auth.IAMService, opts ...Option) (*S3ApiServer, error) {
 	server := &S3ApiServer{
 		app:     app,
 		backend: be,
@@ -45,10 +45,16 @@ func New(app *fiber.App, be backend.Backend, port string, adminUser middlewares.
 		opt(server)
 	}
 
-	app.Use(middlewares.VerifyV4Signature(adminUser, iam, server.debug))
+	// Logging middlewares
 	app.Use(logger.New())
+	app.Use(middlewares.RequestLogger(server.debug))
+
+	// Authentication middlewares
+	app.Use(middlewares.VerifyV4Signature(root, iam, region, server.debug))
 	app.Use(middlewares.VerifyMD5Body())
-	server.router.Init(app, be)
+
+	server.router.Init(app, be, iam)
+
 	return server, nil
 }
 

s3api/server_test.go

@@ -19,17 +19,17 @@ import (
 	"testing"
 
 	"github.com/gofiber/fiber/v2"
+	"github.com/versity/versitygw/auth"
 	"github.com/versity/versitygw/backend"
-	"github.com/versity/versitygw/backend/auth"
 	"github.com/versity/versitygw/s3api/middlewares"
 )
 
 func TestNew(t *testing.T) {
 	type args struct {
-		app       *fiber.App
-		be        backend.Backend
-		port      string
-		adminUser middlewares.AdminConfig
+		app  *fiber.App
+		be   backend.Backend
+		port string
+		root middlewares.RootUserConfig
 	}
 
 	app := fiber.New()
@@ -46,10 +46,10 @@ func TestNew(t *testing.T) {
 		{
 			name: "Create S3 api server",
 			args: args{
-				app:       app,
-				be:        be,
-				port:      port,
-				adminUser: middlewares.AdminConfig{},
+				app:  app,
+				be:   be,
+				port: port,
+				root: middlewares.RootUserConfig{},
 			},
 			wantS3ApiServer: &S3ApiServer{
 				app:     app,
@@ -62,8 +62,8 @@ func TestNew(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			gotS3ApiServer, err := New(tt.args.app, tt.args.be,
-				tt.args.port, tt.args.adminUser, auth.IAMServiceUnsupported{})
+			gotS3ApiServer, err := New(tt.args.app, tt.args.be, tt.args.root,
+				tt.args.port, "us-east-1", &auth.IAMServiceInternal{})
 			if (err != nil) != tt.wantErr {
 				t.Errorf("New() error = %v, wantErr %v", err, tt.wantErr)
 				return

s3api/utils/logger.go

@@ -0,0 +1,55 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package utils
+
+import (
+	"fmt"
+	"log"
+
+	"github.com/gofiber/fiber/v2"
+)
+
+func LogCtxDetails(ctx *fiber.Ctx, respBody []byte) {
+	isDebug, ok := ctx.Locals("isDebug").(bool)
+	_, notLogReqBody := ctx.Locals("logReqBody").(bool)
+	_, notLogResBody := ctx.Locals("logResBody").(bool)
+	if isDebug && ok {
+		// Log request body
+		if !notLogReqBody {
+			fmt.Println()
+			log.Printf("Request Body: %s", ctx.Request().Body())
+		}
+
+		// Log path parameters
+		fmt.Println()
+		log.Println("Path parameters: ")
+		for key, val := range ctx.AllParams() {
+			log.Printf("%s: %s", key, val)
+		}
+
+		// Log response headers
+		fmt.Println()
+		log.Println("Response Headers: ")
+		ctx.Response().Header.VisitAll(func(key, val []byte) {
+			log.Printf("%s: %s", key, val)
+		})
+
+		// Log response body
+		if !notLogResBody && len(respBody) > 0 {
+			fmt.Println()
+			log.Printf("Response body %s", ctx.Response().Body())
+		}
+	}
+}

s3api/utils/utils_test.go

@@ -2,7 +2,6 @@ package utils
 
 import (
 	"bytes"
-	"fmt"
 	"net/http"
 	"reflect"
 	"testing"
@@ -62,8 +61,6 @@ func TestCreateHttpRequestFromCtx(t *testing.T) {
 				return
 			}
 
-			fmt.Println(got.Header, tt.want.Header)
-
 			if !reflect.DeepEqual(got.Header, tt.want.Header) {
 				t.Errorf("CreateHttpRequestFromCtx() got = %v, want %v", got, tt.want)
 			}

s3err/s3err.go

@@ -105,6 +105,7 @@ const (
 	ErrAuthNotSetup
 	ErrNotImplemented
 	ErrPreconditionFailed
+	ErrInvalidObjectState
 
 	ErrExistingObjectIsDirectory
 	ErrObjectParentIsFile
@@ -368,6 +369,11 @@ var errorCodeResponse = map[ErrorCode]APIError{
 		Description:    "At least one of the pre-conditions you specified did not hold",
 		HTTPStatusCode: http.StatusPreconditionFailed,
 	},
+	ErrInvalidObjectState: {
+		Code:           "InvalidObjectState",
+		Description:    "The operation is not valid for the current state of the object",
+		HTTPStatusCode: http.StatusForbidden,
+	},
 	ErrExistingObjectIsDirectory: {
 		Code:           "ExistingObjectIsDirectory",
 		Description:    "Existing Object is a directory.",

s3response/AmazonS3.xsd

@@ -0,0 +1,692 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xsd:schema
+  xmlns:tns="http://s3.amazonaws.com/doc/2006-03-01/"
+  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
+  elementFormDefault="qualified"
+  targetNamespace="http://s3.amazonaws.com/doc/2006-03-01/">
+
+  <xsd:element name="CreateBucket">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="MetadataEntry">
+    <xsd:sequence>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="Value" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="CreateBucketResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="CreateBucketReturn" type="tns:CreateBucketResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="Status">
+    <xsd:sequence>
+      <xsd:element name="Code" type="xsd:int"/>
+      <xsd:element name="Description" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="Result">
+    <xsd:sequence>
+      <xsd:element name="Status" type="tns:Status"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="CreateBucketResult">
+    <xsd:sequence>
+      <xsd:element name="BucketName" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="DeleteBucket">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+  <xsd:element name="DeleteBucketResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="DeleteBucketResponse" type="tns:Status"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="BucketLoggingStatus">
+    <xsd:sequence>
+      <xsd:element name="LoggingEnabled" type="tns:LoggingSettings" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="LoggingSettings">
+    <xsd:sequence>
+      <xsd:element name="TargetBucket" type="xsd:string"/>
+      <xsd:element name="TargetPrefix" type="xsd:string"/>
+      <xsd:element name="TargetGrants" type="tns:AccessControlList" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="GetBucketLoggingStatus">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+  
+  <xsd:element name="GetBucketLoggingStatusResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetBucketLoggingStatusResponse" type="tns:BucketLoggingStatus"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+  
+  <xsd:element name="SetBucketLoggingStatus">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="BucketLoggingStatus" type="tns:BucketLoggingStatus"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetBucketLoggingStatusResponse">
+    <xsd:complexType>
+      <xsd:sequence/>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetObjectAccessControlPolicyResponse" type="tns:AccessControlPolicy"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetBucketAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetBucketAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetBucketAccessControlPolicyResponse" type="tns:AccessControlPolicy"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType abstract="true" name="Grantee"/>
+
+  <xsd:complexType name="User" abstract="true">
+    <xsd:complexContent>
+      <xsd:extension base="tns:Grantee"/>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="AmazonCustomerByEmail">
+    <xsd:complexContent>
+      <xsd:extension base="tns:User">
+        <xsd:sequence>
+          <xsd:element name="EmailAddress" type="xsd:string"/>              
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="CanonicalUser">
+    <xsd:complexContent>
+      <xsd:extension base="tns:User">
+        <xsd:sequence>
+          <xsd:element name="ID" type="xsd:string"/>              
+          <xsd:element name="DisplayName" type="xsd:string" minOccurs="0"/>
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="Group">
+    <xsd:complexContent>
+      <xsd:extension base="tns:Grantee">
+        <xsd:sequence>
+          <xsd:element name="URI" type="xsd:string"/>
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:simpleType name="Permission">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="READ"/>
+      <xsd:enumeration value="WRITE"/>
+      <xsd:enumeration value="READ_ACP"/>
+      <xsd:enumeration value="WRITE_ACP"/>
+      <xsd:enumeration value="FULL_CONTROL"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:simpleType name="StorageClass">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="STANDARD"/>
+      <xsd:enumeration value="REDUCED_REDUNDANCY"/>
+      <xsd:enumeration value="GLACIER"/>
+      <xsd:enumeration value="UNKNOWN"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:complexType name="Grant">
+    <xsd:sequence>
+      <xsd:element name="Grantee" type="tns:Grantee"/>
+      <xsd:element name="Permission" type="tns:Permission"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="AccessControlList">
+    <xsd:sequence>
+      <xsd:element name="Grant" type="tns:Grant" minOccurs="0" maxOccurs="100"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="CreateBucketConfiguration">
+    <xsd:sequence>
+      <xsd:element name="LocationConstraint" type="tns:LocationConstraint"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="LocationConstraint">
+    <xsd:simpleContent>
+      <xsd:extension base="xsd:string"/>
+    </xsd:simpleContent>
+  </xsd:complexType>
+
+  <xsd:complexType name="AccessControlPolicy">
+    <xsd:sequence>
+      <xsd:element name="Owner" type="tns:CanonicalUser"/>
+      <xsd:element name="AccessControlList" type="tns:AccessControlList"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="SetObjectAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetObjectAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence/>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetBucketAccessControlPolicy">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="SetBucketAccessControlPolicyResponse">
+    <xsd:complexType>
+      <xsd:sequence/>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="GetMetadata" type="xsd:boolean"/>
+        <xsd:element name="GetData" type="xsd:boolean"/>
+        <xsd:element name="InlineData" type="xsd:boolean"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+        
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetObjectResponse" type="tns:GetObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="GetObjectResult">
+    <xsd:complexContent>
+      <xsd:extension base="tns:Result">
+        <xsd:sequence>
+          <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
+          <xsd:element name="Data" type="xsd:base64Binary" nillable="true"/>
+          <xsd:element name="LastModified" type="xsd:dateTime"/>
+          <xsd:element name="ETag" type="xsd:string"/>
+        </xsd:sequence>
+      </xsd:extension>
+    </xsd:complexContent>
+  </xsd:complexType>
+
+  <xsd:element name="GetObjectExtended">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="GetMetadata" type="xsd:boolean"/>
+        <xsd:element name="GetData" type="xsd:boolean"/>
+        <xsd:element name="InlineData" type="xsd:boolean"/>
+        <xsd:element name="ByteRangeStart" type="xsd:long" minOccurs="0"/>
+        <xsd:element name="ByteRangeEnd" type="xsd:long" minOccurs="0"/>
+        <xsd:element name="IfModifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="IfUnmodifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="IfMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="IfNoneMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="ReturnCompleteObjectOnConditionFailure" type="xsd:boolean" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="GetObjectExtendedResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="GetObjectResponse" type="tns:GetObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="PutObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="ContentLength" type="xsd:long"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="PutObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="PutObjectResponse" type="tns:PutObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="PutObjectResult">
+    <xsd:sequence>
+      <xsd:element name="ETag" type="xsd:string"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="PutObjectInline">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element minOccurs="0" maxOccurs="100" name="Metadata" type="tns:MetadataEntry"/>
+        <xsd:element name="Data" type="xsd:base64Binary"/>
+        <xsd:element name="ContentLength" type="xsd:long"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="PutObjectInlineResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="PutObjectInlineResponse" type="tns:PutObjectResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="DeleteObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="DeleteObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="DeleteObjectResponse" type="tns:Status"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListBucket">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Prefix" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Marker" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="MaxKeys" type="xsd:int" minOccurs="0"/>
+        <xsd:element name="Delimiter" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListBucketResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="ListBucketResponse" type="tns:ListBucketResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListVersionsResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="ListVersionsResponse" type="tns:ListVersionsResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>  
+
+  <xsd:complexType name="ListEntry">
+    <xsd:sequence>
+      <xsd:element name="Key" type="xsd:string"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="ETag" type="xsd:string"/>
+      <xsd:element name="Size" type="xsd:long"/>
+      <xsd:element name="Owner" type="tns:CanonicalUser" minOccurs="0"/>
+      <xsd:element name="StorageClass" type="tns:StorageClass"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="VersionEntry">
+    <xsd:sequence>
+      <xsd:element name="Key" type="xsd:string"/>
+      <xsd:element name="VersionId" type="xsd:string"/>
+      <xsd:element name="IsLatest" type="xsd:boolean"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="ETag" type="xsd:string"/>
+      <xsd:element name="Size" type="xsd:long"/>
+      <xsd:element name="Owner" type="tns:CanonicalUser" minOccurs="0"/>
+      <xsd:element name="StorageClass" type="tns:StorageClass"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="DeleteMarkerEntry">
+    <xsd:sequence>
+      <xsd:element name="Key" type="xsd:string"/>
+      <xsd:element name="VersionId" type="xsd:string"/>
+      <xsd:element name="IsLatest" type="xsd:boolean"/>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="Owner" type="tns:CanonicalUser" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="PrefixEntry">
+    <xsd:sequence>
+      <xsd:element name="Prefix" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListBucketResult">
+    <xsd:sequence>
+      <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="Prefix" type="xsd:string"/>
+      <xsd:element name="Marker" type="xsd:string"/>
+      <xsd:element name="NextMarker" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="MaxKeys" type="xsd:int"/>
+      <xsd:element name="Delimiter" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="IsTruncated" type="xsd:boolean"/>
+      <xsd:element name="Contents" type="tns:ListEntry" minOccurs="0" maxOccurs="unbounded"/>
+      <xsd:element name="CommonPrefixes" type="tns:PrefixEntry" minOccurs="0" maxOccurs="unbounded"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListVersionsResult">
+    <xsd:sequence>
+      <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="unbounded"/>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="Prefix" type="xsd:string"/>
+      <xsd:element name="KeyMarker" type="xsd:string"/>
+      <xsd:element name="VersionIdMarker" type="xsd:string"/>
+      <xsd:element name="NextKeyMarker" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="NextVersionIdMarker" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="MaxKeys" type="xsd:int"/>
+      <xsd:element name="Delimiter" type="xsd:string" minOccurs="0"/>
+      <xsd:element name="IsTruncated" type="xsd:boolean"/>
+      <xsd:choice minOccurs="0" maxOccurs="unbounded">
+          <xsd:element name="Version" type="tns:VersionEntry"/>
+          <xsd:element name="DeleteMarker" type="tns:DeleteMarkerEntry"/>
+      </xsd:choice>
+      <xsd:element name="CommonPrefixes" type="tns:PrefixEntry" minOccurs="0" maxOccurs="unbounded"/>
+    </xsd:sequence>
+  </xsd:complexType>  
+
+  <xsd:element name="ListAllMyBuckets">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="ListAllMyBucketsResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="ListAllMyBucketsResponse" type="tns:ListAllMyBucketsResult"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="ListAllMyBucketsEntry">
+    <xsd:sequence>
+      <xsd:element name="Name" type="xsd:string"/>
+      <xsd:element name="CreationDate" type="xsd:dateTime"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListAllMyBucketsResult">
+    <xsd:sequence>
+      <xsd:element name="Owner" type="tns:CanonicalUser"/>
+      <xsd:element name="Buckets" type="tns:ListAllMyBucketsList"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="ListAllMyBucketsList">
+    <xsd:sequence>
+      <xsd:element name="Bucket" type="tns:ListAllMyBucketsEntry" minOccurs="0" maxOccurs="unbounded"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:element name="PostResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="Location" type="xsd:anyURI"/>
+        <xsd:element name="Bucket" type="xsd:string"/>
+        <xsd:element name="Key" type="xsd:string"/>
+        <xsd:element name="ETag" type="xsd:string"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:simpleType name="MetadataDirective">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="COPY"/>
+      <xsd:enumeration value="REPLACE"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:element name="CopyObject">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="SourceBucket" type="xsd:string"/>
+        <xsd:element name="SourceKey" type="xsd:string"/>
+        <xsd:element name="DestinationBucket" type="xsd:string"/>
+        <xsd:element name="DestinationKey" type="xsd:string"/>
+        <xsd:element name="MetadataDirective" type="tns:MetadataDirective" minOccurs="0"/>
+        <xsd:element name="Metadata" type="tns:MetadataEntry" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="AccessControlList" type="tns:AccessControlList" minOccurs="0"/>
+	<xsd:element name="CopySourceIfModifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="CopySourceIfUnmodifiedSince" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="CopySourceIfMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="CopySourceIfNoneMatch" type="xsd:string" minOccurs="0" maxOccurs="100"/>
+        <xsd:element name="StorageClass" type="tns:StorageClass" minOccurs="0"/>
+        <xsd:element name="AWSAccessKeyId" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Timestamp" type="xsd:dateTime" minOccurs="0"/>
+        <xsd:element name="Signature" type="xsd:string" minOccurs="0"/>
+        <xsd:element name="Credential" type="xsd:string" minOccurs="0"/>
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:element name="CopyObjectResponse">
+    <xsd:complexType>
+      <xsd:sequence>
+        <xsd:element name="CopyObjectResult" type="tns:CopyObjectResult" />
+      </xsd:sequence>
+    </xsd:complexType>
+  </xsd:element>
+
+  <xsd:complexType name="CopyObjectResult">
+    <xsd:sequence>
+      <xsd:element name="LastModified" type="xsd:dateTime"/>
+      <xsd:element name="ETag" type="xsd:string"/>      
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="RequestPaymentConfiguration">
+    <xsd:sequence>
+      <xsd:element name="Payer" type="tns:Payer" minOccurs="1" maxOccurs="1"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:simpleType name="Payer">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="BucketOwner"/>
+      <xsd:enumeration value="Requester"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+  
+  <xsd:complexType name="VersioningConfiguration">
+    <xsd:sequence>
+      <xsd:element name="Status" type="tns:VersioningStatus" minOccurs="0"/>
+      <xsd:element name="MfaDelete" type="tns:MfaDeleteStatus" minOccurs="0"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:simpleType name="MfaDeleteStatus">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="Enabled"/>
+      <xsd:enumeration value="Disabled"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:simpleType name="VersioningStatus">
+    <xsd:restriction base="xsd:string">
+      <xsd:enumeration value="Enabled"/>
+      <xsd:enumeration value="Suspended"/>
+    </xsd:restriction>
+  </xsd:simpleType>
+
+  <xsd:complexType name="NotificationConfiguration">
+    <xsd:sequence>
+      <xsd:element name="TopicConfiguration" minOccurs="0" maxOccurs="unbounded" type="tns:TopicConfiguration"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+  <xsd:complexType name="TopicConfiguration">
+    <xsd:sequence>
+      <xsd:element name="Topic" minOccurs="1" maxOccurs="1" type="xsd:string"/>
+      <xsd:element name="Event" minOccurs="1" maxOccurs="unbounded" type="xsd:string"/>
+    </xsd:sequence>
+  </xsd:complexType>
+
+</xsd:schema>

s3response/README.txt

@@ -0,0 +1,6 @@
+https://doc.s3.amazonaws.com/2006-03-01/AmazonS3.xsd
+
+see https://blog.aqwari.net/xml-schema-go/
+
+go install aqwari.net/xml/cmd/xsdgen@latest
+xsdgen -o s3api_xsd_generated.go -pkg s3response AmazonS3.xsd

s3response/s3response.go

@@ -0,0 +1,96 @@
+// Copyright 2023 Versity Software
+// This file is licensed under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package s3response
+
+import (
+	"encoding/xml"
+)
+
+// Part describes part metadata.
+type Part struct {
+	PartNumber   int
+	LastModified string
+	ETag         string
+	Size         int64
+}
+
+// ListPartsResponse - s3 api list parts response.
+type ListPartsResponse struct {
+	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListPartsResult" json:"-"`
+
+	Bucket   string
+	Key      string
+	UploadID string `xml:"UploadId"`
+
+	Initiator Initiator
+	Owner     Owner
+
+	// The class of storage used to store the object.
+	StorageClass string
+
+	PartNumberMarker     int
+	NextPartNumberMarker int
+	MaxParts             int
+	IsTruncated          bool
+
+	// List of parts.
+	Parts []Part `xml:"Part"`
+}
+
+// ListMultipartUploadsResponse - s3 api list multipart uploads response.
+type ListMultipartUploadsResponse struct {
+	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListMultipartUploadsResult" json:"-"`
+
+	Bucket             string
+	KeyMarker          string
+	UploadIDMarker     string `xml:"UploadIdMarker"`
+	NextKeyMarker      string
+	NextUploadIDMarker string `xml:"NextUploadIdMarker"`
+	Delimiter          string
+	Prefix             string
+	EncodingType       string `xml:"EncodingType,omitempty"`
+	MaxUploads         int
+	IsTruncated        bool
+
+	// List of pending uploads.
+	Uploads []Upload `xml:"Upload"`
+
+	// Delimed common prefixes.
+	CommonPrefixes []CommonPrefix
+}
+
+// Upload desribes in progress multipart upload
+type Upload struct {
+	Key          string
+	UploadID     string `xml:"UploadId"`
+	Initiator    Initiator
+	Owner        Owner
+	StorageClass string
+	Initiated    string
+}
+
+// CommonPrefix ListObjectsResponse common prefixes (directory abstraction)
+type CommonPrefix struct {
+	Prefix string
+}
+
+// Initiator same fields as Owner
+type Initiator Owner
+
+// Owner bucket ownership
+type Owner struct {
+	ID          string
+	DisplayName string
+}