mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-19 05:36:58 +00:00
* s3tables: add the maintenance configuration APIs Stores the configuration verbatim as the wire shape under a new s3tables.maintenance extended attribute, so Get hands back what Put took and no translation layer can drift from the AWS model. Nothing reads the configuration yet. Put merges a single type into the stored map so configuring compaction does not drop snapshot management, and asserts the attribute's prior value so two concurrent Puts cannot silently clobber each other. * iceberg: apply the maintenance configuration in the worker The worker now reads the per-table and per-bucket maintenance configuration written by the control plane, so the wildcard plugin config is a default rather than the only setting a table can have. Table properties still win by default, since a table declaring its own layout is what every engine honours and the compactor has to agree with whoever writes the files. Clearing table_properties_override makes the maintenance configuration authoritative instead. Status is not part of that contest: a disabled type drops its operations and no property can re-enable them, so the operator's kill switch always holds. Manifest and delete-file rewrites have no AWS equivalent and ride with compaction. Detection reads both attributes from entries it already lists. * s3tables: report maintenance job status The worker records the outcome of each run in its own extended attribute, separate from the configuration so operator and worker writes do not contend, and GetTableMaintenanceJobStatus reads it back. Only the types a run touched are written, so a partial run cannot erase what an earlier one recorded. The reader fills in the rest: Disabled when the configuration switched a type off, Not_Yet_Run otherwise. Status is advisory, so a lost race is logged rather than failing a job whose work already committed. * s3tables: route the maintenance APIs over REST The five actions were only reachable by X-Amz-Target dispatch, which the AWS CLI and SDK do not use for this service. They address the operations by path, so the APIs were unreachable from any official client. * s3tables: fix the table bucket ARN field name GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the wire field is tableBucketARN, as every other response in this package already spells it. Official SDK deserializers ignore the unknown key, so the required field came back unset. * s3tables: carry the compaction strategy through to the worker IcebergCompactionSettings modelled only targetFileSizeMB, so a request naming a strategy was accepted and then dropped on the way to storage. The worker now maps binpack and sort onto its own rewrite strategy and lets auto defer to the worker configuration. z-order is rejected rather than accepted and quietly binpacked. * s3tables: report bucket-level maintenance status GetTableMaintenanceJobStatus read only the table's configuration, so unreferenced file removal — which is configured on the bucket — reported Not_Yet_Run or a stale success after an operator disabled it. The merge helper now lives in this package and the worker shares it. * iceberg: delete orphans only after the non-current window AWS marks a file non-current once it has been unreferenced for unreferencedDays, then deletes it a further nonCurrentDays later. The cutoff was taken from unreferencedDays alone, so a 3/10 configuration hard-deleted on day three and threw away the ten day recovery window. remove_orphans deletes in one step rather than marking, so the cutoff is now the sum of the two. * s3tables: assert every attribute when rewriting an entry UpdateEntry writes the whole entry back from the snapshot the caller read, and its precondition only covers the keys the caller names. Both maintenance writers named one key, so a job status write could revert a maintenance configuration an operator had just disabled, turning an advisory write into a silent re-enable. Both now assert the entry's full attribute set, including the target key when absent so a concurrent create also fails the precondition. * s3tables: assert absent attributes when rewriting an entry The precondition covered the attributes present when the writer read the entry, so an attribute created between that read and the write was absent from it. A first-time PutTableMaintenanceConfiguration disabling a type therefore lands, passes the per-key checks, and is then deleted by the stale whole-entry write. Every attribute this package stores is now asserted, absent ones included. The metadata commit and planning index writers rewrite the same entries and had the same exposure, so both use the shared snapshot too. * iceberg: implement the auto compaction strategy auto was accepted, stored and read back, but left the worker on its own default, so a sorted table configured as auto was compacted with binpack. AWS defines auto as sorting tables that declare a sort order and bin-packing the rest. That needs the table metadata, so the choice is made where the rewrite plan is resolved: an unsorted table falls back to binpack rather than failing the way an explicit sort request does. * s3tables: validate the maintenance setting ranges PUT accepted zero, negative and oversized values for every numeric setting. The worker then ignores a non-positive value and saturates an oversized one, so the configuration read back was not the one that ran. AWS bounds all five to 1..2147483647, which is now enforced. The fields are pointers so an explicit zero is distinguishable from an omitted one and can be rejected rather than silently ignored. * s3tables: give every entry writer the same compare-and-swap updateExtendedAttribute asserted the entry's attributes, but the helpers behind the metadata, policy and tag handlers still wrote the whole entry unconditionally. Any of them could land on a stale snapshot and delete a maintenance configuration an operator had just written. They all share one read-modify-write loop now, so the precondition and the bounded retry apply wherever an entry is rewritten. * s3tables: move the maintenance configuration with a renamed table RenameTable carried the metadata, version, policy and tags to the new name but left the maintenance configuration and job status behind. A table with snapshot management disabled came back enabled under its new name, and the stale configuration stayed on the old name where a table created there would inherit it. The decoupled-delete cleanup left the same two attributes behind. * s3tables: accept every AWS partition in ARNs The route regexes and the ARN patterns both hardcoded arn:aws, so valid aws-cn and aws-us-gov ARNs never reached a handler. The router now shares the partition-tolerant prefix with the parser, and a generated ARN uses the partition its region belongs to so it parses back. * s3tables: generate ARNs in the region's partition The handler's own ARN generators still formatted arn:aws directly rather than going through the partition-aware builder, so a China or GovCloud deployment routed the request but then returned a commercial ARN and matched IAM policies against it. The round-trip test missed this because parsing accepts any partition, so it now asserts the prefix the region implies. * s3tables: complete the ARN partition table aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-* and eusc-* regions fell through to the commercial partition. * s3tables: do not let a rename swallow a concurrent maintenance write Rename copied the source attributes early and cleared the source at the end, so a Put landing in between missed the copy to the destination and was then deleted by the cleanup. It succeeded and vanished. The cleanup now clears the source only while it still holds exactly what was copied, and returns a conflict otherwise. Put checks the catalog identity inside the same conditional mutation, so it also cannot write to a name that a rename or delete has already soft-deleted.
817 lines
31 KiB
Go
817 lines
31 KiB
Go
package s3api
|
|
|
|
import (
|
|
"bytes"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
"net/http"
|
|
"net/url"
|
|
"strconv"
|
|
"strings"
|
|
|
|
"github.com/gorilla/mux"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/glog"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
|
|
"github.com/seaweedfs/seaweedfs/weed/s3api/s3tables"
|
|
)
|
|
|
|
// S3TablesApiServer wraps the S3 Tables handler with S3ApiServer's filer access
|
|
type S3TablesApiServer struct {
|
|
s3a *S3ApiServer
|
|
handler *s3tables.S3TablesHandler
|
|
}
|
|
|
|
// NewS3TablesApiServer creates a new S3 Tables API server
|
|
func NewS3TablesApiServer(s3a *S3ApiServer) *S3TablesApiServer {
|
|
return &S3TablesApiServer{
|
|
s3a: s3a,
|
|
handler: s3tables.NewS3TablesHandler(),
|
|
}
|
|
}
|
|
|
|
// SetRegion sets the AWS region for ARN generation
|
|
func (st *S3TablesApiServer) SetRegion(region string) {
|
|
st.handler.SetRegion(region)
|
|
}
|
|
|
|
// SetAccountID sets the AWS account ID for ARN generation
|
|
func (st *S3TablesApiServer) SetAccountID(accountID string) {
|
|
st.handler.SetAccountID(accountID)
|
|
}
|
|
|
|
// SetDefaultAllow sets whether to allow access by default
|
|
func (st *S3TablesApiServer) SetDefaultAllow(allow bool) {
|
|
st.handler.SetDefaultAllow(allow)
|
|
}
|
|
|
|
// SetIAMAuthorizer injects the IAM authorizer for S3 Tables IAM checks.
|
|
func (st *S3TablesApiServer) SetIAMAuthorizer(authorizer s3tables.IAMAuthorizer) {
|
|
st.handler.SetIAMAuthorizer(authorizer)
|
|
}
|
|
|
|
// S3TablesHandler handles S3 Tables API requests
|
|
func (st *S3TablesApiServer) S3TablesHandler(w http.ResponseWriter, r *http.Request) {
|
|
st.handler.HandleRequest(w, r, st)
|
|
}
|
|
|
|
// WithFilerClient implements the s3tables.FilerClient interface
|
|
func (st *S3TablesApiServer) WithFilerClient(streamingMode bool, fn func(filer_pb.SeaweedFilerClient) error) error {
|
|
return st.s3a.WithFilerClient(streamingMode, fn)
|
|
}
|
|
|
|
// registerS3TablesRoutes registers S3 Tables API routes
|
|
func (s3a *S3ApiServer) registerS3TablesRoutes(router *mux.Router) {
|
|
// Create S3 Tables handler
|
|
s3TablesApi := NewS3TablesApiServer(s3a)
|
|
if s3a.iam != nil && s3a.iam.iamIntegration != nil {
|
|
s3TablesApi.SetDefaultAllow(s3a.iam.iamIntegration.DefaultAllow())
|
|
if s3Integration, ok := s3a.iam.iamIntegration.(*S3IAMIntegration); ok && s3Integration.iamManager != nil {
|
|
s3TablesApi.SetIAMAuthorizer(s3Integration.iamManager)
|
|
}
|
|
} else {
|
|
// If IAM is not configured, allow all access by default
|
|
s3TablesApi.SetDefaultAllow(true)
|
|
}
|
|
|
|
// Regex for S3 Tables Bucket ARN. The partition is not fixed to "aws":
|
|
// aws-cn and aws-us-gov ARNs are valid and have to reach the handlers.
|
|
const tableBucketARNRegex = s3tables.ARNPrefixPatternStr + ":[^/:]*:[^/:]*:bucket/[^/]+"
|
|
|
|
// REST-style S3 Tables API routes (used by AWS CLI)
|
|
targetMatcher := func(r *http.Request, rm *mux.RouteMatch) bool {
|
|
return strings.HasPrefix(r.Header.Get("X-Amz-Target"), "S3Tables.")
|
|
}
|
|
// serviceMatcher gates every S3 Tables route so it only matches when the
|
|
// request is genuinely targeting the s3tables service. The bare paths
|
|
// (/buckets, /get-table) collide with regular S3 buckets of the same
|
|
// name; the ARN-bearing paths (/buckets/<arn>, /namespaces/<arn>, ...)
|
|
// could collide with object keys that look like S3 Tables ARNs inside a
|
|
// bucket named "buckets", "namespaces", "tables", or "tag". A single
|
|
// matcher applied to every route closes both classes of collision.
|
|
serviceMatcher := func(r *http.Request, rm *mux.RouteMatch) bool {
|
|
return isS3TablesSignedRequest(r)
|
|
}
|
|
router.Methods(http.MethodPost).Path("/").MatcherFunc(targetMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.S3TablesHandler), "S3Tables-Target"))
|
|
router.Methods(http.MethodPut).Path("/buckets").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("CreateTableBucket", buildCreateTableBucketRequest)), "S3Tables-CreateTableBucket"))
|
|
router.Methods(http.MethodGet).Path("/buckets").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("ListTableBuckets", buildListTableBucketsRequest)), "S3Tables-ListTableBuckets"))
|
|
router.Methods(http.MethodGet).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTableBucket", buildTableBucketArnRequest)), "S3Tables-GetTableBucket"))
|
|
router.Methods(http.MethodDelete).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("DeleteTableBucket", buildDeleteTableBucketRequest)), "S3Tables-DeleteTableBucket"))
|
|
router.Methods(http.MethodPut).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}/policy").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("PutTableBucketPolicy", buildPutTableBucketPolicyRequest)), "S3Tables-PutTableBucketPolicy"))
|
|
router.Methods(http.MethodGet).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}/policy").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTableBucketPolicy", buildGetTableBucketPolicyRequest)), "S3Tables-GetTableBucketPolicy"))
|
|
router.Methods(http.MethodDelete).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}/policy").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("DeleteTableBucketPolicy", buildDeleteTableBucketPolicyRequest)), "S3Tables-DeleteTableBucketPolicy"))
|
|
|
|
router.Methods(http.MethodPut).Path("/namespaces/{tableBucketARN:" + tableBucketARNRegex + "}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("CreateNamespace", buildCreateNamespaceRequest)), "S3Tables-CreateNamespace"))
|
|
router.Methods(http.MethodGet).Path("/namespaces/{tableBucketARN:" + tableBucketARNRegex + "}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("ListNamespaces", buildListNamespacesRequest)), "S3Tables-ListNamespaces"))
|
|
router.Methods(http.MethodGet).Path("/namespaces/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetNamespace", buildGetNamespaceRequest)), "S3Tables-GetNamespace"))
|
|
router.Methods(http.MethodDelete).Path("/namespaces/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("DeleteNamespace", buildDeleteNamespaceRequest)), "S3Tables-DeleteNamespace"))
|
|
|
|
router.Methods(http.MethodPut).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("CreateTable", buildCreateTableRequest)), "S3Tables-CreateTable"))
|
|
router.Methods(http.MethodGet).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("ListTables", buildListTablesRequest)), "S3Tables-ListTables"))
|
|
router.Methods(http.MethodDelete).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("DeleteTable", buildDeleteTableRequest)), "S3Tables-DeleteTable"))
|
|
|
|
router.Methods(http.MethodPut).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}/policy").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("PutTablePolicy", buildPutTablePolicyRequest)), "S3Tables-PutTablePolicy"))
|
|
router.Methods(http.MethodGet).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}/policy").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTablePolicy", buildGetTablePolicyRequest)), "S3Tables-GetTablePolicy"))
|
|
router.Methods(http.MethodDelete).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}/policy").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("DeleteTablePolicy", buildDeleteTablePolicyRequest)), "S3Tables-DeleteTablePolicy"))
|
|
|
|
router.Methods(http.MethodPut).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}/maintenance/{type}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("PutTableBucketMaintenanceConfiguration", buildPutTableBucketMaintenanceConfigurationRequest)), "S3Tables-PutTableBucketMaintenanceConfiguration"))
|
|
router.Methods(http.MethodGet).Path("/buckets/{tableBucketARN:" + tableBucketARNRegex + "}/maintenance").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTableBucketMaintenanceConfiguration", buildGetTableBucketMaintenanceConfigurationRequest)), "S3Tables-GetTableBucketMaintenanceConfiguration"))
|
|
router.Methods(http.MethodPut).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}/maintenance/{type}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("PutTableMaintenanceConfiguration", buildPutTableMaintenanceConfigurationRequest)), "S3Tables-PutTableMaintenanceConfiguration"))
|
|
router.Methods(http.MethodGet).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}/maintenance").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTableMaintenanceConfiguration", buildGetTableMaintenanceConfigurationRequest)), "S3Tables-GetTableMaintenanceConfiguration"))
|
|
router.Methods(http.MethodGet).Path("/tables/{tableBucketARN:" + tableBucketARNRegex + "}/{namespace}/{name}/maintenance-job-status").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTableMaintenanceJobStatus", buildGetTableMaintenanceJobStatusRequest)), "S3Tables-GetTableMaintenanceJobStatus"))
|
|
|
|
router.Methods(http.MethodPost).Path("/tag/{resourceArn:" + s3tables.ARNPrefixPatternStr + ":.*}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("TagResource", buildTagResourceRequest)), "S3Tables-TagResource"))
|
|
router.Methods(http.MethodGet).Path("/tag/{resourceArn:" + s3tables.ARNPrefixPatternStr + ":.*}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("ListTagsForResource", buildListTagsForResourceRequest)), "S3Tables-ListTagsForResource"))
|
|
router.Methods(http.MethodDelete).Path("/tag/{resourceArn:" + s3tables.ARNPrefixPatternStr + ":.*}").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("UntagResource", buildUntagResourceRequest)), "S3Tables-UntagResource"))
|
|
|
|
router.Methods(http.MethodGet).Path("/get-table").MatcherFunc(serviceMatcher).
|
|
HandlerFunc(track(s3a.authenticateS3Tables(s3TablesApi.handleRestOperation("GetTable", buildGetTableRequest)), "S3Tables-GetTable"))
|
|
|
|
glog.V(1).Infof("S3 Tables API enabled")
|
|
}
|
|
|
|
type s3tablesRequestBuilder func(r *http.Request) (interface{}, error)
|
|
|
|
func (st *S3TablesApiServer) handleRestOperation(operation string, builder s3tablesRequestBuilder) http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
payload, err := builder(r)
|
|
if err != nil {
|
|
writeS3TablesError(w, http.StatusBadRequest, s3tables.ErrCodeInvalidRequest, err.Error())
|
|
return
|
|
}
|
|
if err := setS3TablesRequestBody(r, payload); err != nil {
|
|
writeS3TablesError(w, http.StatusInternalServerError, s3tables.ErrCodeInternalError, err.Error())
|
|
return
|
|
}
|
|
r.Header.Set("X-Amz-Target", "S3Tables."+operation)
|
|
st.S3TablesHandler(w, r)
|
|
}
|
|
}
|
|
|
|
func setS3TablesRequestBody(r *http.Request, payload interface{}) error {
|
|
body, err := json.Marshal(payload)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
r.Body = io.NopCloser(bytes.NewReader(body))
|
|
r.ContentLength = int64(len(body))
|
|
r.Header.Set("Content-Type", "application/x-amz-json-1.1")
|
|
return nil
|
|
}
|
|
|
|
func readS3TablesJSONBody(r *http.Request, v interface{}) error {
|
|
if r.Body == nil {
|
|
return nil
|
|
}
|
|
defer r.Body.Close()
|
|
const maxRequestBodySize = 10 * 1024 * 1024
|
|
if r.ContentLength > maxRequestBodySize {
|
|
return fmt.Errorf("request body too large: exceeds maximum size of %d bytes", maxRequestBodySize)
|
|
}
|
|
limitedReader := io.LimitReader(r.Body, maxRequestBodySize+1)
|
|
body, err := io.ReadAll(limitedReader)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if len(body) > maxRequestBodySize {
|
|
return fmt.Errorf("request body too large: exceeds maximum size of %d bytes", maxRequestBodySize)
|
|
}
|
|
if len(bytes.TrimSpace(body)) == 0 {
|
|
return nil
|
|
}
|
|
return json.Unmarshal(body, v)
|
|
}
|
|
|
|
func writeS3TablesError(w http.ResponseWriter, status int, code, message string) {
|
|
w.Header().Set("Content-Type", "application/x-amz-json-1.1")
|
|
w.WriteHeader(status)
|
|
errorResponse := map[string]interface{}{
|
|
"__type": code,
|
|
"message": message,
|
|
}
|
|
if err := json.NewEncoder(w).Encode(errorResponse); err != nil {
|
|
glog.Errorf("failed to encode S3Tables error response (status=%d, code=%s, message=%q): %v", status, code, message, err)
|
|
}
|
|
}
|
|
|
|
func getDecodedPathParam(r *http.Request, name string) (string, error) {
|
|
value := mux.Vars(r)[name]
|
|
if value == "" {
|
|
return "", nil
|
|
}
|
|
decoded, err := url.PathUnescape(value)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
if decoded == ".." || strings.Contains(decoded, "../") || strings.Contains(decoded, `..\`) || strings.Contains(decoded, "\x00") {
|
|
return "", fmt.Errorf("invalid path parameter %s", name)
|
|
}
|
|
return decoded, nil
|
|
}
|
|
|
|
func buildTableBucketRequestWithARN(r *http.Request, constructor func(string) interface{}) (interface{}, error) {
|
|
arn, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if arn == "" {
|
|
return nil, fmt.Errorf("tableBucketARN is required")
|
|
}
|
|
if _, err := s3tables.ParseBucketNameFromARN(arn); err != nil {
|
|
return nil, err
|
|
}
|
|
return constructor(arn), nil
|
|
}
|
|
|
|
func parseOptionalIntParam(r *http.Request, name string) (int, error) {
|
|
value := r.URL.Query().Get(name)
|
|
if value == "" {
|
|
return 0, nil
|
|
}
|
|
parsed, err := strconv.Atoi(value)
|
|
if err != nil {
|
|
return 0, fmt.Errorf("%s must be an integer", name)
|
|
}
|
|
if parsed <= 0 {
|
|
return 0, fmt.Errorf("%s must be a positive integer", name)
|
|
}
|
|
return parsed, nil
|
|
}
|
|
|
|
func parseOptionalNamespace(r *http.Request, name string) ([]string, error) {
|
|
value := r.URL.Query().Get(name)
|
|
if value == "" {
|
|
return nil, nil
|
|
}
|
|
parts, err := s3tables.ParseNamespace(value)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("invalid %s: %w", name, err)
|
|
}
|
|
return parts, nil
|
|
}
|
|
|
|
func parseRequiredNamespacePathParam(r *http.Request, name string) ([]string, error) {
|
|
value, err := getDecodedPathParam(r, name)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if value == "" {
|
|
return nil, fmt.Errorf("%s is required", name)
|
|
}
|
|
return s3tables.ParseNamespace(value)
|
|
}
|
|
|
|
// parseTagKeys handles tag key parsing from query parameters.
|
|
// If a single value contains commas, it is split into multiple keys (e.g., "key1,key2,key3").
|
|
// Otherwise, multiple query values are returned as-is.
|
|
func parseTagKeys(values []string) []string {
|
|
if len(values) == 0 {
|
|
return nil
|
|
}
|
|
out := make([]string, 0, len(values))
|
|
for _, value := range values {
|
|
for _, part := range strings.Split(value, ",") {
|
|
part = strings.TrimSpace(part)
|
|
if part != "" {
|
|
out = append(out, part)
|
|
}
|
|
}
|
|
}
|
|
if len(out) == 0 {
|
|
return nil
|
|
}
|
|
return out
|
|
}
|
|
|
|
func buildCreateTableBucketRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.CreateTableBucketRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
return &req, nil
|
|
}
|
|
|
|
func buildListTableBucketsRequest(r *http.Request) (interface{}, error) {
|
|
maxBuckets, err := parseOptionalIntParam(r, "maxBuckets")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.ListTableBucketsRequest{
|
|
Prefix: r.URL.Query().Get("prefix"),
|
|
ContinuationToken: r.URL.Query().Get("continuationToken"),
|
|
MaxBuckets: maxBuckets,
|
|
}, nil
|
|
}
|
|
|
|
func buildTableBucketArnRequest(r *http.Request) (interface{}, error) {
|
|
return buildTableBucketRequestWithARN(r, func(arn string) interface{} {
|
|
return &s3tables.GetTableBucketRequest{TableBucketARN: arn}
|
|
})
|
|
}
|
|
|
|
func buildDeleteTableBucketRequest(r *http.Request) (interface{}, error) {
|
|
return buildTableBucketRequestWithARN(r, func(arn string) interface{} {
|
|
return &s3tables.DeleteTableBucketRequest{TableBucketARN: arn}
|
|
})
|
|
}
|
|
|
|
func buildPutTableBucketPolicyRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.PutTableBucketPolicyRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
req.TableBucketARN = tableBucketARN
|
|
return &req, nil
|
|
}
|
|
|
|
func buildGetTableBucketPolicyRequest(r *http.Request) (interface{}, error) {
|
|
return buildTableBucketRequestWithARN(r, func(arn string) interface{} {
|
|
return &s3tables.GetTableBucketPolicyRequest{TableBucketARN: arn}
|
|
})
|
|
}
|
|
|
|
func buildDeleteTableBucketPolicyRequest(r *http.Request) (interface{}, error) {
|
|
return buildTableBucketRequestWithARN(r, func(arn string) interface{} {
|
|
return &s3tables.DeleteTableBucketPolicyRequest{TableBucketARN: arn}
|
|
})
|
|
}
|
|
|
|
func buildCreateNamespaceRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.CreateNamespaceRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
req.TableBucketARN = tableBucketARN
|
|
return &req, nil
|
|
}
|
|
|
|
func buildListNamespacesRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
maxNamespaces, err := parseOptionalIntParam(r, "maxNamespaces")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.ListNamespacesRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Prefix: r.URL.Query().Get("prefix"),
|
|
ContinuationToken: r.URL.Query().Get("continuationToken"),
|
|
MaxNamespaces: maxNamespaces,
|
|
}, nil
|
|
}
|
|
|
|
func buildGetNamespaceRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.GetNamespaceRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
}, nil
|
|
}
|
|
|
|
func buildDeleteNamespaceRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.DeleteNamespaceRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
}, nil
|
|
}
|
|
|
|
func buildCreateTableRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.CreateTableRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
req.TableBucketARN = tableBucketARN
|
|
req.Namespace = namespace
|
|
return &req, nil
|
|
}
|
|
|
|
func buildListTablesRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseOptionalNamespace(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
maxTables, err := parseOptionalIntParam(r, "maxTables")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.ListTablesRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
Prefix: r.URL.Query().Get("prefix"),
|
|
ContinuationToken: r.URL.Query().Get("continuationToken"),
|
|
MaxTables: maxTables,
|
|
}, nil
|
|
}
|
|
|
|
func buildGetTableRequest(r *http.Request) (interface{}, error) {
|
|
query := r.URL.Query()
|
|
tableARN := query.Get("tableArn")
|
|
req := &s3tables.GetTableRequest{
|
|
TableARN: tableARN,
|
|
}
|
|
if tableARN == "" {
|
|
req.TableBucketARN = query.Get("tableBucketARN")
|
|
namespace, err := parseOptionalNamespace(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
req.Namespace = namespace
|
|
req.Name = query.Get("name")
|
|
if req.TableBucketARN == "" || len(req.Namespace) == 0 || req.Name == "" {
|
|
return nil, fmt.Errorf("either tableArn or (tableBucketARN, namespace, name) must be provided")
|
|
}
|
|
}
|
|
return req, nil
|
|
}
|
|
|
|
func buildDeleteTableRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
name, err := getDecodedPathParam(r, "name")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if name == "" {
|
|
return nil, fmt.Errorf("name is required")
|
|
}
|
|
if _, err := s3tables.ValidateTableName(name); err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.DeleteTableRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
Name: name,
|
|
VersionToken: r.URL.Query().Get("versionToken"),
|
|
}, nil
|
|
}
|
|
|
|
func buildPutTablePolicyRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.PutTablePolicyRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
name, err := getDecodedPathParam(r, "name")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if name == "" {
|
|
return nil, fmt.Errorf("name is required")
|
|
}
|
|
if _, err := s3tables.ValidateTableName(name); err != nil {
|
|
return nil, err
|
|
}
|
|
req.TableBucketARN = tableBucketARN
|
|
req.Namespace = namespace
|
|
req.Name = name
|
|
return &req, nil
|
|
}
|
|
|
|
func buildGetTablePolicyRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
name, err := getDecodedPathParam(r, "name")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if name == "" {
|
|
return nil, fmt.Errorf("name is required")
|
|
}
|
|
if _, err := s3tables.ValidateTableName(name); err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.GetTablePolicyRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
Name: name,
|
|
}, nil
|
|
}
|
|
|
|
func buildDeleteTablePolicyRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
namespace, err := parseRequiredNamespacePathParam(r, "namespace")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
name, err := getDecodedPathParam(r, "name")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if name == "" {
|
|
return nil, fmt.Errorf("name is required")
|
|
}
|
|
if _, err := s3tables.ValidateTableName(name); err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.DeleteTablePolicyRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
Name: name,
|
|
}, nil
|
|
}
|
|
|
|
func buildTagResourceRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.TagResourceRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
resourceARN, err := getDecodedPathParam(r, "resourceArn")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if resourceARN == "" {
|
|
return nil, fmt.Errorf("resourceArn is required")
|
|
}
|
|
req.ResourceARN = resourceARN
|
|
return &req, nil
|
|
}
|
|
|
|
func buildListTagsForResourceRequest(r *http.Request) (interface{}, error) {
|
|
resourceARN, err := getDecodedPathParam(r, "resourceArn")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if resourceARN == "" {
|
|
return nil, fmt.Errorf("resourceArn is required")
|
|
}
|
|
return &s3tables.ListTagsForResourceRequest{
|
|
ResourceARN: resourceARN,
|
|
}, nil
|
|
}
|
|
|
|
func buildUntagResourceRequest(r *http.Request) (interface{}, error) {
|
|
resourceARN, err := getDecodedPathParam(r, "resourceArn")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if resourceARN == "" {
|
|
return nil, fmt.Errorf("resourceArn is required")
|
|
}
|
|
tagKeys := parseTagKeys(r.URL.Query()["tagKeys"])
|
|
if len(tagKeys) == 0 {
|
|
return nil, fmt.Errorf("tagKeys is required for %s", resourceARN)
|
|
}
|
|
return &s3tables.UntagResourceRequest{
|
|
ResourceARN: resourceARN,
|
|
TagKeys: tagKeys,
|
|
}, nil
|
|
}
|
|
|
|
// isS3TablesSignedRequest reports whether the request is targeting the
|
|
// S3 Tables service. The signal is the AWS V4 credential scope, which
|
|
// names SERVICE=s3tables for S3 Tables SDKs and SERVICE=s3 for regular
|
|
// S3 SDKs. The credential scope appears in the Authorization header
|
|
// (Credential=AK/DATE/REGION/SERVICE/aws4_request) for signed requests
|
|
// and in the X-Amz-Credential query parameter for presigned requests.
|
|
//
|
|
// The credential scope is the only acceptable signal: a content-type-
|
|
// based fallback would let an anonymous regular-S3 request (e.g. a
|
|
// PutObject with body type application/x-amz-json-1.1) sneak through
|
|
// to an S3 Tables route whenever the object key is shaped like an
|
|
// S3 Tables ARN. Clients that genuinely target S3 Tables — including
|
|
// internal test harnesses running against a default-allow server —
|
|
// must sign with SERVICE=s3tables.
|
|
func isS3TablesSignedRequest(r *http.Request) bool {
|
|
scope := extractCredentialScope(r)
|
|
// Credential scope is AK/DATE/REGION/SERVICE/aws4_request. Slashes
|
|
// do not appear inside any other component (access keys are
|
|
// alphanumeric), so /s3tables/ matches iff SERVICE is exactly
|
|
// s3tables.
|
|
return scope != "" && strings.Contains(scope, "/s3tables/")
|
|
}
|
|
|
|
// extractCredentialScope returns the raw credential value from either the
|
|
// Authorization header or the X-Amz-Credential query parameter, without the
|
|
// "Credential=" prefix. Returns the empty string when neither is present.
|
|
func extractCredentialScope(r *http.Request) string {
|
|
if auth := r.Header.Get("Authorization"); auth != "" {
|
|
idx := strings.Index(auth, "Credential=")
|
|
if idx >= 0 {
|
|
tail := auth[idx+len("Credential="):]
|
|
if comma := strings.IndexByte(tail, ','); comma >= 0 {
|
|
tail = tail[:comma]
|
|
}
|
|
return strings.TrimSpace(tail)
|
|
}
|
|
}
|
|
if cred := r.URL.Query().Get("X-Amz-Credential"); cred != "" {
|
|
return cred
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// authenticateS3Tables wraps the handler with IAM authentication using AuthSignatureOnly
|
|
// This authenticates the request but delegates authorization to the S3 Tables handler
|
|
// which performs granular permission checks based on the specific operation.
|
|
func (s3a *S3ApiServer) authenticateS3Tables(f http.HandlerFunc) http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
glog.V(2).Infof("S3Tables: authenticateS3Tables called, iam.isEnabled()=%t", s3a.iam.isEnabled())
|
|
if !s3a.iam.isEnabled() {
|
|
f(w, r)
|
|
return
|
|
}
|
|
|
|
// Use AuthSignatureOnly to authenticate the request without authorizing specific actions
|
|
identity, errCode := s3a.iam.AuthSignatureOnly(r)
|
|
if errCode != s3err.ErrNone {
|
|
// If IAM is enabled but DefaultAllow is true, we can proceed even if unauthenticated
|
|
// authorization checks in handlers will then use DefaultAllow logic.
|
|
if s3a.iam.iamIntegration != nil && s3a.iam.iamIntegration.DefaultAllow() {
|
|
glog.V(2).Infof("S3Tables: AuthSignatureOnly failed (%v), but DefaultAllow is true, proceeding", errCode)
|
|
} else {
|
|
glog.Errorf("S3Tables: AuthSignatureOnly failed: %v", errCode)
|
|
s3err.WriteErrorResponse(w, r, errCode)
|
|
return
|
|
}
|
|
}
|
|
|
|
// Store the authenticated identity in request context
|
|
if identity != nil && identity.Name != "" {
|
|
glog.V(2).Infof("S3Tables: authenticated identity Name=%s Account.Id=%s", identity.Name, identity.Account.Id)
|
|
r = r.WithContext(recordIdentityInContext(r, identity))
|
|
} else {
|
|
glog.V(2).Infof("S3Tables: authenticated identity is nil or empty name")
|
|
}
|
|
|
|
f(w, r)
|
|
}
|
|
}
|
|
|
|
// maintenanceTableTarget pulls the bucket/namespace/table triple every
|
|
// table-scoped maintenance route carries in its path.
|
|
func maintenanceTableTarget(r *http.Request) (tableBucketARN string, namespace []string, name string, err error) {
|
|
if tableBucketARN, err = getDecodedPathParam(r, "tableBucketARN"); err != nil {
|
|
return "", nil, "", err
|
|
}
|
|
if namespace, err = parseRequiredNamespacePathParam(r, "namespace"); err != nil {
|
|
return "", nil, "", err
|
|
}
|
|
if name, err = getDecodedPathParam(r, "name"); err != nil {
|
|
return "", nil, "", err
|
|
}
|
|
if name == "" {
|
|
return "", nil, "", fmt.Errorf("name is required")
|
|
}
|
|
if _, err = s3tables.ValidateTableName(name); err != nil {
|
|
return "", nil, "", err
|
|
}
|
|
return tableBucketARN, namespace, name, nil
|
|
}
|
|
|
|
func buildPutTableBucketMaintenanceConfigurationRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.PutTableBucketMaintenanceConfigurationRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
maintenanceType, err := getDecodedPathParam(r, "type")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
req.TableBucketARN = tableBucketARN
|
|
req.Type = maintenanceType
|
|
return &req, nil
|
|
}
|
|
|
|
func buildGetTableBucketMaintenanceConfigurationRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, err := getDecodedPathParam(r, "tableBucketARN")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.GetTableBucketMaintenanceConfigurationRequest{TableBucketARN: tableBucketARN}, nil
|
|
}
|
|
|
|
func buildPutTableMaintenanceConfigurationRequest(r *http.Request) (interface{}, error) {
|
|
var req s3tables.PutTableMaintenanceConfigurationRequest
|
|
if err := readS3TablesJSONBody(r, &req); err != nil {
|
|
return nil, err
|
|
}
|
|
tableBucketARN, namespace, name, err := maintenanceTableTarget(r)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
maintenanceType, err := getDecodedPathParam(r, "type")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
req.TableBucketARN = tableBucketARN
|
|
req.Namespace = namespace
|
|
req.Name = name
|
|
req.Type = maintenanceType
|
|
return &req, nil
|
|
}
|
|
|
|
func buildGetTableMaintenanceConfigurationRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, namespace, name, err := maintenanceTableTarget(r)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.GetTableMaintenanceConfigurationRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
Name: name,
|
|
}, nil
|
|
}
|
|
|
|
func buildGetTableMaintenanceJobStatusRequest(r *http.Request) (interface{}, error) {
|
|
tableBucketARN, namespace, name, err := maintenanceTableTarget(r)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return &s3tables.GetTableMaintenanceJobStatusRequest{
|
|
TableBucketARN: tableBucketARN,
|
|
Namespace: namespace,
|
|
Name: name,
|
|
}, nil
|
|
}
|