Let captureHTTPClient answer with a configurable header and body, and
use it for the GetObjectAttributes tests instead of a separate stub
client that repeated the same client setup.
The test was skipped for the gateway with a link to #925, which is fixed
here. The gateway already returns InvalidArgument for an unknown
attribute, so the test passes, and the skip would point to a closed
issue once #925 is closed.
S3Proxy.GetObjectAttributes dropped VersionId and DeleteMarker from the
upstream response, so the gateway never sent x-amz-version-id or
x-amz-delete-marker for objects behind an s3 backend. Copy them into the
response, and add tests for how the upstream response is mapped,
including the parts count.
ListObjects and ListObjectsV2 ignored the encoding-type query parameter.
Any value was accepted, and with encoding-type=url the response carried
neither an EncodingType element nor encoded key names.
S3 accepts only "url" and answers anything else with InvalidArgument
("Invalid Encoding Method specified in Request", argument encoding-type).
With "url" it sets EncodingType and encodes the key names in the
listing as a query component, "a+ b.txt" becoming "a%2B+b.txt", with
"/" left as is: Key, Prefix, Delimiter and CommonPrefixes in both
operations, Marker and NextMarker in ListObjects, and StartAfter in
ListObjectsV2. The continuation tokens are opaque and are not encoded.
The encoding is applied to a copy of the backend result in the
controllers, so it covers every backend.
Fixes#1985Fixes#1984
Fixes#2511
`CreateMultipartUpload` in the posix backend creates the upload directory under `.sgwtmp/multipart` with `os.MkdirAll` and returned any failure wrapped as `create upload temp dir`, so a full filesystem quota or a full disk came back as a 500 `InternalError` (which SDKs also retry) instead of the error the other write paths return. `PutObject`, `UploadPart`, `UploadPartCopy` and `CompleteMultipartUpload` already map these errors.
`syscall.EDQUOT` from the directory creation now returns `QuotaExceeded` (403) and `syscall.ENOSPC` returns `InsufficientStorage` (507), the same mapping the rest of the posix write paths use.
The posix backend stages multipart uploads and temporary files inside the bucket directory under `.sgwtmp`, but nothing stopped an object key from resolving there. A client with access to the bucket could read the parts of other clients' in-progress uploads with `GetObject`/`HeadObject`, copy them out with `CopyObject`, overwrite or delete them with `PutObject`/`DeleteObject` so the upload failed on completion with `InvalidPart`, and store data that never shows up in listings. Spellings such as `./.sgwtmp/...`, `x/../.sgwtmp/...`, a leading `/` and, on case-insensitive file systems, `.SGWTMP/...` reached the same files.
Add the non-AWS `ReservedObjectKey` error (400) and return it whenever an object key resolves to `.sgwtmp` or a path inside it. The key is resolved the same way `ObjectPath` resolves it, and its first path element is compared case-insensitively. The check runs after the bucket existence check in `PutObject`, `GetObject`, `HeadObject`, `GetObjectAttributes`, `DeleteObject`, `DeleteObjects` (reported per key), `CopyObject` (source and destination), the `UploadPartCopy` source, `CreateMultipartUpload`, `CompleteMultipartUpload`, object tagging, legal hold and retention. The exported `ValidateObjectKey` covers the ScoutFS `GetObject` and `RestoreObject` entry points. `UploadPart`, `ListParts` and `AbortMultipartUpload` only hash the key and stay unchanged, so uploads created with a reserved key before this fix can still be aborted.
`ListObjects` and `ListObjectsV2` also listed the staged parts when the prefix named a directory inside `.sgwtmp`, because the prefix fast path in `quickWalk` compared only the last element of the prefix directory against the skipped directories. `inSkipDir` now checks every element, ignoring case.
The controller now passes the requested attributes to the backend in
ObjectAttributes, so posix and azure only do the extra HeadObject for the
parts count when ObjectParts was asked for. This also lets the s3 backend
reach the upstream service, which rejected the request before because the
field was missing.
The s3 backend now copies TotalPartsCount into PartsCount, leaves out
ObjectParts when the upstream response has none, and sets the part number
markers, which were dropped by inverted strconv.Atoi error checks.
Fixes#2513
On Windows the posix backend stored a key as `bucket/key` verbatim, but Windows rejects `" * : < > ? \ |` and control characters in file names, so such a PUT failed with `ERROR_INVALID_NAME`, which `isErrNameTooLong` reported as a misleading `KeyTooLongError`. The same mapping did worse elsewhere: `\` acted as a path separator, so `a\b` aliased `a/b` and `..\..\x` resolved outside the bucket because `IsObjectNameValid` only splits on `/`; `a:b` wrote an NTFS alternate data stream; Win32 strips a trailing dot or space, so `foo.` overwrote `foo`; and device names such as `CON` or `nul.txt` opened the device and silently discarded the data.
With `encodeKeys`, enabled by default only on Windows, `encodeKey` stores each key segment under a name Windows accepts: every invalid character, a trailing dot or space, and the first letter of a device name are stored as the private use character `U+F000` plus the character code, the convention Cygwin and WSL use for invalid characters. A key character in `U+F000`-`U+F07F` is stored after `U+F000`, so every stored name belongs to exactly one key. Device names are checked with `filepath.IsLocal`, which asks the system, so `CON.txt` stays a plain file on Windows 11 where it is valid. Keys that need no encoding keep their names, so existing data is unaffected, and nothing changes on other platforms.
Public backend methods keep taking raw keys and encode them once for every path, metadata, version-hash and publish-lock use, while responses, errors and the multipart `objname` attribute keep the raw key and multipart upload directories keep hashing it. Listings walk `bucketFS`, which decodes the entry names, sorts them in key byte order to match S3 and skips names no key is stored under. `HeadObject` passes the raw key to `GetObjectLegalHold` and `GetObjectRetention` to avoid encoding twice, and `NormalizeObjectKey` resolves the encoded key, so policy resources see `a\b` as one literal key.
The comment claimed the short read's error is carried to the next call. It
is not: it is returned when nothing was read, and the next read of the
connection produces it again for both EOF and an expired deadline.
Fixes#2261
A client that sent http:// to a port serving TLS got an empty reply (curl
exit code 52) and no hint about what it had done wrong. The handshake fails
on the first bytes -- crypto/tls reports that they aren't a TLS record -- and
the server, which just sees a failed read, closes the connection. It was an
accident, not a refusal: nothing in the path ever writes a response.
The listener `NewMultiAddrTLSListener` builds for each endpoint now holds the
first five bytes back from the handshake. A TLS ClientHello starts with a
record type byte (0x16 for TLS 1.0 through 1.3), so a request line there means
the client spoke plaintext HTTP, and it gets a 400 that says so. The response
is plain text rather than an S3 error: one listener backs the S3, admin, IAM,
website and web UI servers, whose error bodies are XML, HTML and JSON, and none
of them has seen the request. A real TLS client's bytes are replayed untouched,
so the handshake is unaffected.
The reply half-closes and drains briefly before closing, since closing with the
rest of the request unread turns the close into a RST that can beat the
response to the client.
Covered by internal/netutil/tls_plaintext_http_test.go: a real HTTP client and a
raw absolute-form request line both get the 400, a HEAD gets it without a body,
and a tls.Dial client still completes the handshake and round-trips data.
GetObjectAttributes never returned ObjectParts, so a request for it on
an object completed from a multipart upload got no PartsCount. S3
returns <ObjectParts><PartsCount>N</PartsCount></ObjectParts> for these
objects and leaves ObjectParts out for objects that weren't uploaded in
parts.
The posix and azure backends already keep the part layout of completed
uploads, and HeadObject reports the parts count when a part number is
requested. GetObjectAttributes now uses that to fill ObjectParts. The
ObjectParts fields that aren't set are left out of the response, as S3
does.
Fixes#925
The api layer validates a client's LocationConstraint against the gateway
region, but the bucket is created in the s3 backend's own region
(--region / VGW_S3_REGION). They are separate settings, so passing the
validated client constraint through to the backend only worked when the two
regions matched. With the gateway on the default us-east-1 and the backend
on eu-west-1 the client sends no constraint at all, and the backend still
answered
IllegalLocationConstraintException: The unspecified location constraint is
incompatible for the region specific endpoint this request was sent to.
which is the error in #2490.
Set the constraint in S3Proxy.CreateBucket from the backend client's region
instead, next to the existing empty-configuration drop. A backend on
us-east-1 still gets no constraint, and the api layer's rejection of a
client constraint that disagrees with the gateway region is unchanged.
Fixes#2490
A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
DeleteObjects dropped the ETag, Size and LastModifiedTime of each object
when it called DeleteObject, so an object with a stale ETag was still
deleted. The posix and azure backends now forward them to DeleteObject.
EvaluateObjectDeletePreconditions trims the quotes an SDK sends around
the ETag and treats "*" as a match for any ETag.
The REST ETag mismatch test no longer skips.
Taken from #2485, without its posix and integration tests.
Refs #2112
Fixes#2499
An empty tag key returned `InvalidTag` without the `TagKey` element because the field was `omitempty`. S3 always sends `<TagKey>`, empty for an empty key, so it is now always written.
`PutBucketTagging`, `PutObjectTagging`, the POST object `tagging` field and `CreateBucket` tags now share one validator, `backend.ParseTagSet`, which matches S3: a duplicate key error carries `TagKey`, a `Tag` missing its `Key` or `Value` element is `MalformedXML` and is checked across the whole document first, every tag is validated before the tag count limit, and the 128/256 length limits count UTF-16 code units instead of bytes. The duplicate tag regex in `s3api/utils` is removed.
For the `x-amz-tagging` header, the length limits now apply to the URL-decoded key and value, the `InvalidArgument` error echoes the whole header as `ArgumentValue` instead of the unparsed remainder, and more than 10 tags is rejected with `BadRequest`.
Fixes#2489
An unsupported `Authorization` algorithm returned only the algorithm in `ArgumentValue`, while AWS S3 echoes the whole header as received. `sigv4auth.ParseAuthorization` now carries the full header in `ParseError.Input`, which the S3 error mapper returns. Also drop the trailing period from the `InvalidArgAuthHeader` message to match AWS.
`govulncheck` reported vulnerabilities in `golang.org/x/net` `v0.59.0`, so it is bumped to `v0.60.0`, which fixes them.
Go `1.27.2` changed the compiler export data format, and the latest staticcheck release can't read the new format yet. The `staticcheck` job is pinned to Go `1.27.1`, with `install-go: false` so the action doesn't install Go `stable` over the pinned version.
This pin is temporary until dominikh/go-tools#1834, which adds Go `1.27.2` compatibility, is merged and released. After that, the job should go back to `stable`.
The posix backend publishes an object by linking or renaming a temp file
into place, but never flushes it. An acknowledged PUT can therefore be
lost, or come back empty, after a crash or power loss, depending on the
filesystem's writeback interval.
With --fsync (VGW_FSYNC), the backend fsyncs the temp file, which covers
its data and xattr metadata, before publishing it. After publishing, it
fsyncs the object's directory and each parent up to the bucket, so that
the new entry and any directories created for it are durable. A failed
flush fails the request. The option is off by default, so existing
deployments are unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The azure backend tracks zero-byte parts separately and reports its own
block id as the ETag when the client ETag doesn't match, so the test
failed against azurite. Upload some data so the part is a regular block,
and send the ETag unquoted so both backends echo it back as sent.
Once a bucket had any policy, `verifyResourceAccess` evaluated the policy alone and ignored the bucket ACL, so a non-root bucket owner kept only what the policy explicitly allowed. A Deny-only policy (e.g. Deny `*` `s3:GetObject` on one prefix) therefore locked the owner out of every operation, including `DeleteBucketPolicy`, and only root/admin could recover. ACL grants also authorized every bucket operation through the controller's `AclPermission`, so a non-owner with a `WRITE` grant could install a bucket policy that locked out the owner and every other grantee.
The policy and the ACL are now combined per action the way AWS does it: an explicit policy Deny wins, and otherwise the action is allowed by either a policy Allow or an ACL grant. The bucket owner keeps everything a policy doesn't explicitly deny and, like an AWS account root user, can always call `GetBucketPolicy`, `PutBucketPolicy` and `DeleteBucketPolicy` even past a Deny. The protection covers only those three APIs; `GetBucketPolicyStatus` and every other action stay deniable, matching AWS.
ACL grants to non-owners now follow AWS's ACL-to-action mapping through the new `aclActionPermissions` table: `READ` grants `s3:ListBucket`, `s3:ListBucketVersions`, `s3:ListBucketMultipartUploads`, `s3:GetObject`, `s3:GetObjectVersion` and `s3:GetObjectAttributes`; `WRITE` grants `s3:PutObject` and `s3:DeleteObject`; `READ_ACP` and `WRITE_ACP` grant reading and writing the bucket and object ACLs. Bucket configuration, the bucket policy, tagging, object lock settings and `DeleteBucket` are the owner's alone unless a policy grants them. `verifyACL` is replaced by `aclAllows`, which derives the required permission from the action, so `AccessOptions.AclPermission` now only feeds the readonly gate. The anonymous path uses the same table in place of `publicACLAllowedActions`, which leaves its behavior unchanged.
`GetBucketPolicyStatus` also reported a Deny-only `Principal: "*"` policy as public; only Allow statements now count toward `IsPublic`.
LuisFigueroaG
Ben McClellandandGitHub
Javier GarciaandBen McClelland
niksis02
dependabot[bot]andGitHub
Luke McCrone
dualfroz
Tung Lam
theluckystrikeandniksis02
Keith KennedyandClaude Opus 5.5