GitHub has no FreeBSD runners, so the new `functional-freebsd.yml` workflow runs the `runtests.sh` inside a FreeBSD 15.1 VM. The VM is booted under KVM on `ubuntu-latest` by `vmactions/freebsd-vm`, the most widely used action for this.
The `prepare` step installs `bash`, `git` and `gmake`, downloads the latest stable Go from `go.dev` to match `setup-go`'s `stable`, and creates a `tester` user that owns the synced workspace. `git` is required because the synced `.git` directory makes `go build` stamp VCS info. The `run` step then runs `gmake testbin`, `runtests.sh` and the coverage report as `tester`, matching the unprivileged Linux job.
FreeBSD enables ASLR by default (`kern.elf64.aslr.enable=1`), and the race-enabled `versitygw` built by `testbin` refuses to start under it with "This sanitizer is not compatible with enabled ASLR", so `prepare` turns ASLR off with `sysctl`.
The nested VM freezes for up to ~13 seconds at a time under the parallel suite, longer than the few seconds of margin some object-lock tests gave their retain until dates, so a date could expire in transit (the gateway then rejects it with `InvalidArgument`) or a COMPLIANCE lock could expire mid-test. A new `retryOnPastRetainDate` helper resends a request whose seconds-out date expired in transit, `complianceTestRetention` grows from 10s to 30s, and the tests that locked for twice that window, or extended by a whole one, now stay within a single window, which keeps the added suite time to about 10 seconds per run.
Follow-up to #2486
The AWS SDKs and CLI send `ObjectIdentifier.LastModifiedTime` as an HTTP date, but `s3response.DeleteObjects` decoded it into the SDK's `time.Time` field, which accepts only RFC 3339. Any `DeleteObjects` request that carried a `LastModifiedTime` was therefore rejected as a whole with `InvalidRequest`, and the RFC 3339 form the gateway did accept is one that S3 rejects.
`DeleteObjects` now has its own `UnmarshalXML`. It embeds `types.ObjectIdentifier`, decodes `LastModifiedTime` as a string and parses it with `smithytime.ParseHTTPDate`, falling back to `time.RFC1123Z` for numeric zone offsets. This accepts IMF-fixdate, RFC 850, ANSIC and `+hhmm` offsets and rejects RFC 3339, which matches S3 directory buckets, the only bucket type where S3 evaluates this field.
Errors are reported as S3 reports them. An unparseable date returns `InvalidRequest` with "Invalid value for date attribute: <value>" through the new `s3err.GetInvalidDateAttributeErr`, which echoes the raw value of the first bad date in the body. An empty, whitespace-only or self-closing `LastModifiedTime` returns `MalformedXML`, which takes precedence over an invalid date in any object. The `DeleteObjects` controller now passes S3 errors from the decoder through instead of turning every decode failure into `InvalidRequest`.
fiber `v3.6.0` (gofiber/fiber#4695) normalizes every request path before routing as RFC 3986 Section 6.2.2 describes: it decodes the escapes of unreserved characters and removes `.` and `..` segments, and `ctx.Path()`, `ctx.Params()` and every `ctx.Path(override)` return the normalized path with no way to opt out. `DecodeURL` sets the decoded path back through `ctx.Path()`, so keys were decoded twice (`100%2541` was stored as `100A`, `q%7euser` as `q~user`) and dot segments rerouted requests: `PUT /bucket/.` became `CreateBucket` and `/bucket/../../etc/passwd` named the bucket `etc`.
The new `httpctx.Ctx`, created with `fiber.NewWithCustomCtx`, keeps a path set with `Path()` as it is and has fiber route a copy in which every `%` and the dots of a `.` or `..` segment are `_`. The copy keeps the length of the path, since fiber matches routes on it but takes the route parameters from `Path()` at the matched offsets, so `Params("bucket")` holds the real bucket. The S3, admin and website servers and the RDMA ops tracker use it.
S3 stores keys holding `.` and `..` segments as they are (`a/../b`, `x/.` and `..` are objects of their own) and only rejects, before authentication, a request path whose decoded `..` segments climb above the root, separating the segments at `\` as well as at `/` and skipping the empty ones. `DecodeURL` now rejects such a path with `InvalidURI` and keeps the dot segments in the key otherwise.
`Backend.StoresDotSegmentKeys()` reports whether a backend stores such keys as their own objects: `azure` and `s3proxy` do, while `posix` and the `BackendUnsupported` default don't, since a filesystem resolves the segments to another object or to a path outside the bucket. `IsObjectNameValid` takes the backend and rejects keys holding dot segments for those backends in the request path, the `DeleteObjects` keys, the `PostObject` key and the copy sources, so `posix` now also rejects `a/../b` and `a/./b`, which it stored as `b` and `a/b`. The unused `path.go` is removed.
Fixes#2525
`CompleteMultipartUpload` returned the stored part's ETag in `InvalidPart` errors instead of the ETag the client sent: posix on an ETag mismatch, and azure on a zero-byte part mismatch, where it returned the expected block ID. Azure also returned an empty ETag for a part that was never uploaded. All three now return the client's ETag. Like S3, `GetInvalidPartErr` removes every `"` from it, inner quotes included, and keeps surrounding whitespace.
`SetResponseHeaders` turns off header name normalization and leaves it off, so response header names stay lowercase. Before it runs, the `RequestIDs` middleware stored `X-Amz-Request-Id` and `X-Amz-Id-2`. Afterwards, the `EnsureRequestIDs` call in `ProcessController` stored `x-amz-request-id` and `x-amz-id-2`. Every response with a headers map, success responses included, therefore sent both pairs. `EnsureRequestIDs`, `RequestID` and `HostID` now use `SetCanonical` and `PeekCanonical` with the lowercase names, so the normalization setting doesn't affect them, and each header is sent once in lowercase as S3 does.
The test was skipped for the gateway with a link to #925, which is fixed
here. The gateway already returns InvalidArgument for an unknown
attribute, so the test passes, and the skip would point to a closed
issue once #925 is closed.
ListObjects and ListObjectsV2 ignored the encoding-type query parameter.
Any value was accepted, and with encoding-type=url the response carried
neither an EncodingType element nor encoded key names.
S3 accepts only "url" and answers anything else with InvalidArgument
("Invalid Encoding Method specified in Request", argument encoding-type).
With "url" it sets EncodingType and encodes the key names in the
listing as a query component, "a+ b.txt" becoming "a%2B+b.txt", with
"/" left as is: Key, Prefix, Delimiter and CommonPrefixes in both
operations, Marker and NextMarker in ListObjects, and StartAfter in
ListObjectsV2. The continuation tokens are opaque and are not encoded.
The encoding is applied to a copy of the backend result in the
controllers, so it covers every backend.
Fixes#1985Fixes#1984
The posix backend stages multipart uploads and temporary files inside the bucket directory under `.sgwtmp`, but nothing stopped an object key from resolving there. A client with access to the bucket could read the parts of other clients' in-progress uploads with `GetObject`/`HeadObject`, copy them out with `CopyObject`, overwrite or delete them with `PutObject`/`DeleteObject` so the upload failed on completion with `InvalidPart`, and store data that never shows up in listings. Spellings such as `./.sgwtmp/...`, `x/../.sgwtmp/...`, a leading `/` and, on case-insensitive file systems, `.SGWTMP/...` reached the same files.
Add the non-AWS `ReservedObjectKey` error (400) and return it whenever an object key resolves to `.sgwtmp` or a path inside it. The key is resolved the same way `ObjectPath` resolves it, and its first path element is compared case-insensitively. The check runs after the bucket existence check in `PutObject`, `GetObject`, `HeadObject`, `GetObjectAttributes`, `DeleteObject`, `DeleteObjects` (reported per key), `CopyObject` (source and destination), the `UploadPartCopy` source, `CreateMultipartUpload`, `CompleteMultipartUpload`, object tagging, legal hold and retention. The exported `ValidateObjectKey` covers the ScoutFS `GetObject` and `RestoreObject` entry points. `UploadPart`, `ListParts` and `AbortMultipartUpload` only hash the key and stay unchanged, so uploads created with a reserved key before this fix can still be aborted.
`ListObjects` and `ListObjectsV2` also listed the staged parts when the prefix named a directory inside `.sgwtmp`, because the prefix fast path in `quickWalk` compared only the last element of the prefix directory against the skipped directories. `inSkipDir` now checks every element, ignoring case.
Fixes#2513
On Windows the posix backend stored a key as `bucket/key` verbatim, but Windows rejects `" * : < > ? \ |` and control characters in file names, so such a PUT failed with `ERROR_INVALID_NAME`, which `isErrNameTooLong` reported as a misleading `KeyTooLongError`. The same mapping did worse elsewhere: `\` acted as a path separator, so `a\b` aliased `a/b` and `..\..\x` resolved outside the bucket because `IsObjectNameValid` only splits on `/`; `a:b` wrote an NTFS alternate data stream; Win32 strips a trailing dot or space, so `foo.` overwrote `foo`; and device names such as `CON` or `nul.txt` opened the device and silently discarded the data.
With `encodeKeys`, enabled by default only on Windows, `encodeKey` stores each key segment under a name Windows accepts: every invalid character, a trailing dot or space, and the first letter of a device name are stored as the private use character `U+F000` plus the character code, the convention Cygwin and WSL use for invalid characters. A key character in `U+F000`-`U+F07F` is stored after `U+F000`, so every stored name belongs to exactly one key. Device names are checked with `filepath.IsLocal`, which asks the system, so `CON.txt` stays a plain file on Windows 11 where it is valid. Keys that need no encoding keep their names, so existing data is unaffected, and nothing changes on other platforms.
Public backend methods keep taking raw keys and encode them once for every path, metadata, version-hash and publish-lock use, while responses, errors and the multipart `objname` attribute keep the raw key and multipart upload directories keep hashing it. Listings walk `bucketFS`, which decodes the entry names, sorts them in key byte order to match S3 and skips names no key is stored under. `HeadObject` passes the raw key to `GetObjectLegalHold` and `GetObjectRetention` to avoid encoding twice, and `NormalizeObjectKey` resolves the encoded key, so policy resources see `a\b` as one literal key.
GetObjectAttributes never returned ObjectParts, so a request for it on
an object completed from a multipart upload got no PartsCount. S3
returns <ObjectParts><PartsCount>N</PartsCount></ObjectParts> for these
objects and leaves ObjectParts out for objects that weren't uploaded in
parts.
The posix and azure backends already keep the part layout of completed
uploads, and HeadObject reports the parts count when a part number is
requested. GetObjectAttributes now uses that to fill ObjectParts. The
ObjectParts fields that aren't set are left out of the response, as S3
does.
Fixes#925
A conditional delete of a key with no current version used to succeed. `DeleteObject` with `If-Match`, and a `DeleteObjects` key with an `ETag`, reported success for a missing key. When the current version was a delete marker, the condition was checked against the ETag the marker kept from the version it hides, so it could pass and create a second marker. S3 fails both with `NoSuchKey` and creates no marker. The posix backend now checks the conditions against the current version in `checkDeletePreconditions`, before its missing-key shortcuts. The azure backend returns `NoSuchKey` for a missing blob instead of skipping the conditions. `ObjectDeletePreconditions.IsSet` reports whether a delete has any condition.
A condition only applies to the current version. `DeleteObject` with a `versionId` and `If-Match` now returns `NotImplemented` with `<Header>If-Match</Header>` and the S3 additional message, ahead of authorization. Before, it checked the condition against that version and deleted the version. In `DeleteObjects`, an empty `<ETag>` fails its key with `InvalidArgument` and a `VersionId` together with an `ETag` fails it with `NotImplemented`, both ahead of the key's authorization.
The `PreconditionFailed` and `NotImplemented` messages drop their trailing period, which S3 doesn't use on any operation.
DeleteObjects dropped the ETag, Size and LastModifiedTime of each object
when it called DeleteObject, so an object with a stale ETag was still
deleted. The posix and azure backends now forward them to DeleteObject.
EvaluateObjectDeletePreconditions trims the quotes an SDK sends around
the ETag and treats "*" as a match for any ETag.
The REST ETag mismatch test no longer skips.
Taken from #2485, without its posix and integration tests.
Refs #2112
Fixes#2499
An empty tag key returned `InvalidTag` without the `TagKey` element because the field was `omitempty`. S3 always sends `<TagKey>`, empty for an empty key, so it is now always written.
`PutBucketTagging`, `PutObjectTagging`, the POST object `tagging` field and `CreateBucket` tags now share one validator, `backend.ParseTagSet`, which matches S3: a duplicate key error carries `TagKey`, a `Tag` missing its `Key` or `Value` element is `MalformedXML` and is checked across the whole document first, every tag is validated before the tag count limit, and the 128/256 length limits count UTF-16 code units instead of bytes. The duplicate tag regex in `s3api/utils` is removed.
For the `x-amz-tagging` header, the length limits now apply to the URL-decoded key and value, the `InvalidArgument` error echoes the whole header as `ArgumentValue` instead of the unparsed remainder, and more than 10 tags is rejected with `BadRequest`.
Fixes#2489
An unsupported `Authorization` algorithm returned only the algorithm in `ArgumentValue`, while AWS S3 echoes the whole header as received. `sigv4auth.ParseAuthorization` now carries the full header in `ParseError.Input`, which the S3 error mapper returns. Also drop the trailing period from the `InvalidArgAuthHeader` message to match AWS.
The azure backend tracks zero-byte parts separately and reports its own
block id as the ETag when the client ETag doesn't match, so the test
failed against azurite. Upload some data so the part is a regular block,
and send the ETag unquoted so both backends echo it back as sent.
Once a bucket had any policy, `verifyResourceAccess` evaluated the policy alone and ignored the bucket ACL, so a non-root bucket owner kept only what the policy explicitly allowed. A Deny-only policy (e.g. Deny `*` `s3:GetObject` on one prefix) therefore locked the owner out of every operation, including `DeleteBucketPolicy`, and only root/admin could recover. ACL grants also authorized every bucket operation through the controller's `AclPermission`, so a non-owner with a `WRITE` grant could install a bucket policy that locked out the owner and every other grantee.
The policy and the ACL are now combined per action the way AWS does it: an explicit policy Deny wins, and otherwise the action is allowed by either a policy Allow or an ACL grant. The bucket owner keeps everything a policy doesn't explicitly deny and, like an AWS account root user, can always call `GetBucketPolicy`, `PutBucketPolicy` and `DeleteBucketPolicy` even past a Deny. The protection covers only those three APIs; `GetBucketPolicyStatus` and every other action stay deniable, matching AWS.
ACL grants to non-owners now follow AWS's ACL-to-action mapping through the new `aclActionPermissions` table: `READ` grants `s3:ListBucket`, `s3:ListBucketVersions`, `s3:ListBucketMultipartUploads`, `s3:GetObject`, `s3:GetObjectVersion` and `s3:GetObjectAttributes`; `WRITE` grants `s3:PutObject` and `s3:DeleteObject`; `READ_ACP` and `WRITE_ACP` grant reading and writing the bucket and object ACLs. Bucket configuration, the bucket policy, tagging, object lock settings and `DeleteBucket` are the owner's alone unless a policy grants them. `verifyACL` is replaced by `aclAllows`, which derives the required permission from the action, so `AccessOptions.AclPermission` now only feeds the readonly gate. The anonymous path uses the same table in place of `publicACLAllowedActions`, which leaves its behavior unchanged.
`GetBucketPolicyStatus` also reported a Deny-only `Principal: "*"` policy as public; only Allow statements now count toward `IsPublic`.
In posix, when the upload directory rename in `CompleteMultipartUploadWithCopy` fails with `ENOENT`, the call is treated as a retry of an already completed upload. That fallback returned 200 whenever any regular file existed at the target bucket/key, with an ETag computed from the request's part ETags, and never checked that the object came from this upload. Completing with the upload ID of another bucket or key, a random upload ID, or an aborted upload was reported as stored while nothing was written. The later `mp-metadata` upload ID check was unreachable whenever the object existed.
The fallback now succeeds only when the object at the key carries this upload ID in its `mp-metadata` attribute and, outside `--data-integrity-etag` mode, when its stored ETag equals the multipart ETag computed from the request's parts. This matches AWS, which rejects a retry with different part ETags with `NoSuchUpload` but ignores part numbers. Every other case returns `NoSuchUpload`, as AWS does for a foreign, random or aborted upload ID and for a retry after the object was overwritten or deleted. Keys with a trailing slash are rejected up front because `ObjectPath` resolves `foo/` to the file `foo`, and attribute read errors other than a missing attribute are no longer reported as `NoSuchUpload`. A successful retry now returns the stored ETag and the object's `version-id`, matching AWS, which returns the original VersionId on a retry in a versioned bucket.
The posix ListObjectsV2 reported the listing marker as the response
ContinuationToken. The marker is the larger of start-after and the
continuation token, so a request with only start-after got its
start-after value back as a ContinuationToken, and a start-after past
the continuation token replaced it. S3 echoes the continuation token
from the request, and leaves ContinuationToken out when the request
had none.
Return the request continuation token instead of the marker, and
enable the REST test that covers it.
Fixes#2007
CompleteMultipartUpload with a part ETag that is not a hex MD5 returned
a bare InvalidPart error, without the UploadId, PartNumber and ETag
elements. S3 includes them, and the gateway already does for a part
that is missing or whose ETag doesn't match.
Pass the upload ID to ComputeMultipartETagFromPartETags so it can
return the detailed InvalidPart error for the offending part, and
enable the REST test that covers it.
Fixes#1008
CopyObject ignored the destination's `If-Match` and `If-None-Match` headers and always overwrote or created the destination, so a stale `If-Match`, `If-None-Match: *` on an existing key, `If-Match` on a missing key and `If-Match` over a delete marker all returned 200. The controller now parses both headers and passes them to the backend through new `IfMatch` and `IfNoneMatch` fields on `s3response.CopyObjectInput`, which match the SDK's `s3.CopyObjectInput`.
As in S3, the destination is evaluated before the copy source is looked up, so a failing destination precondition wins over a missing source, a failing `x-amz-copy-source-*` precondition and an invalid copy onto itself. Posix checks the destination with `checkPutPreconditions` right after the destination bucket check, then forwards the headers to `PutObject`, which repeats the check under the object publish lock. The unversioned in-place self copy does not go through `PutObject`, so it now takes `lockObjectPublish` and repeats the check itself.
`checkPutPreconditions` now treats a current delete marker as a missing key: `If-Match` returns `NoSuchKey` and `If-None-Match: *` creates a new version. Before, the hidden version's etag was evaluated, which also affected `PutObject` and `CompleteMultipartUpload`.
`EvaluateObjectPutPreconditions` now rejects unsupported headers in S3's order: both headers present, then `If-None-Match` other than `*`, then `If-Match: *`, each with `NotImplemented`. `If-Match: *` was previously compared as a literal etag. The new `NmpAdditionalMessageIfMatch` message is added, and a stray backslash is removed from the `If-None-Match` message.
Azure evaluates the destination preconditions before copying, and s3proxy forwards the headers. `s3:if-match` and `s3:if-none-match` are now populated in the policy condition context for CopyObject, as S3 does, while UploadPartCopy stays excluded.
S3 never lists an empty result for another region: a valid region
other than the endpoint's own gets 400 InvalidArgument, "Requests with
bucket-region specified must be made to the corresponding regional
endpoint", with no ArgumentValue. The gateway is a single regional
endpoint, so it now does the same instead of clearing the listing.
Match the region case-insensitively like S3, and check an empty
bucket-region= too by looking for the query argument itself, since an
empty value is not a valid AWS region either.
ListBuckets ignored bucket-region: a value that is not a region was
accepted, and a region other than the gateway one still listed every
bucket. S3 rejects an invalid value with 400 InvalidArgument, "Argument
value abc is not a valid AWS Region", and lists only the buckets of the
requested region.
Check bucket-region against the AWS region code format, always
accepting the gateway region whatever its name, and return an empty
list for any other region, since every bucket lives in the gateway
region. Enable the REST tests that cover both cases.
Fixes#1923Fixes#1930
With an upload-id-marker the lister ranges over Uploads[startIndex:],
but passed the index of that subslice to hasMoreAfter, which indexes
the full list. A page that ended on the last remaining upload was still
reported as truncated, with next markers pointing at that upload, so a
client had to make one more request to get an empty page.
Iterate over the full list from startIndex, so hasMoreAfter gets the
real position of the next upload.
Ben McClellandandGitHub
niksis02
LuisFigueroaG
Javier GarciaandBen McClelland
Luke McCrone
dualfroz
theluckystrikeandniksis02