Azure's ListContainers Marker parameter requires an opaque internal token
(e.g. /accountname/containername) rather than a plain container name, so
passing MaxResults and our ContinuationToken directly to the Azure API
caused 400 OutOfRangeInput errors. Rework ListBuckets to iterate all Azure
pages client-side, skip entries at or before the ContinuationToken (matching
the posix backend's "start after" semantics), and stop once MaxBuckets items
have been collected, setting ContinuationToken to the last returned bucket
name. This avoids using Azure's NextMarker entirely and correctly handles
both unpaginated and paginated requests.
Azure Storage's StageBlock REST API rejects Content-Length: 0
with InvalidHeaderValue. The tests (PresignedAuth_UploadPart,
UploadPart_success) upload a nil/empty body, which causes the
Azure SDK to send Content-Length: 0. Azurite is lenient and
accepts it; real Azure Storage does not.
Use a new metadata key ("Zerobytesparts") sett on the
.sgwtmp/multipart/<uploadId>/<object-hash> blob to track and
0 length parts.
Fixes#1917
When the `--webui` flag uses `localhost:<port>`, the hostname `localhost` resolves to both IPv6 and IPv4 addresses. Previously, the Fiber web server was initialized without an explicit network setting, and Fiber defaults to `tcp4` (IPv4-only). Because `tcp4` cannot bind to IPv6 addresses, any resolved IPv6 address (e.g., `::1`) was treated as invalid for the listener.
The network mode is now changed to `tcp`, which enables dual-stack behavior and allows binding to both IPv4 and IPv6 addresses.
The logic to return a `NotImplemented` error on object upload operations, when any ACL header is present has been removed. Now all object ACL headers are by default ignored. The `-noacl` flag is preserved to disabled bucket ACLs.
**Testing**
The Put/Get object ACL tests are moved to `NotImplemented` integration tests group as a default gateway behavior. The existing `_acl_not_supported` tests are modified to expect no error, when ACLs are used on object uploads.
Closes#1857
Adds object Tagging support for directory objects in `PutObject` posix. Updates the integration tests to test object metadata and tagging both for file and directory objects.
In sidecar mode, the three StoreAttribute/storeChecksums calls after the
copy loop were using objPath (the source object path) instead of the
destination part's bucket and partPath. This caused checksums and the
internal part-crc64nvme to be written under the source object's sidecar
directory, making them unresolvable when CompleteMultipartUploadWithCopy
tried to retrieve them. All three stores now use *upi.Bucket and partPath,
consistent with the etag store directly below them.
Replace the length-comparison condition with an explicit predicate that
is both more readable and correctly scoped: skip only when the visited
directory is a strict ancestor of the specified prefix (not a descendant
and not when prefix is empty).
Adds tests from the original bug report (#1864) to verify the fix and
guard against future regressions.
Closes#1847
This PR introduces a global optional gateway CLI flag `--disable-acl` (`VGW_DISABLE_ACL`) to disable ACL handling. When this flag is enabled, the gateway ignores all ACL-related headers, particularly in `CreateBucket`, `PutObject`, `CopyObject`, and `CreateMultipartUpload`.
`GetBucketAcl` behavior is unchanged simply returning the bucket ACL config.
There's no change in object ACL actions(`PutObjectACL`, `GetObjectACL`). They return a`NotImplemented` error as before.
A new custom error is added for PutBucketAcl calls when ACLs are disabled at the gateway level. Its HTTP status code and error code match AWS S3’s behavior, with only a slightly different error message.
In the access-control checker, ACL evaluation is fully bypassed. If ACLs are disabled only the bucket owner gets access to the bucket and all grantee checks are ignored.
The PR also includes minor refactoring of the S3 API server and router. The growing list of parameters passed to the router’s Init method has been consolidated into fields within the router struct, initialized during router construction. Parameters not needed by the S3 server are no longer stored in the server configuration and are instead forwarded directly to the router.
This PR optimizes multipart upload checksum handling. When a checksum algorithm/type is specified at multipart-upload initiation, each `UploadPart` request computes, validates, and stores the corresponding part checksum. During `CompleteMultipartUpload`, the final checksum is derived either via composite checksum calculation or by composing the CRC-family checksums.
When **no** checksum algorithm is specified during multipart-upload initiation, each `UploadPart` may supply a different checksum algorithm for data-integrity verification. To support this scenario, a new mechanism has been implemented: for every `UploadPart`, a **crc64nvme** checksum is always computed.
* If the client uses crc64nvme for the part upload, a single hash reader is used.
* Otherwise, two hash readers are used—one for crc64nvme and one for the user-provided checksum.
The crc64nvme value is stored in part xattrs under `user.part-crc64nvme` and later used during `CompleteMultipartUpload` as a composable checksum source.
In `CompleteMultipartUpload`, the hash reader is entirely removed; the gateway no longer re-reads part data to compute the final checksum. The logic now follows two distinct paths:
1. **Checksum algorithm/type specified at MP initiation**
* All required per-part checksums have already been stored.
* If the checksum type is `FULL_OBJECT`, the gateway uses the composable path.
* If the type is `COMPOSITE`, the gateway follows the checksum-combining path.
2. **No checksum algorithm specified at MP initiation**
* The gateway loads the stored per-part `crc64nvme` values and composes them to compute the final checksum.
The previous `composableCRC` check has been removed because all `FULL_OBJECT` algorithms are inherently composable (`crc32`, `crc32c`, `crc64nvme`). Validation now relies solely on `checksum.Type`.
Previously, if no object checksum type/algorithm was specified when initiating a multipart upload, the CompleteMultipartUpload request would compute the final object’s CRC64NVME checksum but not persist it. This logic has now been fixed, and in the scenario described above the checksum is stored on the final object. There should no longer be any case where a CompleteMultipartUpload request finishes without persisting the final object checksum.
This brings scoutfs in-line with the posix concurrency limiter.
This fixes a hang with scoutfs due to not correctly initializing
the concurrency in posix leading to a concurrency of 0 allowed.
This also adds a sane default to the posix concurrency when not
initialized.
Add data-integrity checksum support in `PutObject` in the POSIX backend for directory objects. Since the only way to upload a directory object is via `PutObject`, this logic validates and stores the checksum of the empty payload. Support for `GetObject` has also been added to retrieve and return directory-object checksums.
