In sidecar mode, the three StoreAttribute/storeChecksums calls after the
copy loop were using objPath (the source object path) instead of the
destination part's bucket and partPath. This caused checksums and the
internal part-crc64nvme to be written under the source object's sidecar
directory, making them unresolvable when CompleteMultipartUploadWithCopy
tried to retrieve them. All three stores now use *upi.Bucket and partPath,
consistent with the etag store directly below them.
Replace the length-comparison condition with an explicit predicate that
is both more readable and correctly scoped: skip only when the visited
directory is a strict ancestor of the specified prefix (not a descendant
and not when prefix is empty).
Adds tests from the original bug report (#1864) to verify the fix and
guard against future regressions.
Closes#1847
This PR introduces a global optional gateway CLI flag `--disable-acl` (`VGW_DISABLE_ACL`) to disable ACL handling. When this flag is enabled, the gateway ignores all ACL-related headers, particularly in `CreateBucket`, `PutObject`, `CopyObject`, and `CreateMultipartUpload`.
`GetBucketAcl` behavior is unchanged simply returning the bucket ACL config.
There's no change in object ACL actions(`PutObjectACL`, `GetObjectACL`). They return a`NotImplemented` error as before.
A new custom error is added for PutBucketAcl calls when ACLs are disabled at the gateway level. Its HTTP status code and error code match AWS S3’s behavior, with only a slightly different error message.
In the access-control checker, ACL evaluation is fully bypassed. If ACLs are disabled only the bucket owner gets access to the bucket and all grantee checks are ignored.
The PR also includes minor refactoring of the S3 API server and router. The growing list of parameters passed to the router’s Init method has been consolidated into fields within the router struct, initialized during router construction. Parameters not needed by the S3 server are no longer stored in the server configuration and are instead forwarded directly to the router.
This PR optimizes multipart upload checksum handling. When a checksum algorithm/type is specified at multipart-upload initiation, each `UploadPart` request computes, validates, and stores the corresponding part checksum. During `CompleteMultipartUpload`, the final checksum is derived either via composite checksum calculation or by composing the CRC-family checksums.
When **no** checksum algorithm is specified during multipart-upload initiation, each `UploadPart` may supply a different checksum algorithm for data-integrity verification. To support this scenario, a new mechanism has been implemented: for every `UploadPart`, a **crc64nvme** checksum is always computed.
* If the client uses crc64nvme for the part upload, a single hash reader is used.
* Otherwise, two hash readers are used—one for crc64nvme and one for the user-provided checksum.
The crc64nvme value is stored in part xattrs under `user.part-crc64nvme` and later used during `CompleteMultipartUpload` as a composable checksum source.
In `CompleteMultipartUpload`, the hash reader is entirely removed; the gateway no longer re-reads part data to compute the final checksum. The logic now follows two distinct paths:
1. **Checksum algorithm/type specified at MP initiation**
* All required per-part checksums have already been stored.
* If the checksum type is `FULL_OBJECT`, the gateway uses the composable path.
* If the type is `COMPOSITE`, the gateway follows the checksum-combining path.
2. **No checksum algorithm specified at MP initiation**
* The gateway loads the stored per-part `crc64nvme` values and composes them to compute the final checksum.
The previous `composableCRC` check has been removed because all `FULL_OBJECT` algorithms are inherently composable (`crc32`, `crc32c`, `crc64nvme`). Validation now relies solely on `checksum.Type`.
Previously, if no object checksum type/algorithm was specified when initiating a multipart upload, the CompleteMultipartUpload request would compute the final object’s CRC64NVME checksum but not persist it. This logic has now been fixed, and in the scenario described above the checksum is stored on the final object. There should no longer be any case where a CompleteMultipartUpload request finishes without persisting the final object checksum.
This brings scoutfs in-line with the posix concurrency limiter.
This fixes a hang with scoutfs due to not correctly initializing
the concurrency in posix leading to a concurrency of 0 allowed.
This also adds a sane default to the posix concurrency when not
initialized.
Add data-integrity checksum support in `PutObject` in the POSIX backend for directory objects. Since the only way to upload a directory object is via `PutObject`, this logic validates and stores the checksum of the empty payload. Support for `GetObject` has also been added to retrieve and return directory-object checksums.
Fixes#1870Fixes#1863
A validation has been added to **PutBucketCors** for `CORSRule.AllowedOrigins`. The `AllowedOrigins` list can no longer be empty—otherwise a **MalformedXML** error is returned. Additionally, each origin is now validated to ensure it does not contain more than one wildcard.
A similar validation has been added for `AllowedMethods`. The list must not be empty, or a **MalformedXML** error is returned. Previously, empty method values (e.g., `[]string{""}`) were incorrectly treated as valid. This has been fixed, and an **UnsupportedCORSMethod** error is now returned.
On small screens the sidebar now collapses out of view by default,
replaced by a visible toggle button that slides it back in. Without
this, the sidebar occupied the full screen width on phones and tablets,
leaving no room for page content.
Co-authored-by: Ben McClelland <ben.mcclelland@versity.com>
AWS introduced a relatively newer option for data integrity checks
that not all non-AWS server support yet. See this for mmore info:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
This change adds a new option: disable-data-integrity-check
to disable the data integrity checks in the client sdk for the
servers that may not yet support this. Use this only when the s3
service for the proxy does not support the data integrity features.
Fixes#1867
New cli options added:
webui-gateways - override auto-detected S3 gateway URLs for WebUI
webui-admin-gateways - override auto-detected admin gateway URLs
for WebUI
These also accept env vars VGW_WEBUI_GATEWAYS and
VGW_WEBUI_ADMIN_GATEWAYS for the options.
When setting these, this will override the url auto-detection for
the webui service urls dropdown options. By default, the gateway
auto-detects URLs based on the configured port settings. Use these
options to specify custom URLs when the auto-detected values are
incorrect (e.g., when running behind a reverse proxy or load
balancer). Multiple URLs can be specified with repeated options
or a comma-separated list with the environment variables.
for example:
--webui-gateways https://s3.example.com \
--webui-gateways http://192.168.1.100:7070
or
VGW_WEBUI_GATEWAYS=https://s3.example.com,http://192.168.1.100:7070
The gateway will validate the provided URLs with warnings for any
invalid URL specified. The gateway will terminate if these options
are set but contain no valid URLs.
Also added sorting to the auto-detected URLs so that localhost
URLs will be last in the list, since these will not likely work
on remote systems. The specified lists when provided are left
in the order they are specified to allow admins to determine
dropdown list ordering.
Fixes#1851
Fixes#1869
Generally, when object ownership is not explicitly specified during bucket creation, it defaults to `BucketOwnerEnforced`. With `BucketOwnerEnforced`, ACLs are disabled and any attempt to set one results in an `InvalidBucketAclWithObjectOwnership` error.
However, there is an edge case. When the `private` canned ACL is used during bucket creation—which is effectively the default ACL for all buckets—`BucketOwnerEnforced` is still permitted. Moreover, if no explicit object ownership is specified together with the `private` canned ACL, the ownership defaults to `BucketOwnerPreferred`.
This fix also resolves the issue with rclone bucket creation, since rclone sends `x-amz-acl: private` by default:
```
rclone mkdir vgw:test
```
This allows specifying the following options more than once:
port, admin-port, webui
or using a comma-separated list for the env vars:
e.g., VGW_PORT=:7070,:8080,localhost:9090
This will also expand multiple interfaces from hostnames, for example
"localhost" in this case would resolve to both IPv4 and IPv6 interfaces:
localhost has address 127.0.0.1
localhost has IPv6 address ::1
This updates the banner to reflect all of the listening interfaces/ports,
and starts the service listener on all requested interfaces/ports.
Fixes#1761
Fixes#1849
If no `Content-Type` is provided during object upload, S3 defaults it to `application/octet-stream`. This behavior was missing in the gateway, causing backends to persist an empty `Content-Type`, which Fiber then overrides with its default `text/plain`. The behavior has now been corrected for the object upload operations: `PutObject`, `CreateMultipartUpload`, and `CopyObject`.
