mirrors/versitygw
1
0
Fork 0
mirror of https://github.com/versity/versitygw.git synced 2026-08-27 19:36:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,906 Commits 33 Branches 53 Tags
8c3e49d0bbaf4f2b6512bd85af746de0019d0e3d
Commit Graph
1906 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
niksis02andBen McClelland 8c3e49d0bb fix: fixes checksum header and algorithm mismatch error 
Fixes #1598

`PutObject` and `UploadPart` accept x-amz-checksum-* calculated checksum headers and `x-amz-sdk-checksum-algorithm`. If the checksum algorithm specified in sdk algorithm doesn't match the one in x-amz-checksum-*, it now returns the correct error message: `Value for x-amz-sdk-checksum-algorithm header is invalid.`.
 2025-10-28 14:40:28 -07:00
Ben McClellandandGitHub 559d636846 Merge pull request #1599 from versity/sis/put-actions-checksum 
feat: implements checksum calculation for all actions
 2025-10-28 14:39:46 -07:00
Luke McCrone adbf8e138c test: quick fix 2025-10-24 20:18:38 -03:00
niksis02 12f4920c8d feat: implements checksum calculation for all actions 
Closes #1549
Fixes #1593
Fixes #1521
Fixes #1427
Fixes #1311
Fixes #1301
Fixes #1040

This PR primarily focuses on checksum calculation within the gateway, but it also includes several related fixes and improvements.

It introduces a middleware responsible for handling and calculating checksums for the `x-amz-checksum-*` headers and `Content-MD5`. The middleware is applied only to actions that expect a request body or checksum headers. It also enforces validation for actions that require a non-empty request body, returning an error if the body is missing. Similarly, it returns an error for actions where at least one checksum header (`Content-MD5` or `x-amz-checksum-*`) is required but none is provided.
The implementation is based on [https://gist.github.com/niksis02/eec3198f03e561a0998d67af75c648d7](the reference table), tested directly against S3:

It also fixes the error case where the `x-amz-sdk-checksum-algorithm` header is present but no corresponding `x-amz-checksum-*` or `x-amz-trailer` header is included.

Additionally, the PR improves validation for the `x-amz-content-sha256` header. For actions that require this header, an error is now returned when it’s missing. For actions that don’t require it, the middleware no longer enforces its presence. Following the common S3 pattern, the header remains mandatory for admin routes.

Finally, the `x-amz-content-sha256` header is now optional for anonymous requests, as it is not required in that case.
 2025-10-25 01:58:03 +04:00
Ben McClellandandGitHub d63b5818f1 Merge pull request #1594 from versity/dependabot/go_modules/dev-dependencies-df0f97cf04 
chore(deps): bump the dev-dependencies group with 23 updates
 2025-10-20 16:14:30 -07:00
dependabot[bot]andGitHub dff3eb0887 chore(deps): bump the dev-dependencies group with 23 updates 
Bumps the dev-dependencies group with 23 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) | `1.6.2` | `1.6.3` |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.39.2` | `1.39.3` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.88.4` | `1.88.5` |
| [github.com/aws/smithy-go](https://github.com/aws/smithy-go) | `1.23.0` | `1.23.1` |
| [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) | `1.46.1` | `1.47.0` |
| [github.com/aws/aws-sdk-go-v2/feature/ec2/imds](https://github.com/aws/aws-sdk-go-v2) | `1.18.9` | `1.18.10` |
| [github.com/aws/aws-sdk-go-v2/internal/ini](https://github.com/aws/aws-sdk-go-v2) | `1.8.3` | `1.8.4` |
| [github.com/aws/aws-sdk-go-v2/service/sso](https://github.com/aws/aws-sdk-go-v2) | `1.29.6` | `1.29.7` |
| [github.com/aws/aws-sdk-go-v2/service/ssooidc](https://github.com/aws/aws-sdk-go-v2) | `1.35.1` | `1.35.2` |
| [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) | `1.38.6` | `1.38.7` |
| [golang.org/x/net](https://github.com/golang/net) | `0.45.0` | `0.46.0` |
| [github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream](https://github.com/aws/aws-sdk-go-v2) | `1.7.1` | `1.7.2` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.31.12` | `1.31.13` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.18.16` | `1.18.17` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.19.12` | `1.19.13` |
| [github.com/aws/aws-sdk-go-v2/internal/configsources](https://github.com/aws/aws-sdk-go-v2) | `1.4.9` | `1.4.10` |
| [github.com/aws/aws-sdk-go-v2/internal/endpoints/v2](https://github.com/aws/aws-sdk-go-v2) | `2.7.9` | `2.7.10` |
| [github.com/aws/aws-sdk-go-v2/internal/v4a](https://github.com/aws/aws-sdk-go-v2) | `1.4.9` | `1.4.10` |
| [github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding](https://github.com/aws/aws-sdk-go-v2) | `1.13.1` | `1.13.2` |
| [github.com/aws/aws-sdk-go-v2/service/internal/checksum](https://github.com/aws/aws-sdk-go-v2) | `1.9.0` | `1.9.1` |
| [github.com/aws/aws-sdk-go-v2/service/internal/presigned-url](https://github.com/aws/aws-sdk-go-v2) | `1.13.9` | `1.13.10` |
| [github.com/aws/aws-sdk-go-v2/service/internal/s3shared](https://github.com/aws/aws-sdk-go-v2) | `1.19.9` | `1.19.10` |
| [github.com/klauspost/compress](https://github.com/klauspost/compress) | `1.18.0` | `1.18.1` |


Updates `github.com/Azure/azure-sdk-for-go/sdk/storage/azblob` from 1.6.2 to 1.6.3
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/sdk-breaking-changes-guide-migration.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/storage/azblob/v1.6.2...sdk/storage/azblob/v1.6.3)

Updates `github.com/aws/aws-sdk-go-v2` from 1.39.2 to 1.39.3
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.39.2...v1.39.3)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.88.4 to 1.88.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.88.4...service/s3/v1.88.5)

Updates `github.com/aws/smithy-go` from 1.23.0 to 1.23.1
- [Release notes](https://github.com/aws/smithy-go/releases)
- [Changelog](https://github.com/aws/smithy-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/smithy-go/compare/v1.23.0...v1.23.1)

Updates `github.com/nats-io/nats.go` from 1.46.1 to 1.47.0
- [Release notes](https://github.com/nats-io/nats.go/releases)
- [Commits](https://github.com/nats-io/nats.go/compare/v1.46.1...v1.47.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.18.9 to 1.18.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.18.9...config/v1.18.10)

Updates `github.com/aws/aws-sdk-go-v2/internal/ini` from 1.8.3 to 1.8.4
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.8.3...service/tnb/v1.8.4)

Updates `github.com/aws/aws-sdk-go-v2/service/sso` from 1.29.6 to 1.29.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.29.6...config/v1.29.7)

Updates `github.com/aws/aws-sdk-go-v2/service/ssooidc` from 1.35.1 to 1.35.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pi/v1.35.1...service/pi/v1.35.2)

Updates `github.com/aws/aws-sdk-go-v2/service/sts` from 1.38.6 to 1.38.7
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/sts/v1.38.6...service/sts/v1.38.7)

Updates `golang.org/x/net` from 0.45.0 to 0.46.0
- [Commits](https://github.com/golang/net/compare/v0.45.0...v0.46.0)

Updates `github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream` from 1.7.1 to 1.7.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.7.1...service/mq/v1.7.2)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.31.12 to 1.31.13
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.31.12...config/v1.31.13)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.18.16 to 1.18.17
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.18.16...config/v1.18.17)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.19.12 to 1.19.13
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/efs/v1.19.12...service/efs/v1.19.13)

Updates `github.com/aws/aws-sdk-go-v2/internal/configsources` from 1.4.9 to 1.4.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/m2/v1.4.9...service/m2/v1.4.10)

Updates `github.com/aws/aws-sdk-go-v2/internal/endpoints/v2` from 2.7.9 to 2.7.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/internal/endpoints/v2.7.9...internal/endpoints/v2.7.10)

Updates `github.com/aws/aws-sdk-go-v2/internal/v4a` from 1.4.9 to 1.4.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/m2/v1.4.9...service/m2/v1.4.10)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding` from 1.13.1 to 1.13.2
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.13.1...service/m2/v1.13.2)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.9.0 to 1.9.1
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.9.0...v1.9.1)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/presigned-url` from 1.13.9 to 1.13.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.13.9...service/mq/v1.13.10)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/s3shared` from 1.19.9 to 1.19.10
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/m2/v1.19.9...service/sqs/v1.19.10)

Updates `github.com/klauspost/compress` from 1.18.0 to 1.18.1
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.18.0...v1.18.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.6.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.39.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.88.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/smithy-go
  dependency-version: 1.23.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/nats-io/nats.go
  dependency-version: 1.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds
  dependency-version: 1.18.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/ini
  dependency-version: 1.8.4
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sso
  dependency-version: 1.29.7
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ssooidc
  dependency-version: 1.35.2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-version: 1.38.7
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.46.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream
  dependency-version: 1.7.2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.31.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.18.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-version: 1.19.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/configsources
  dependency-version: 1.4.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
  dependency-version: 2.7.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/internal/v4a
  dependency-version: 1.4.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding
  dependency-version: 1.13.2
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-version: 1.9.1
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/presigned-url
  dependency-version: 1.13.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/s3shared
  dependency-version: 1.19.10
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.1
  dependency-type: indirect
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-20 21:27:11 +00:00
Ben McClellandandGitHub 69a3483269 Merge pull request #1592 from versity/sis/bucket-object-tag-validation 
fix: fixes the bucket/object tagging key/value name validation
 2025-10-20 12:21:01 -07:00
Ben McClellandandGitHub d256ea5929 Merge pull request #1589 from versity/sis/complete-mp-composite-checksum 
fix: fixes the composite checksums in CompleteMultipartUpload
 2025-10-20 09:25:17 -07:00
niksis02 ebf7a030cc fix: fixes the bucket/object tagging key/value name validation 
Fixes #1579

S3 enforces a specific rule for validating bucket and object tag key/value names. This PR integrates the regexp pattern used by S3 for tag validation.
Official S3 documentation for tag validation rules: [AWS S3 Tag](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_Tag.html)

There are two types of tagging inputs for buckets and objects:

1. **On existing buckets/objects** — used in the `PutObjectTagging` and `PutBucketTagging` actions, where tags are provided in the request body.
2. **On object creation** — used in the `PutObject`, `CreateMultipartUpload`, and `CopyObject` actions, where tags are provided in the request headers and must be URL-encoded.

This implementation ensures correct validation for both types of tag inputs.
 2025-10-20 15:19:38 +04:00
Ben McClellandandGitHub 2bf4ccd244 Merge pull request #1580 from versity/sis/complete-mp-default-checksum 
fix: sets crc64nvme as defualt checksum for complete mp action
 2025-10-17 21:49:51 -07:00
Ben McClellandandGitHub 7e44a5e703 Merge pull request #1590 from versity/sis/malformed-requests 
fix: implements proper error handling for malformed http requests
 2025-10-17 21:44:54 -07:00
Ben McClellandandGitHub b5bea5b659 Merge pull request #1591 from madic-creates/ldap-allow-selfsigned 
Allow self-signed ldaps certificates
 2025-10-17 21:43:22 -07:00
Ben McClellandandGitHub 2a4a0f000a Merge pull request #1553 from versity/test/direct_tests_three 
test: update tests to use new bucket creation function
 2025-10-17 21:42:34 -07:00
madic-creates 2dd442c24d Allow self-signed certificates 2025-10-17 17:37:35 +02:00
niksis02 27dc84b5fd fix: implements proper error handling for malformed http requests 
Fixes #1364

When a completely malformed request is sent to the gateway, Fiber/Fasthttp fails to parse the request, and the code execution never reaches the routers or handlers. Instead, the error is caught by the global error handler. These kinds of errors (malformed requests that fail during request parsing) are prefixed with **"error when reading request headers"** in Fiber. The implementation adds a check in the global error handler for this specific error message and returns an S3-like XML **BadRequest** error instead.
 2025-10-17 19:19:47 +04:00
niksis02 932f1c9da7 fix: sets crc64nvme as defualt checksum for complete mp action 
Fixes #1547

When no checksum is specified during multipart upload initialization, the complete multipart upload request should default to **CRC64NVME FULL_OBJECT**. The checksum will not be stored in the final object metadata, as it is used solely for data integrity verification. Note that although CRC64NVME is composable, it is calculated using the standard hash reader, since the part checksums are missing and the final checksum calculation is instead based directly on the parts data.
 2025-10-17 17:18:29 +04:00
niksis02 24679a82ac fix: fixes the composite checksums in CompleteMultipartUpload 
Fixes #1359

The composite checksums in **CompleteMultipartUpload** generally follow the format `checksum-<number_of_parts>`. Previously, the gateway treated composite checksums as regular checksums without distinguishing between the two formats.

In S3, the `x-amz-checksum-*` headers accept both plain checksum values and the `checksum-<number_of_parts>` format. However, after a successful `CompleteMultipartUpload` request, the final checksum is always stored with the part number included.

This implementation adds support for parsing both formats—checksums with and without the part number. From now on, composite checksums are consistently stored with the part number included.

Additionally, two integration tests are added:

* One verifies the final composite checksum with part numbers.
* Another ensures invalid composite checksums are correctly rejected.
 2025-10-17 16:45:07 +04:00
Ben McClellandandGitHub fa2023c18f Merge pull request #1586 from versity/sis/mp-checksum-error 
fix: fixes the checksum type/algo mismatch error in create mp
 2025-10-15 14:36:06 -07:00
Ben McClellandandGitHub bfe090df38 Merge pull request #1585 from versity/sis/expected-bucket-owner 
feat: adds the x-amz-expected-bucket-owner check in the gateway
 2025-10-15 14:35:07 -07:00
Ben McClellandandGitHub 9c6a09260a Merge pull request #1584 from versity/sis/response-headers-case 
fix: fixes the response header names normalizing
 2025-10-15 14:34:02 -07:00
Luke McCrone 6ea9950ead test: more dockerfile/direct updates, testing, fixes 2025-10-15 17:30:27 -03:00
niksis02 5bc6852f2c fix: fixes the checksum type/algo mismatch error in create mp 
Fixes #1329

Fixes the checksum type/algorithm mismatch error in `CreateMultipartUpload`. The algorithm an type were messed in the error description. It also adds an integration test to target the unsupported checksum type/algorithm pairs.
 2025-10-15 23:22:06 +04:00
niksis02 d39685947d feat: adds the x-amz-expected-bucket-owner check in the gateway 
Fixes #1428

The `x-amz-expected-bucket-owner` header in S3 specifies the account ID of the expected bucket owner. If the account ID provided does not match the actual owner of the bucket, the request fails with an HTTP 403 Forbidden (AccessDenied) error. If the provided account ID is not 12 characters long, S3 returns a 400 Bad Request error.

In our case, we expect the header to contain the bucket owner’s access key ID, and we skip validation errors related to the access key ID, since there is no validation mechanism for user access key IDs. If the provided value does not match the bucket owner’s access key ID, the gateway returns an AccessDenied error.

A few integration tests are added for random actions, as this feature applies to all actions, but it is unnecessary to add test cases for every single one.
 2025-10-15 19:20:04 +04:00
Ben McClellandandGitHub af550c8f80 Merge pull request #1581 from versity/sis/bucket-policy-version 
feat: adds bucket policy version support
 2025-10-14 14:42:43 -07:00
niksis02 d15d348226 fix: fixes the response header names normalizing 
Fixes #1484

Removes response header name normalization to prevent Fiber from converting them to camel case. Also fixes the `HeadBucket` response headers by changing their capital letters to lowercase and corrects the `x-amz-meta` headers to use lowercase instead of camel case.
 2025-10-15 01:27:53 +04:00
Ben McClellandandGitHub 66e29d4aa4 Merge pull request #1582 from versity/dependabot/go_modules/dev-dependencies-47300381c3 
chore(deps): bump the dev-dependencies group with 11 updates
 2025-10-14 10:09:16 -07:00
dependabot[bot]andGitHub cdc4358257 chore(deps): bump the dev-dependencies group with 11 updates 
Bumps the dev-dependencies group with 11 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) | `1.12.0` | `1.13.0` |
| [github.com/DataDog/datadog-go/v5](https://github.com/DataDog/datadog-go) | `5.8.0` | `5.8.1` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.88.3` | `1.88.4` |
| [github.com/valyala/fasthttp](https://github.com/valyala/fasthttp) | `1.66.0` | `1.67.0` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.36.0` | `0.37.0` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.42.0` | `0.43.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.44.0` | `0.45.0` |
| [golang.org/x/text](https://github.com/golang/text) | `0.29.0` | `0.30.0` |
| [golang.org/x/time](https://github.com/golang/time) | `0.13.0` | `0.14.0` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.19.11` | `1.19.12` |
| [github.com/aws/aws-sdk-go-v2/service/internal/checksum](https://github.com/aws/aws-sdk-go-v2) | `1.8.9` | `1.9.0` |


Updates `github.com/Azure/azure-sdk-for-go/sdk/azidentity` from 1.12.0 to 1.13.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/sdk-breaking-changes-guide-migration.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.12.0...sdk/azcore/v1.13.0)

Updates `github.com/DataDog/datadog-go/v5` from 5.8.0 to 5.8.1
- [Release notes](https://github.com/DataDog/datadog-go/releases)
- [Changelog](https://github.com/DataDog/datadog-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/DataDog/datadog-go/compare/v5.8.0...v5.8.1)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.88.3 to 1.88.4
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.88.3...service/s3/v1.88.4)

Updates `github.com/valyala/fasthttp` from 1.66.0 to 1.67.0
- [Release notes](https://github.com/valyala/fasthttp/releases)
- [Commits](https://github.com/valyala/fasthttp/compare/v1.66.0...v1.67.0)

Updates `golang.org/x/sys` from 0.36.0 to 0.37.0
- [Commits](https://github.com/golang/sys/compare/v0.36.0...v0.37.0)

Updates `golang.org/x/crypto` from 0.42.0 to 0.43.0
- [Commits](https://github.com/golang/crypto/compare/v0.42.0...v0.43.0)

Updates `golang.org/x/net` from 0.44.0 to 0.45.0
- [Commits](https://github.com/golang/net/compare/v0.44.0...v0.45.0)

Updates `golang.org/x/text` from 0.29.0 to 0.30.0
- [Release notes](https://github.com/golang/text/releases)
- [Commits](https://github.com/golang/text/compare/v0.29.0...v0.30.0)

Updates `golang.org/x/time` from 0.13.0 to 0.14.0
- [Commits](https://github.com/golang/time/compare/v0.13.0...v0.14.0)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.19.11 to 1.19.12
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/sqs/v1.19.11...service/efs/v1.19.12)

Updates `github.com/aws/aws-sdk-go-v2/service/internal/checksum` from 1.8.9 to 1.9.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/v1.9.0/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/tnb/v1.8.9...v1.9.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-version: 1.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/DataDog/datadog-go/v5
  dependency-version: 5.8.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.88.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/valyala/fasthttp
  dependency-version: 1.67.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/sys
  dependency-version: 0.37.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/crypto
  dependency-version: 0.43.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.45.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/text
  dependency-version: 0.30.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: golang.org/x/time
  dependency-version: 0.14.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-version: 1.19.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/checksum
  dependency-version: 1.9.0
  dependency-type: indirect
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-13 21:09:45 +00:00
niksis02 ce6193b191 feat: adds bucket policy version support 
Closes #1536

Adds bucket policy version support. Two versions are supported: **2008-10-17** and **2012-10-17**. If the `Version` field is omitted in the bucket policy document, it defaults to **2008-10-17**. However, if an empty string (`""`) is provided, it is considered invalid.
 2025-10-13 21:39:04 +04:00
Ben McClellandandGitHub 8bb22debad Merge pull request #1578 from versity/sis/fiber-server-graceful-shutdown 
feat: graceful shutdown of s3api and admin servers
 2025-10-11 13:48:25 -07:00
niksis02 64f50cc504 feat: gracul shutdown of s3api and admin servers 
Implements graceful shutdown for the admin and s3api servers. They are shut down before other components (IAM, s3logger, etc.) to allow the servers to properly handle any pending requests while dependencies are still active. The shutdown process is controlled by a context with a 10-second timeout. If it exceeds this duration, all remaining requests are forcefully terminated and the servers are closed.
 2025-10-10 22:10:56 +04:00
Ben McClellandandGitHub e2534afafe Merge pull request #1576 from versity/ben/cleanup 
chore: cleanup unused constants
 2025-10-10 09:54:02 -07:00
Ben McClellandandGitHub 341d51107c Merge pull request #1577 from versity/sis/overwrite-locked-objects 
feat: prevents locked objects overwrite with CopyObject and CompleteMultipartUpload
 2025-10-10 09:53:41 -07:00
niksis02 707af47769 feat: prevents locked objects overwrite with CopyObject and CompleteMultipartUpload 
Closes #1566

When an object is locked and bucket versioning is not configured at the gateway level, any object overwrite request should be rejected with an object locked error. The `PutObject` operation already follows this behavior, but `CopyObject` and `CompleteMultipartUpload` were missing this check. This change introduces the locking mechanism for `CopyObject` and `CompleteMultipartUpload` operations.
 2025-10-10 00:39:12 +04:00
Ben McClelland 40da4a31d3 chore: cleanup unused constants 
We have some leftover constants from some previous changes. This
just cleans up all that are no longer needed.
 2025-10-09 12:19:00 -07:00
Ben McClellandandGitHub 874165cdcf Merge pull request #1575 from versity/sis/locked-objects-overwrite 
fix: fixes locked objects overwrite in versioning-enabled buckets
 2025-10-09 10:55:47 -07:00
Ben McClellandandGitHub e750cf9718 Merge pull request #1573 from versity/ben/relax-bucket-check 
feat: add option to disable strict bucket name checks
 2025-10-09 08:10:51 -07:00
Luke McCrone c158dfeb0d test: temp fixes for legal-hold/retention tests 2025-10-08 18:52:04 -03:00
Ben McClelland 4c3965d87e feat: add option to disable strict bucket name checks 
Some systems may choose to allow non-aws compliant bucket names
and/or handle the bucket naem validation in the backend instead.
This adds the option to turn off the strict bucket name validation
checks in the frontend API handlers.

When frontend bucket name validation is disabled, we need to do
sanity checks for posix compliant names in the posix/scoutfs
backends. This is automatically enabled when strict bucket
name validation is disabled.

Fixes #1564
 2025-10-08 14:34:52 -07:00
niksis02 5c084b8452 fix: fixes locked objects overwrite in versioning-enabled buckets 
Fixes #1574

When versioning is enabled at the gateway level and object lock is enabled for a bucket, any overwrite request on a locked object should succeed since it results in the creation of a new object version. This PR fixes the logic by adding a bucket versioning status check in `CheckObjectAccess`.
 2025-10-09 01:01:18 +04:00
Ben McClellandandGitHub 78cf20075f Merge pull request #1572 from versity/sis/concurrent-integration-tests 
feat: concurrent execution of integration tests
 2025-10-08 08:09:48 -07:00
niksis02 a4dc837f54 feat: concurrent execution of integration tests 
This change introduces concurrent execution for integration tests. It adds a mechanism to run tests either synchronously or in parallel, controlled by a new flag. By default, tests continue to run in synchronous mode to maintain predictable behavior during local development. In GitHub Actions, the tests are now executed in parallel mode to significantly reduce overall runtime.

The implementation uses a semaphore-based concurrency control to limit the number of parallel test executions and ensures graceful shutdown through context cancellation. This approach improves test performance while keeping the system stable and backward compatible.
 2025-10-07 23:22:08 +04:00
Ben McClellandandGitHub a7d83b42fd Merge pull request #1571 from versity/dependabot/go_modules/dev-dependencies-58d9c26195 
chore(deps): bump the dev-dependencies group with 3 updates
 2025-10-06 16:16:52 -07:00
Ben McClellandandGitHub 54bd4ec841 Merge pull request #1570 from dtufood-kihen/vault-namespace 
auth/vault: add Vault namespace support
 2025-10-06 16:16:25 -07:00
dependabot[bot]andGitHub 9ae68076c1 chore(deps): bump the dev-dependencies group with 3 updates 
Bumps the dev-dependencies group with 3 updates: [github.com/go-ldap/ldap/v3](https://github.com/go-ldap/ldap), [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) and [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2).


Updates `github.com/go-ldap/ldap/v3` from 3.4.11 to 3.4.12
- [Release notes](https://github.com/go-ldap/ldap/releases)
- [Commits](https://github.com/go-ldap/ldap/compare/v3.4.11...v3.4.12)

Updates `github.com/nats-io/nats.go` from 1.46.0 to 1.46.1
- [Release notes](https://github.com/nats-io/nats.go/releases)
- [Commits](https://github.com/nats-io/nats.go/compare/v1.46.0...v1.46.1)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.19.10 to 1.19.11
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/iam/v1.19.10...service/sqs/v1.19.11)

---
updated-dependencies:
- dependency-name: github.com/go-ldap/ldap/v3
  dependency-version: 3.4.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/nats-io/nats.go
  dependency-version: 1.46.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-version: 1.19.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-10-06 21:07:33 +00:00
Ben McClellandandGitHub 13fdbaf35a Merge pull request #1569 from versity/ben/docker 2025-10-03 22:14:00 -07:00
Kim Henriksen 45f55c2283 auth/vault: add Vault namespace support 
New CLI flags:
- --iam-vault-namespace
- --iam-vault-auth-namespace
- --iam-vault-secret-storage-namespace

Behavior:
- Auth requests use the auth namespace
- KV operations use the secret storage namespace
- If a specific namespace is not set, the shared namespace is used
- With AppRole, different auth and secret namespaces are rejected
 2025-10-03 23:33:34 +02:00
Ben McClelland 7aa733ae9e feat: use docker entrypoint for flexible env var docker config 
Using Docker ENTRYPOINT should allow for configuration of running
versitygw within Docker container similar to how the systemd
service is setup with environment variables.

This also adds the backends azure and plugin to the acceptable
backend options for both docker and systemd.

Fixes #1335
 2025-10-03 09:34:51 -07:00
Ben McClellandandGitHub bef297f6ad Merge pull request #1552 from versity/test/direct_tests_two 
Test/direct tests two
 2025-10-03 08:11:04 -07:00
Luke McCrone 25cde72fa3 test: more dockerfile/direct updates, testing, fixes 2025-10-03 11:12:58 -03:00
Ben McClellandandGitHub 48f438b1e0 Merge pull request #1568 from versity/sis/object-lock-on-existing-buckets 
fix: correct a few object lock behaviors
 2025-10-02 15:10:25 -07:00