patches: add v1.11.0 owner-scoped access patch and its leil-rig proof
0003-owner-scoped-access.patch changes authorization, applied after the de-brand and dark-theme patches: - a bucket's ACL and policy may name only the owner's keys (accounts on the owner's UserID), whoever writes them, root included; - public access is read-only: all-users READ, and "*" allowed only s3:GetObject and s3:ListBucket; - the bucket policy never denies the owner's key; - DeleteBucket, PutBucketVersioning, PutObjectLockConfiguration and PutBucketOwnershipControls are reserved to root and admins; - an ACL with the all-users READ grant can be written back unchanged. docs/owner-scoped-access.md has the rules, what v1.11.0 already did, and the leil-rig proof procedure with its 2026-11-10 result: 63 of 63 on the patched build, 29 of 63 failing on stock v1.11.0. tests/owner-scope/ holds the proof script and a stdlib-only SigV4 request helper. README and the pipeline doc list the patch and how to re-verify it after a rebase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
57_WolveandClaude Opus 5.5 committed
1 parent
1984762186
commit
152ea01107
6 files changed
+1963
-6
No files matched your search
@@ -14,6 +14,13 @@ Artifact names are intentionally **unchanged** — the binary stays
|
||||
drop-in compatible with existing configs, systemd units, and the
|
||||
`versitygw` CLI. Only the embedded web UI is rebranded to "S3 Gateway".
|
||||
|
||||
The v1.11.0 series also carries one behavioural patch,
|
||||
**owner-scoped bucket access**: a bucket's ACL and policy may name only
|
||||
the owner's own keys, public access is read-only, and deleting a bucket
|
||||
or changing its versioning, object lock or ownership controls is left
|
||||
to the gateway's root and admins. See
|
||||
[docs/owner-scoped-access.md](docs/owner-scoped-access.md).
|
||||
|
||||
```
|
||||
upstream tag (vX.Y.Z) this repo
|
||||
github.com/versity/versitygw patches/<tag>/*.patch
|
||||
@@ -40,15 +47,19 @@ s3-gateway/
|
||||
├── patches/
|
||||
│ ├── v1.11.0/ # the supported series
|
||||
│ │ ├── 0001-webui-debrand.patch
|
||||
│ │ └── 0002-webui-dark-theme.patch
|
||||
│ │ ├── 0002-webui-dark-theme.patch
|
||||
│ │ └── 0003-owner-scoped-access.patch
|
||||
│ └── archive/ # superseded; kept, not built
|
||||
│ ├── v1.10.0/
|
||||
│ ├── v1.8.0/
|
||||
│ ├── v1.7.0/
|
||||
│ ├── v1.6.0/
|
||||
│ └── v1.5.0/
|
||||
├── tests/owner-scope/ # leil-rig proof test for 0003 (proof.sh, s3req.py)
|
||||
├── .gitea/workflows/build.yml
|
||||
└── docs/multi-version-pipeline.md
|
||||
└── docs/
|
||||
├── multi-version-pipeline.md
|
||||
└── owner-scoped-access.md # what 0003 does, and its proof test
|
||||
```
|
||||
|
||||
## Build locally
|
||||
@@ -86,9 +97,9 @@ publish a build of a superseded release.
|
||||
|
||||
Currently maintained:
|
||||
|
||||
| Upstream tag | Patch set | Contents |
|
||||
|--------------|---------------|-----------------------------------------------------------|
|
||||
| `v1.11.0` | `v1.11.0/` | `0001-webui-debrand.patch`, `0002-webui-dark-theme.patch` |
|
||||
| Upstream tag | Patch set | Contents |
|
||||
|--------------|---------------|---------------------------------------------------------------------------------------------|
|
||||
| `v1.11.0` | `v1.11.0/` | `0001-webui-debrand.patch`, `0002-webui-dark-theme.patch`, `0003-owner-scoped-access.patch` |
|
||||
|
||||
**v1.11.0 is the only supported version.** It supersedes `v1.10.0`
|
||||
with 42 further upstream fixes, so the older series — `v1.10.0`,
|
||||
@@ -141,6 +152,35 @@ The only markup change is the login page's light background gradient,
|
||||
which becomes a solid dark surface. It is part of the supported
|
||||
`v1.11.0` series, and of every archived set except `v1.5.0`.
|
||||
|
||||
### What the owner-scoped access patch does
|
||||
|
||||
`0003-owner-scoped-access.patch` changes authorization in `auth/` (plus
|
||||
one call in `s3api/controllers/bucket-put.go`). A bucket's owner is the
|
||||
account its ACL names; the owner's *keys* are every account with that
|
||||
account's `UserID`.
|
||||
|
||||
- **Only the owner's keys.** `PutBucketPolicy`, `PutBucketAcl` (body
|
||||
and grant headers) and `CreateBucket`'s grant headers — so also
|
||||
`versitygw admin create-bucket --grant-*` — refuse any account whose
|
||||
`UserID` is not the owner's. Root is refused too.
|
||||
- **Public means read.** `all-users` gets `READ` only; a policy may
|
||||
allow `"*"` only `s3:GetObject` and `s3:ListBucket`.
|
||||
`public-read-write` and every other public grant are refused.
|
||||
- **A policy adds to the ACL.** An `Allow` from either is enough and a
|
||||
policy `Deny` wins — except that the owner's key is never denied.
|
||||
- **Root/admin only:** `DeleteBucket`, `PutBucketVersioning`,
|
||||
`PutObjectLockConfiguration`, `PutBucketOwnershipControls` (and
|
||||
`DeleteBucketOwnershipControls`).
|
||||
- An ACL holding the `all-users` `READ` grant can be written back
|
||||
unchanged (upstream answers `500`).
|
||||
|
||||
It is the series' only patch outside `webui/`, and it has unit tests:
|
||||
after a rebase run `go test ./auth/ ./s3api/...` (on Linux — on Windows
|
||||
upstream's standalone-IAM, posix and CLI tests fail with or without the
|
||||
series) and
|
||||
the leil-rig proof test in
|
||||
[docs/owner-scoped-access.md](docs/owner-scoped-access.md#proving-the-patch-is-in-on-leil-rig).
|
||||
|
||||
## Adding / refreshing a version
|
||||
|
||||
When upstream cuts a new release you want a build of, you rebase the
|
||||
|
||||
@@ -110,7 +110,24 @@ When upstream releases `vX.Y.Z`:
|
||||
VGW_REPO_URL="/path/to/local/versitygw" VGW_TAG=vX.Y.Z bash ci/run.sh
|
||||
```
|
||||
|
||||
A clean `git am` plus a populated `src/dist/` means the series is good.
|
||||
A clean `git am` plus a populated `src/dist/` means the web UI patches
|
||||
are good. `0003-owner-scoped-access.patch` changes Go code in `auth/`, so
|
||||
a clean apply is not enough for it: run its unit tests in the patched
|
||||
tree, on Linux (on Windows several upstream packages fail regardless),
|
||||
|
||||
```bash
|
||||
go test ./auth/ ./s3api/...
|
||||
```
|
||||
|
||||
and its proof test on leil-rig — see
|
||||
[owner-scoped-access.md](owner-scoped-access.md#proving-the-patch-is-in-on-leil-rig).
|
||||
If upstream has reworked `auth/access-control.go` or `auth/acl.go`,
|
||||
re-read that document's rules against the new code rather than only
|
||||
resolving the conflict.
|
||||
|
||||
`src/` may hold someone's local work tree; `ci/clone-source.sh` deletes
|
||||
it. Point `VGW_SRC_DIR` somewhere fresh (any `*-src/` name is
|
||||
gitignored) when in doubt.
|
||||
|
||||
## Retiring a version
|
||||
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# Owner-scoped bucket access (`0003-owner-scoped-access.patch`)
|
||||
|
||||
`patches/v1.11.0/0003-owner-scoped-access.patch` applies after the de-brand and dark-theme patches. Unlike them it
|
||||
changes the gateway's behaviour, not its look: who may be named in a bucket's ACL and policy, what the public may be
|
||||
given, and which bucket settings only the gateway operator may change.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Touches | `auth/acl.go`, `auth/access-control.go`, `auth/owner_scope.go` (new), `s3api/controllers/bucket-put.go`, and tests |
|
||||
| Base | upstream `v1.11.0` (`e42ae7ab`) + `0001` + `0002` |
|
||||
| Unit tests | `go test ./auth/ ./s3api/...` (table-driven; `owner_scope_test.go` in `auth` and `s3api/controllers`) |
|
||||
| Proof test | [`tests/owner-scope/proof.sh`](../tests/owner-scope/proof.sh), on leil-rig, below |
|
||||
|
||||
## The rules
|
||||
|
||||
A bucket belongs to the account its ACL names as owner — the key in the `user.acl` xattr — and to the person behind
|
||||
that account, who may hold several keys: every key whose IAM account carries the owner account's `UserID`. Sharing a
|
||||
bucket means handing someone one of those keys, never naming somebody else's.
|
||||
|
||||
| Rule | Where | Refused with |
|
||||
|---|---|---|
|
||||
| **Only the owner's keys.** Every account named by `PutBucketPolicy`, `PutBucketAcl` (body and `x-amz-grant-*` headers) and `CreateBucket`'s grant headers — so also `versitygw admin create-bucket --grant-*` — must have the owner's `UserID`. The owner is looked up in IAM too: an owner unknown to IAM, or with `UserID` 0, can name no other key. The check is on the document, not the caller: root is refused too. | `auth.UpdateACL`, `auth.VerifyPolicyOwnerScope` | `403 AccessDenied` *"<key> is not an account of the bucket owner <owner>"* |
|
||||
| **Public means read.** `all-users` may only be granted `READ`; a policy may `Allow` `"*"` only `s3:GetObject` and `s3:ListBucket`. Canned `public-read-write`, any other `all-users` grant and any other public `Allow` — wildcards such as `s3:Get*` included — are refused. A `Deny` for `"*"` grants nothing and is accepted. | same | `403 AccessDenied` *"public access is read-only: …"* |
|
||||
| **A policy adds to the ACL.** An `Allow` from either is enough; a policy `Deny` wins — except that the bucket owner's key is never denied by the bucket policy. The owner's *other* keys get no exemption. | `auth.verifyResourceAccess` | — |
|
||||
| **Gateway root/admin only:** `DeleteBucket`, `PutBucketVersioning`, `PutObjectLockConfiguration`, `PutBucketOwnershipControls` (and `DeleteBucketOwnershipControls`, which authorizes as it). Refused to the owner, to grantees, to anything a policy grants and to public requests, before any ACL or policy is read. | `auth.verifyAccessGates` | `403 AccessDenied` *"s3:DeleteBucket is reserved to the gateway's root and admin accounts"* |
|
||||
| An ACL holding the `all-users` `READ` grant, exactly as `GetBucketAcl` returns it, can be written back with `PutBucketAcl`. Upstream looks `all-users` up as an account and answers `500`. | `auth.UpdateACL` | — |
|
||||
|
||||
### What v1.11.0 already did, and what is new
|
||||
|
||||
The request was written against a checkout of upstream `main` at `4dc0debf`. That checkout is **older** than
|
||||
v1.11.0, not newer — `git describe` gives `v1.8.0-78-g4dc0debf` and v1.11.0 is not in its history — and v1.11.0 has
|
||||
already reworked the code it describes:
|
||||
|
||||
- *"`verifyResourceAccess` skips the ACL once a policy exists"* — true at `4dc0debf`, not in v1.11.0. There a policy
|
||||
statement that does not match falls back to the ACL, and a policy `Deny` wins, so "a policy adds to the ACL" was
|
||||
already the behaviour. The proof test's control run below confirms it: a key named only in a policy works on stock
|
||||
v1.11.0.
|
||||
- *"`verifyBucketPolicy` gives the owner no exemption"* — v1.11.0 exempts the owner only for reading, replacing and
|
||||
deleting the policy itself (`isBucketPolicyAction`). A `Deny` naming the owner, or `"*"`, still locks the owner
|
||||
out of everything else. 0003 widens the exemption to every action and removes `isBucketPolicyAction`.
|
||||
- `PutBucketOwnershipControls`, `PutBucketVersioning` and `PutObjectLockConfiguration` live in
|
||||
`s3api/controllers/bucket-put.go` as stated; `DeleteBucket` and `DeleteBucketOwnershipControls` are in
|
||||
`bucket-delete.go`. The gate is in `auth`, so the controllers are unchanged.
|
||||
|
||||
### Edges worth knowing
|
||||
|
||||
- **ARN principals** (the standalone IAM service, `PrincipalResolver`) skip the per-principal `UserID` check: that
|
||||
backend fixes every bucket's owner to root and gives every identity root's configured `UserID`, so the rule holds
|
||||
by construction. The public read-only rule still applies.
|
||||
- **The check is at write time.** A policy or ACL stored before this build keeps working as stored. The
|
||||
root/admin-only gate is the exception: it is evaluated per request, so a stored grant of those four actions stops
|
||||
working at once.
|
||||
- **An unknown grantee** still gets upstream's error (`accounts does not exist: …`, a `500`); 0003 only adds refusals.
|
||||
- **Writing `user.acl` directly** (leil-cluster's `leilfs-user` appends same-UID grantees with `setfattr`) bypasses
|
||||
the gateway and so this check. Today it only ever adds the user's own keys, which the rule allows.
|
||||
- **Bucket creation** still lets a `userplus` creator set object lock and object ownership on its own new bucket;
|
||||
only changing them afterwards is reserved.
|
||||
|
||||
## Proving the patch is in (on leil-rig)
|
||||
|
||||
`--version` reads `1.11.0` with or without the patch, so the proof is behaviour. `tests/owner-scope/proof.sh` starts
|
||||
a throwaway gateway from a given binary on `127.0.0.1:7390`, with its own internal IAM and posix root under
|
||||
`/var/tmp/s3gw-owner-scope`, drives every rule through signed S3 requests (`tests/owner-scope/s3req.py`, Python
|
||||
standard library only, so nothing is installed), and deletes the lot again. Nothing installed is touched — no
|
||||
package, unit or config — so it does not disturb whichever `versitygw` the rig is pinned to.
|
||||
|
||||
It creates four accounts through the admin API, all on leil-cluster's reserved test band:
|
||||
|
||||
| Key | UID | Role | Is |
|
||||
|---|---|---|---|
|
||||
| `po-owner` | 74990 | userplus | owner of bucket `proof` (`create-bucket --owner po-owner --object-ownership BucketOwnerPreferred --grant-full-control po-full`) |
|
||||
| `po-full` | 74990 | user | a `FULL_CONTROL` grantee — one of the owner's keys |
|
||||
| `po-obj` | 74990 | user | in no ACL; named in one policy statement allowing `s3:GetObject`/`PutObject`/`DeleteObject` on `shared/report.txt` |
|
||||
| `po-foreign` | 74991 | user | somebody else's key |
|
||||
|
||||
Take the lease first, and give it back when done (`rig-lease.sh` is in `../leil-rig`):
|
||||
|
||||
```bash
|
||||
RIG_LEASE_WHO="<you>: s3-gateway 0003 proof" ./bin/rig-lease.sh claim --why "s3-gateway 0003 proof on leil-test-1" --minutes 60
|
||||
```
|
||||
|
||||
Build, then copy the binary and the test to the test node (it runs no cluster service, and has `python3`):
|
||||
|
||||
```bash
|
||||
VGW_SRC_DIR=build-src VGW_TAG=v1.11.0 bash ci/run.sh
|
||||
cd ../leil-rig && source bin/rig-lib.sh
|
||||
on leil-test-1 'install -d -o leil -g leil /var/tmp/s3gw-0003'
|
||||
rig_scp ../s3-gateway/build-src/dist/versitygw_linux_amd64_v1/versitygw leil-test-1 /var/tmp/s3gw-0003/versitygw
|
||||
tar -czf /tmp/owner-scope.tgz -C ../s3-gateway/tests owner-scope
|
||||
rig_scp /tmp/owner-scope.tgz leil-test-1 /var/tmp/s3gw-0003/owner-scope.tgz
|
||||
on leil-test-1 'cd /var/tmp/s3gw-0003 && chmod 0755 versitygw && tar -xzf owner-scope.tgz && bash owner-scope/proof.sh $PWD/versitygw'
|
||||
```
|
||||
|
||||
(`VGW_SRC_DIR` keeps `ci/run.sh` away from `src/`, which it would `rm -rf`.) Every line must read `PASS` and the last
|
||||
`N passed, 0 failed`; the exit status says the same. `KEEP=1` leaves the gateway directory and its log behind,
|
||||
`PROOF_PORT` moves it off 7390. What it covers, in order:
|
||||
|
||||
| Checks | Shows |
|
||||
|---|---|
|
||||
| A1–A6 | a foreign key refused as a grantee — by `admin create-bucket --grant-full-control` as root (and no bucket made), by `PutBucketAcl` grant header as the owner and as root, by `PutBucketAcl` body as root — while the owner's own key is granted |
|
||||
| B1–B2 | a foreign key refused as a policy principal, as the owner and as root |
|
||||
| C1–C8 | `"*"` `s3:PutObject`, canned `public-read-write` and an `all-users` `WRITE` body refused; a policy allowing `"*"` `s3:GetObject` + `s3:ListBucket` accepted, after which anonymous `GetObject` and `ListBucket` work and anonymous `PutObject` does not |
|
||||
| D1–D4 | the owner key still writes, reads, deletes and reads the ACL under that public-read-only policy |
|
||||
| E1–E8 | the owner key still writes, reads, lists and deletes under a policy denying `po-owner` `s3:*` and `"*"` `s3:PutObject`, while the same `Deny` beats `po-full`'s `FULL_CONTROL` grant; the owner then deletes the policy |
|
||||
| F1–F12 | `po-obj` puts, gets and deletes `shared/report.txt` and nothing else: no other object, no listing, no ACL or policy read or write |
|
||||
| G0–G11 | with the owner's policy granting `po-full` all four, `po-full` is refused ownership controls (put and delete), versioning, object lock and `DeleteBucket`; so is the owner; `po-full`'s object grant still works, and root changes ownership controls |
|
||||
| H1–H5 | after canned `public-read`, the ACL `GetBucketAcl` returns is written back unchanged and still grants `all-users` `READ` |
|
||||
| Z1 | root deletes the bucket |
|
||||
|
||||
Release the lease and clean up: `on leil-test-1 'rm -rf /var/tmp/s3gw-0003'`, then `./bin/rig-lease.sh release`.
|
||||
|
||||
### Result on leil-rig, 2026-11-10: **PASSED**
|
||||
|
||||
Run on `leil-test-1` (AlmaLinux 9.8, kernel 5.14.0-687.53.1, Python 3.9.25) under the lease, from a `ci/run.sh`
|
||||
build of this series:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Build | `VGW_TAG=v1.11.0 bash ci/run.sh` (fresh clone of `v1.11.0`, 0001–0003 by `git am --3way`, GoReleaser); `--version` 1.11.0, `Build` `98b6c69b` |
|
||||
| `versitygw` (linux amd64) sha256 | `4e06d35b746602ce4c1de4a3f3c307c4648faa2f058625f72305a8848002bfd4` |
|
||||
| `versitygw_1.11.0_linux_amd64.rpm` sha256 | `e4b0e77ecf003e6ec1d57757d841941ded0bcbd9175e668a6794c0a1c6bf33ba` (built locally; not published, not pinned) |
|
||||
| `proof.sh` | **63 passed, 0 failed** |
|
||||
| Unit tests on the node (go1.27.3 linux/amd64) | `go test` of every package except `plugins/`: 35 of 35 ok, among them `auth` and `s3api/...` |
|
||||
|
||||
**Control: stock v1.11.0** (0001 + 0002 only, same script, same node): **34 passed, 29 failed.** Root and the owner
|
||||
named `po-foreign` as grantee and as principal (A1–A5, B1–B2); `"*"` `PutObject` and `public-read-write` were
|
||||
accepted, and anonymous `PutObject` then succeeded (C1, C2, C8); an `all-users` `WRITE` body and the `GetBucketAcl`
|
||||
round trip answered `500` (C3, H3); the owner's own `Deny` locked it out of put, get, list and delete (E2–E5); with
|
||||
the bucket public-read-write, `po-obj` reached objects beyond its one (F4–F7); and the policy grant let `po-full` —
|
||||
and the owner — change ownership controls, enable object lock, and reach versioning and `DeleteBucket` (G1–G8, which
|
||||
failed only on "versioning not configured" and "bucket not empty"). Stock v1.11.0 passed F1–F3 and F8–F12 on its own:
|
||||
the single-object key already worked there; 0003 keeps it working.
|
||||
|
||||
Run this test again before pinning any build that carries 0003, in leil-rig or leil-cluster.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,212 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Proof test for patches/v1.11.0/0003-owner-scoped-access.patch.
|
||||
#
|
||||
# sudo bash proof.sh /path/to/versitygw
|
||||
#
|
||||
# Starts a throwaway gateway from the given binary on 127.0.0.1:${PROOF_PORT:-7390},
|
||||
# with its own internal IAM and posix root under ${PROOF_DIR:-/var/tmp/s3gw-owner-scope},
|
||||
# drives every rule in the patch through real S3 requests, and removes the lot
|
||||
# again. Nothing installed is touched: no package, no unit, no config. Prints one
|
||||
# line per check and exits non-zero if any check fails. KEEP=1 leaves the gateway
|
||||
# directory (and its log) behind for a look.
|
||||
#
|
||||
# The accounts, all made by root through the admin API:
|
||||
#
|
||||
# po-owner uid 74990 userplus owns bucket "proof"
|
||||
# po-full uid 74990 user FULL_CONTROL grantee (an owner's key)
|
||||
# po-obj uid 74990 user in no ACL; one policy statement on one object
|
||||
# po-foreign uid 74991 user somebody else's key
|
||||
#
|
||||
# The UIDs are from leil-cluster's reserved test band (74990-74999); nothing
|
||||
# is chowned, so they only have to be distinct.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
VGW=${1:?usage: proof.sh /path/to/versitygw}
|
||||
HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PORT=${PROOF_PORT:-7390}
|
||||
W=${PROOF_DIR:-/var/tmp/s3gw-owner-scope}
|
||||
export S3_ENDPOINT=http://127.0.0.1:$PORT S3_REGION=us-east-1
|
||||
|
||||
[ -x "$VGW" ] || { echo "FATAL: $VGW is not an executable" >&2; exit 2; }
|
||||
command -v python3 >/dev/null || { echo "FATAL: python3 is required" >&2; exit 2; }
|
||||
if (exec 3<>"/dev/tcp/127.0.0.1/$PORT") 2>/dev/null; then
|
||||
echo "FATAL: something already listens on 127.0.0.1:$PORT (set PROOF_PORT)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
rm -rf "$W" && mkdir -p "$W/iam" "$W/data" "$W/tmp" || exit 2
|
||||
secret() { python3 -c 'import secrets; print(secrets.token_hex(16))'; }
|
||||
declare -A KEY SECRET
|
||||
KEY[root]=proofroot
|
||||
for who in root po-owner po-full po-obj po-foreign; do
|
||||
[ "$who" = root ] || KEY[$who]=$who
|
||||
SECRET[$who]=$(secret)
|
||||
done
|
||||
|
||||
"$VGW" --access "${KEY[root]}" --secret "${SECRET[root]}" --region "$S3_REGION" \
|
||||
--port "127.0.0.1:$PORT" --iam-dir "$W/iam" posix "$W/data" >"$W/gateway.log" 2>&1 &
|
||||
GW=$!
|
||||
cleanup() {
|
||||
kill "$GW" 2>/dev/null
|
||||
wait "$GW" 2>/dev/null
|
||||
[ "${KEEP:-0}" = 1 ] && echo "kept $W" || rm -rf "$W"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
for _ in $(seq 1 50); do
|
||||
(exec 3<>"/dev/tcp/127.0.0.1/$PORT") 2>/dev/null && break
|
||||
kill -0 "$GW" 2>/dev/null || { cat "$W/gateway.log"; echo "FATAL: the gateway exited" >&2; exit 2; }
|
||||
sleep 0.2
|
||||
done
|
||||
echo "gateway $("$VGW" --version 2>&1 | head -1) on 127.0.0.1:$PORT, sha256 $(sha256sum "$VGW" | cut -c1-16)"
|
||||
|
||||
admin() {
|
||||
ADMIN_ACCESS_KEY_ID=${KEY[root]} ADMIN_SECRET_ACCESS_KEY=${SECRET[root]} \
|
||||
"$VGW" admin --region "$S3_REGION" --endpoint-url "$S3_ENDPOINT" "$@"
|
||||
}
|
||||
|
||||
# req <who> <s3req.py args...> -- one request as <who> ("anon" for none); prints the status
|
||||
req() {
|
||||
local who=$1; shift
|
||||
if [ "$who" = anon ]; then
|
||||
S3_KEY= S3_SECRET= python3 "$HERE/s3req.py" "$@" 2>"$W/tmp/err"
|
||||
else
|
||||
S3_KEY=${KEY[$who]} S3_SECRET=${SECRET[$who]} python3 "$HERE/s3req.py" "$@" 2>"$W/tmp/err"
|
||||
fi
|
||||
}
|
||||
|
||||
PASS=0 FAIL=0
|
||||
# check <id> <want> <what> <who> <s3req.py args...>
|
||||
check() {
|
||||
local id=$1 want=$2 what=$3; shift 3
|
||||
local got
|
||||
got=$(req "$@")
|
||||
if [ "$got" = "$want" ]; then
|
||||
PASS=$((PASS + 1)); printf 'PASS %-4s %s %s\n' "$id" "$got" "$what"
|
||||
else
|
||||
FAIL=$((FAIL + 1)); printf 'FAIL %-4s %s (want %s) %s [%s]\n' "$id" "${got:-none}" "$want" "$what" "$(cat "$W/tmp/err")"
|
||||
fi
|
||||
}
|
||||
# check_cmd <id> <want: ok|refused> <what> <command...>
|
||||
check_cmd() {
|
||||
local id=$1 want=$2 what=$3; shift 3
|
||||
local got=ok
|
||||
"$@" >"$W/tmp/err" 2>&1 || got=refused
|
||||
if [ "$got" = "$want" ]; then
|
||||
PASS=$((PASS + 1)); printf 'PASS %-4s %s %s\n' "$id" "$got" "$what"
|
||||
else
|
||||
FAIL=$((FAIL + 1)); printf 'FAIL %-4s %s (want %s) %s [%s]\n' "$id" "$got" "$want" "$what" "$(tr '\n' ' ' <"$W/tmp/err")"
|
||||
fi
|
||||
}
|
||||
|
||||
doc() { printf '%s' "$2" >"$W/tmp/$1"; echo "$W/tmp/$1"; }
|
||||
policy() { doc policy.json "{\"Version\":\"2012-10-17\",\"Statement\":[$1]}"; }
|
||||
acl_body() { # acl_body <grantee> <type> <permission>
|
||||
doc acl.xml "<AccessControlPolicy xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Owner><ID>po-owner</ID></Owner><AccessControlList><Grant><Grantee xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:type=\"$2\"><ID>$1</ID></Grantee><Permission>$3</Permission></Grant></AccessControlList></AccessControlPolicy>"
|
||||
}
|
||||
BUCKET_RES='["arn:aws:s3:::proof","arn:aws:s3:::proof/*"]'
|
||||
S3NS='xmlns="http://s3.amazonaws.com/doc/2006-03-01/"'
|
||||
|
||||
echo "--- setup (root, admin API)"
|
||||
check_cmd S1 ok "create-user po-owner" admin create-user --access po-owner --secret "${SECRET[po-owner]}" --role userplus --user-id 74990 --group-id 74990
|
||||
check_cmd S2 ok "create-user po-full" admin create-user --access po-full --secret "${SECRET[po-full]}" --role user --user-id 74990 --group-id 74990
|
||||
check_cmd S3 ok "create-user po-obj" admin create-user --access po-obj --secret "${SECRET[po-obj]}" --role user --user-id 74990 --group-id 74990
|
||||
check_cmd S4 ok "create-user po-foreign" admin create-user --access po-foreign --secret "${SECRET[po-foreign]}" --role user --user-id 74991 --group-id 74991
|
||||
check_cmd S5 ok "create-bucket proof --owner po-owner --grant-full-control po-full" \
|
||||
admin create-bucket --owner po-owner --bucket proof --object-ownership BucketOwnerPreferred --grant-full-control po-full
|
||||
printf 'hello\n' >"$W/tmp/hello.txt"
|
||||
|
||||
echo "--- a foreign key is refused as a grantee"
|
||||
check_cmd A1 refused "admin create-bucket --grant-full-control po-foreign (root)" \
|
||||
admin create-bucket --owner po-owner --bucket proof-foreign --object-ownership BucketOwnerPreferred --grant-full-control po-foreign
|
||||
check A2 404 " ...and no bucket proof-foreign was made" root HEAD /proof-foreign
|
||||
check A3 403 "PutBucketAcl x-amz-grant-read: po-foreign, as the owner" po-owner PUT /proof acl -H "x-amz-grant-read: po-foreign"
|
||||
check A4 403 "PutBucketAcl x-amz-grant-read: po-foreign, as root" root PUT /proof acl -H "x-amz-grant-read: po-foreign"
|
||||
check A5 403 "PutBucketAcl body naming po-foreign, as root" root PUT /proof acl --body "$(acl_body po-foreign CanonicalUser FULL_CONTROL)"
|
||||
check A6 200 "PutBucketAcl x-amz-grant-full-control: po-full (owner's key)" po-owner PUT /proof acl -H "x-amz-grant-full-control: po-full"
|
||||
|
||||
echo "--- a foreign key is refused as a principal"
|
||||
P=$(policy "{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"po-foreign\"},\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::proof/*\"}")
|
||||
check B1 403 "PutBucketPolicy naming po-foreign, as the owner" po-owner PUT /proof policy --body "$P"
|
||||
check B2 403 "PutBucketPolicy naming po-foreign, as root" root PUT /proof policy --body "$P"
|
||||
|
||||
echo "--- public access is read-only"
|
||||
P=$(policy "{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::proof/*\"}")
|
||||
check C1 403 "PutBucketPolicy allowing \"*\" s3:PutObject" po-owner PUT /proof policy --body "$P"
|
||||
check C2 403 "PutBucketAcl x-amz-acl: public-read-write" po-owner PUT /proof acl -H "x-amz-acl: public-read-write"
|
||||
check C3 403 "PutBucketAcl body granting all-users WRITE" po-owner PUT /proof acl --body "$(acl_body all-users Group WRITE)"
|
||||
check C4 200 "owner PutObject public/hello.txt" po-owner PUT /proof/public/hello.txt --body "$W/tmp/hello.txt"
|
||||
P=$(policy "{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":[\"s3:GetObject\",\"s3:ListBucket\"],\"Resource\":$BUCKET_RES}")
|
||||
check C5 204 "PutBucketPolicy allowing \"*\" s3:GetObject + s3:ListBucket" po-owner PUT /proof policy --body "$P"
|
||||
check C6 200 "anonymous GetObject public/hello.txt" anon GET /proof/public/hello.txt
|
||||
check C7 200 "anonymous ListBucket" anon GET /proof list-type=2
|
||||
check C8 403 "anonymous PutObject" anon PUT /proof/public/anon.txt --body "$W/tmp/hello.txt"
|
||||
|
||||
echo "--- the owner key keeps its bucket under a public-read-only policy"
|
||||
check D1 200 "owner PutObject" po-owner PUT /proof/owner/a.txt --body "$W/tmp/hello.txt"
|
||||
check D2 200 "owner GetObject" po-owner GET /proof/owner/a.txt
|
||||
check D3 204 "owner DeleteObject" po-owner DELETE /proof/owner/a.txt
|
||||
check D4 200 "owner GetBucketAcl" po-owner GET /proof acl
|
||||
|
||||
echo "--- ...and under a policy that denies it"
|
||||
P=$(policy "{\"Effect\":\"Deny\",\"Principal\":{\"AWS\":\"po-owner\"},\"Action\":\"s3:*\",\"Resource\":$BUCKET_RES},{\"Effect\":\"Deny\",\"Principal\":\"*\",\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::proof/*\"}")
|
||||
check E1 204 "PutBucketPolicy denying po-owner s3:* and \"*\" s3:PutObject" po-owner PUT /proof policy --body "$P"
|
||||
check E2 200 "owner PutObject" po-owner PUT /proof/owner/b.txt --body "$W/tmp/hello.txt"
|
||||
check E3 200 "owner GetObject" po-owner GET /proof/owner/b.txt
|
||||
check E4 200 "owner ListBucket" po-owner GET /proof list-type=2
|
||||
check E5 204 "owner DeleteObject" po-owner DELETE /proof/owner/b.txt
|
||||
check E6 403 "FULL_CONTROL grantee PutObject: the Deny beats its ACL grant" po-full PUT /proof/full/b.txt --body "$W/tmp/hello.txt"
|
||||
check E7 200 "FULL_CONTROL grantee GetObject: the ACL still grants it" po-full GET /proof/public/hello.txt
|
||||
check E8 204 "owner DeleteBucketPolicy" po-owner DELETE /proof policy
|
||||
|
||||
echo "--- a key for one object"
|
||||
P=$(policy "{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"po-obj\"},\"Action\":[\"s3:GetObject\",\"s3:PutObject\",\"s3:DeleteObject\"],\"Resource\":\"arn:aws:s3:::proof/shared/report.txt\"}")
|
||||
check F1 204 "PutBucketPolicy for po-obj on shared/report.txt" po-owner PUT /proof policy --body "$P"
|
||||
check F2 200 "po-obj PutObject shared/report.txt" po-obj PUT /proof/shared/report.txt --body "$W/tmp/hello.txt"
|
||||
check F3 200 "po-obj GetObject shared/report.txt" po-obj GET /proof/shared/report.txt
|
||||
check F4 403 "po-obj GetObject public/hello.txt" po-obj GET /proof/public/hello.txt
|
||||
check F5 403 "po-obj PutObject shared/other.txt" po-obj PUT /proof/shared/other.txt --body "$W/tmp/hello.txt"
|
||||
check F6 403 "po-obj DeleteObject public/hello.txt" po-obj DELETE /proof/public/hello.txt
|
||||
check F7 403 "po-obj ListBucket" po-obj GET /proof list-type=2
|
||||
check F8 403 "po-obj GetBucketAcl" po-obj GET /proof acl
|
||||
check F9 403 "po-obj PutBucketAcl" po-obj PUT /proof acl -H "x-amz-grant-read: po-obj"
|
||||
check F10 403 "po-obj GetBucketPolicy" po-obj GET /proof policy
|
||||
check F11 403 "po-obj PutBucketPolicy" po-obj PUT /proof policy --body "$P"
|
||||
check F12 204 "po-obj DeleteObject shared/report.txt" po-obj DELETE /proof/shared/report.txt
|
||||
|
||||
echo "--- bucket settings are root's and admins' alone"
|
||||
OWN=$(doc own.xml "<OwnershipControls $S3NS><Rule><ObjectOwnership>BucketOwnerPreferred</ObjectOwnership></Rule></OwnershipControls>")
|
||||
VER=$(doc ver.xml "<VersioningConfiguration $S3NS><Status>Enabled</Status></VersioningConfiguration>")
|
||||
LCK=$(doc lck.xml "<ObjectLockConfiguration $S3NS><ObjectLockEnabled>Enabled</ObjectLockEnabled></ObjectLockConfiguration>")
|
||||
# A FULL_CONTROL grant alone never reached these four; a policy did, so grant them outright.
|
||||
P=$(policy "{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"po-full\"},\"Action\":[\"s3:PutBucketOwnershipControls\",\"s3:PutBucketVersioning\",\"s3:PutBucketObjectLockConfiguration\",\"s3:DeleteBucket\"],\"Resource\":\"arn:aws:s3:::proof\"}")
|
||||
check G0 204 "PutBucketPolicy granting po-full all four" po-owner PUT /proof policy --body "$P"
|
||||
check G1 403 "FULL_CONTROL grantee PutBucketOwnershipControls" po-full PUT /proof ownershipControls --body "$OWN"
|
||||
check G2 403 "FULL_CONTROL grantee DeleteBucketOwnershipControls" po-full DELETE /proof ownershipControls
|
||||
check G3 403 "FULL_CONTROL grantee PutBucketVersioning" po-full PUT /proof versioning --body "$VER"
|
||||
check G4 403 "FULL_CONTROL grantee PutObjectLockConfiguration" po-full PUT /proof object-lock --body "$LCK"
|
||||
check G5 403 "FULL_CONTROL grantee DeleteBucket" po-full DELETE /proof
|
||||
check G6 403 "owner PutBucketVersioning" po-owner PUT /proof versioning --body "$VER"
|
||||
check G7 403 "owner PutObjectLockConfiguration" po-owner PUT /proof object-lock --body "$LCK"
|
||||
check G8 403 "owner DeleteBucket" po-owner DELETE /proof
|
||||
check G9 204 "owner DeleteBucketPolicy" po-owner DELETE /proof policy
|
||||
check G10 200 "FULL_CONTROL grantee PutObject (its grant still works)" po-full PUT /proof/full/c.txt --body "$W/tmp/hello.txt"
|
||||
check G11 200 "root PutBucketOwnershipControls" root PUT /proof ownershipControls --body "$OWN"
|
||||
|
||||
echo "--- an ACL with the public read grant writes back unchanged"
|
||||
check H1 200 "PutBucketAcl x-amz-acl: public-read" po-owner PUT /proof acl -H "x-amz-acl: public-read"
|
||||
check H2 200 "GetBucketAcl" po-owner GET /proof acl -o "$W/tmp/acl-read.xml"
|
||||
check H3 200 "PutBucketAcl with that body, unchanged" po-owner PUT /proof acl --body "$W/tmp/acl-read.xml"
|
||||
check H4 200 "GetBucketAcl again" po-owner GET /proof acl -o "$W/tmp/acl-again.xml"
|
||||
check_cmd H5 ok " ...and it still grants all-users READ" \
|
||||
grep -q '<ID>all-users</ID>.*<Permission>READ</Permission>' "$W/tmp/acl-again.xml"
|
||||
|
||||
echo "--- cleanup (root)"
|
||||
for k in public/hello.txt full/c.txt; do req root DELETE "/proof/$k" >/dev/null; done
|
||||
check Z1 204 "root DeleteBucket" root DELETE /proof
|
||||
|
||||
echo
|
||||
echo "$PASS passed, $FAIL failed"
|
||||
[ "$FAIL" = 0 ]
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env python3
|
||||
"""One S3 request, SigV4-signed with the key in the environment.
|
||||
|
||||
S3_ENDPOINT=http://127.0.0.1:7390 S3_KEY=ak S3_SECRET=sk \
|
||||
s3req.py PUT /bucket/key [query] [--body FILE] [-H 'Name: value'] [-o FILE]
|
||||
|
||||
Prints the HTTP status on stdout, and for an error response the S3 error
|
||||
code and message on stderr. With S3_KEY empty or unset the request is
|
||||
anonymous. Python standard library only, so it runs on a bare EL9 node.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import datetime
|
||||
import hashlib
|
||||
import hmac
|
||||
import http.client
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.parse
|
||||
|
||||
UNRESERVED = "-_.~"
|
||||
|
||||
|
||||
def quote(s, safe=""):
|
||||
return urllib.parse.quote(s, safe=UNRESERVED + safe)
|
||||
|
||||
|
||||
def canonical_query(query):
|
||||
pairs = urllib.parse.parse_qsl(query, keep_blank_values=True)
|
||||
return "&".join(f"{quote(k)}={quote(v)}" for k, v in sorted(pairs))
|
||||
|
||||
|
||||
def sign(method, host, path, query, headers, body, key, secret, region):
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
amz_date = now.strftime("%Y%m%dT%H%M%SZ")
|
||||
day = now.strftime("%Y%m%d")
|
||||
headers["host"] = host
|
||||
headers["x-amz-date"] = amz_date
|
||||
headers["x-amz-content-sha256"] = hashlib.sha256(body).hexdigest()
|
||||
|
||||
signed = sorted(headers)
|
||||
canonical = "\n".join([
|
||||
method,
|
||||
quote(path, safe="/"),
|
||||
canonical_query(query),
|
||||
"".join(f"{h}:{headers[h].strip()}\n" for h in signed),
|
||||
";".join(signed),
|
||||
headers["x-amz-content-sha256"],
|
||||
])
|
||||
scope = f"{day}/{region}/s3/aws4_request"
|
||||
to_sign = "\n".join(["AWS4-HMAC-SHA256", amz_date, scope,
|
||||
hashlib.sha256(canonical.encode()).hexdigest()])
|
||||
|
||||
k = ("AWS4" + secret).encode()
|
||||
for part in (day, region, "s3", "aws4_request"):
|
||||
k = hmac.new(k, part.encode(), hashlib.sha256).digest()
|
||||
signature = hmac.new(k, to_sign.encode(), hashlib.sha256).hexdigest()
|
||||
headers["authorization"] = (f"AWS4-HMAC-SHA256 Credential={key}/{scope}, "
|
||||
f"SignedHeaders={';'.join(signed)}, Signature={signature}")
|
||||
|
||||
|
||||
def main(argv):
|
||||
args, headers, body, out = [], {}, b"", None
|
||||
it = iter(argv)
|
||||
for a in it:
|
||||
if a == "--body":
|
||||
path = next(it)
|
||||
body = sys.stdin.buffer.read() if path == "-" else open(path, "rb").read()
|
||||
elif a == "-H":
|
||||
name, _, value = next(it).partition(":")
|
||||
headers[name.strip().lower()] = value.strip()
|
||||
elif a == "-o":
|
||||
out = next(it)
|
||||
else:
|
||||
args.append(a)
|
||||
if len(args) not in (2, 3):
|
||||
sys.exit(__doc__)
|
||||
method, path = args[0].upper(), args[1]
|
||||
query = args[2] if len(args) == 3 else ""
|
||||
|
||||
endpoint = urllib.parse.urlsplit(os.environ.get("S3_ENDPOINT", "http://127.0.0.1:7390"))
|
||||
region = os.environ.get("S3_REGION", "us-east-1")
|
||||
key, secret = os.environ.get("S3_KEY", ""), os.environ.get("S3_SECRET", "")
|
||||
|
||||
if body:
|
||||
headers["content-md5"] = base64.b64encode(hashlib.md5(body).digest()).decode()
|
||||
if key:
|
||||
sign(method, endpoint.netloc, path, query, headers, body, key, secret, region)
|
||||
|
||||
conn = http.client.HTTPConnection(endpoint.hostname, endpoint.port, timeout=30)
|
||||
target = quote(path, safe="/") + ("?" + query if query else "")
|
||||
conn.request(method, target, body=body if method in ("PUT", "POST") else None, headers=headers)
|
||||
resp = conn.getresponse()
|
||||
data = resp.read()
|
||||
if out:
|
||||
with open(out, "wb") as f:
|
||||
f.write(data)
|
||||
print(resp.status)
|
||||
if resp.status >= 300 and data:
|
||||
code = re.search(rb"<Code>(.*?)</Code>", data)
|
||||
msg = re.search(rb"<Message>(.*?)</Message>", data)
|
||||
print((code.group(1) if code else b"").decode(), "-",
|
||||
(msg.group(1) if msg else data[:200]).decode(errors="replace"), file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1:])
|
||||
Reference in new issue
Block a user