s3-gateway/docs/vault-atomic-update.md

# Atomic Vault IAM update (`0004-vault-atomic-update-user.patch`)

`patches/v1.11.0/0004-vault-atomic-update-user.patch` applies after `0003`. It changes one call of VersityGW's Vault
IAM backend — `UpdateUserAccount`, which serves the admin API's `update-user` and so every key rotation Norn makes —
from delete-then-create into one check-and-set write of the account's KV v2 entry. Nothing else changes: the
internal IAM, the other Vault calls, the admin API and the S3 side are as upstream has them.

| | |
|---|---|
| Touches | `auth/iam_vault.go`, `auth/iam_vault_update_test.go` (new) |
| Base | upstream `v1.11.0` (`e42ae7ab`) + `0001` + `0002` + `0003` |
| Unit tests | `go test ./auth/ -run Vault` (table-driven, against a fake KV v2 + AppRole server) |
| Proof test | [`tests/vault-update/proof.sh`](../tests/vault-update/proof.sh), on leil-rig, below |

## The problem

Upstream v1.11.0's `VaultIAMService.UpdateUserAccount` reads the account (`GetUserAccount`), deletes its KV entry —
metadata and every version, `KvV2DeleteMetadataAndAllVersions` — and creates it again with `cas: 0`
(`CreateAccount`). Between the delete and the create the account does not exist, and nothing ties the three calls
together:

- **The account goes missing during a rotation.** Several gateway instances of one namespace share a KV path, and
  each keeps a 120 s IAM cache that never caches a missing account. An instance whose cache misses inside the gap —
  its entry for that key expired, or it had never looked the key up — finds no entry and answers
  `InvalidAccessKeyId`, to every request in the gap. The gap lasts one OpenBao round trip of the writing instance
  plus OpenBao's write: milliseconds on one host, more with OpenBao across the network or behind a Raft cluster.
- **A failure in the middle loses the account for good.** If the create fails after the delete succeeded — OpenBao
  unreachable or sealed, a token that may no longer write, the gateway stopping — the update reports an error and
  the account is gone, on every instance once its cache entry expires.
- **Two updates at once lose one of them.** Each deletes whatever is there and writes its own copy of what it read;
  both report success.

## The change

| | |
|---|---|
| **One write** | `UpdateUserAccount` reads the entry with its KV version, applies the change, and writes it back with one `KvV2Write` carrying `cas` = that version. The account exists throughout; a write that fails leaves it exactly as it was. |
| **Conflicts retry** | If another writer changed the entry in between, OpenBao refuses the write (`400`, "check-and-set parameter did not match the current version") and the update starts again from a fresh read, up to 5 attempts (`vaultUpdateAttempts`). After that it fails with the last conflict and the account keeps the other writer's state. |
| **No such user** | An account that does not exist answers `ErrNoSuchUser` ("user not found", which the admin API maps to its user-not-found error), as before. So does one deleted between the read and the write: the conflict sends the update back to the read, which finds nothing — an update never re-creates a deleted account. |
| **403** | A `403` on the read or the write logs in again with the AppRole and retries that call once, as the backend's other calls do. |
| **Shared read** | The read moved into `readAccountEntry`, which `GetUserAccount` now calls too; its behaviour is unchanged. `CreateAccount`, `DeleteUserAccount` and `ListUserAccounts` are untouched. |

### What else changes with it

- **The gateways' policy** needs `read` and `update` on `<mount>/data/<path>/*` for an update, and no longer
  `delete` on `<mount>/metadata/<path>/*` (which `delete-user` still needs). leil-cluster's proposed policy covers
  both, and is what the proof runs with.
- **KV keeps history now.** Each update adds a version; the previous ones, earlier secrets included, stay readable at
  `?version=N` until the mount's `max_versions` (OpenBao's default: 10) pushes them out. Upstream's delete wiped them
  at every update. Only a token that can read the current secret can read them, but if retired secrets should not
  linger, set `max_versions` low on the IAM mount (`bao write s3gw-iam/config max_versions=2`, or per entry) — the
  version number keeps counting up, so check-and-set is unaffected.
- **Updating the root account's key** now answers "user not found", as the internal backend does. Upstream answered
  "user exists" (its re-create refused root's key) after deleting a KV path that never held anything.

### What it does not change

- **Other instances still learn of a rotation through their cache.** The writing instance drops its own cache entry;
  every other instance keeps accepting the old secret, and refusing the new one, until its entry expires — up to the
  IAM cache time (120 s). The proof measures it (R5). Norn has to give a rotation that long before it retires the old
  secret.
- **Concurrent creates** were already safe (`cas: 0`: the second gets `ErrUserExists`), and **deletes** still remove
  the entry and its history at once.
- **ABA**: if an account is deleted and created again between an update's read and its write, and the new entry is
  at the version the update read (both at version 1), the write lands on the new account. That takes a delete and a
  create of the same key inside one update; nothing in Norn's flows does that.

## Proving the patch is in (on leil-rig)

`--version` reads `1.11.0` with or without the patch, so again the proof is behaviour. `tests/vault-update/proof.sh`
starts its own OpenBao dev server (`bao server -dev`, in memory, `127.0.0.1:18200`) with the installed `bao` binary —
not the rig's OpenBao service, whose data it never touches — and gives it the gateways' layout: a KV v2 mount
`s3gw-iam`, accounts at `proof/0x0/accounts`, AppRole login with leil-cluster's proposed policy. It then runs two
gateways from the given binary on that one KV path, each with the default 120 s IAM cache:

| Gateway | Listens on | Reaches OpenBao |
|---|---|---|
| A | `127.0.0.1:7391` | through `vaultproxy.py`, which holds every request 3 s (`PROOF_DELAY`), as from a distant OpenBao, and can refuse KV writes |
| B | `127.0.0.1:7392` | directly |

The proxy is what makes upstream's gap observable on one host: at 3 s per call, the delete reaches OpenBao 6 s into
the update and the create 9 s in, so the gap is 3 s wide instead of a millisecond or two. The patched build makes the
same calls minus the delete, so the delay costs it nothing but time. The script times the rotation so that the gap
spans the moment B's cache entry expires. A KV sampler (`watch.py kv`, every 5 ms) and an S3 sampler on B
(`watch.py s3`, `ListBuckets` signed with the old secret every 50 ms) record what happens. Everything is under
`/var/tmp/s3gw-vault-update` and is removed at exit (`KEEP=1` keeps it); nothing installed is touched, and nothing
needs root.

One account, created through A's admin API on leil-cluster's reserved test band: `vu-rot`, role `user`, UID 74992,
GID 74993.

Take the lease, build, copy the binary and both test directories (the proof signs with
`tests/owner-scope/s3req.py`) to the test node, run it with the node's `bao`:

```bash
RIG_LEASE_WHO="<you>: s3-gateway 0004 proof" ./bin/rig-lease.sh claim --why "s3-gateway 0004 proof on leil-test-1" --minutes 60
VGW_SRC_DIR=build-src VGW_TAG=v1.11.0 bash ci/run.sh
cd ../leil-rig && source bin/rig-lib.sh
on leil-test-1 'install -d -o leil -g leil /var/tmp/s3gw-0004'
rig_scp ../s3-gateway/build-src/dist/versitygw_linux_amd64_v1/versitygw leil-test-1 /var/tmp/s3gw-0004/versitygw
tar -czf /tmp/vault-update.tgz -C ../s3-gateway/tests owner-scope vault-update
rig_scp /tmp/vault-update.tgz leil-test-1 /var/tmp/s3gw-0004/vault-update.tgz
on leil-test-1 'cd /var/tmp/s3gw-0004 && chmod 0755 versitygw && tar -xzf vault-update.tgz && runuser -u leil -- bash vault-update/proof.sh $PWD/versitygw /usr/bin/bao'
```

It needs the ports free on loopback: 7391–7392 for the gateways, 18200 for OpenBao's API, 18201 for the dev server's
cluster listener and 18202 for the proxy. `/usr/bin/bao` belongs to the rig's `openbao.service`; the proof only runs it
in `-dev` mode and never talks to that service.

It takes about four minutes, most of it waiting out B's cache. Every line must read `PASS` and the last
`N passed, 0 failed`; the exit status says the same. `PROOF_PORT` / `PROOF_BAO_PORT` move the ports,
`PROOF_DELAY` the proxy's hold, `PROOF_CACHE_TTL` both gateways' cache time (shorter only for a quick try: the
rotation is lined up with B's expiry, and with a short cache it cannot be). What it covers, in order:

| Checks | Shows |
|---|---|
| S1–S4 | OpenBao dev server, mount, AppRole and policy; `create-user vu-rot` through A; the key signs on B (which caches it from then on) and on A |
| R1–R2 | `update-user --secret <new>` through A, timed so upstream's gap would span B's cache expiry; A takes the new secret at once |
| R3 | the KV entry was there at every 5 ms sample through the rotation |
| R4 | B, signing with the old secret through its cache expiry and past it, never answered `InvalidAccessKeyId` |
| R5 | B took the old secret until its cache entry expired, and refused it (`SignatureDoesNotMatch`) from then on — the propagation time other instances have |
| R6–R7 | B takes the new secret after its cache time; the entry still has role `user`, UID 74992, GID 74993, project 0 |
| R8 | one write: the entry's KV version went up by one and its creation time is the same |
| C1–C4 | `update-user --project-id 41` through A while, between A's read and its write, `update-user --secret <newer>` goes through B: both succeed and the entry has both changes, UID and GID intact (A's write hit the conflict and was retried); `admin list-users` on B agrees |
| F1–F4 | with the proxy refusing A's KV writes, `update-user --secret <another>` through A fails, and the entry, its version and its secret are as they were: both gateways still sign with the current secret |

Release the lease and clean up: `on leil-test-1 'rm -rf /var/tmp/s3gw-0004'`, then `./bin/rig-lease.sh release`.

### Result on leil-rig, 2026-11-11: **PASSED**

Run on `leil-test-1` (AlmaLinux 9.8, kernel 5.14.0-687.53.1, Python 3.9.25) as `leil`, under the lease, against a dev server of the node's
OpenBao 2.6.2 (`dd9c19c3`), from `ci/run.sh` builds of this repository's series:

| | |
|---|---|
| Build | `VGW_TAG=v1.11.0 bash ci/run.sh` (fresh clone of `v1.11.0`, 0001–0004 by `git am --3way`, GoReleaser); `--version` 1.11.0 |
| `versitygw` (linux amd64) sha256 | `035be817fac9659f9bbbe25307e0eedca16ab4a5eb67ee739c1aa995bc755aed` |
| `versitygw_1.11.0_linux_amd64.rpm` sha256 | `cbedf61c60ad207172b4adbe458f2ec73141c6f6fd72e737c501c022f0decde0` (built locally; not published, not pinned) |
| `proof.sh` | **20 passed, 0 failed** |
| Unit tests on the node (go1.27.3 linux/amd64) | `go test` of every package except `plugins/`: 35 of 35 ok, among them `auth` (with the new Vault cases) and `s3api/...` |

The patched rotation took 6.2 s with OpenBao 3 s away from A — a read and a write. The KV entry was present in all
16164 samples over 128 s, at version 1 then 2. B answered its 2415 requests with the old secret `200` (2247) or
`SignatureDoesNotMatch` (168), never `InvalidAccessKeyId`, and switched 0.04–0.10 s after its cache entry expired.
In the concurrent update A's first write hit the conflict and its retry landed: version 4 holds B's secret and A's
project.

**Control: 0001–0003 only** (same script, same node; `versitygw` sha256 `3ac9be149f1bc35d…`, built by `ci/run.sh`
from the same repository state with `PATCH_DIR` holding 0001–0003): **13 passed, 7 failed.**

- R3: the KV entry was missing (`404`) in 396 of 16134 samples, for 3.00 s — the delete-to-create gap, one proxy
  delay wide. The rotation took 9.2 s: a read, a delete and a create.
- R4: B answered `InvalidAccessKeyId` 28 times, every request from its cache expiry until the create landed 1.70 s
  later (R5's switch to `SignatureDoesNotMatch` at +1.70 s).
- R8: the entry came back at version 1 with a new creation time: deleted and made again, its history gone.
- C3: B's rotation was lost — A's delete removed it and A's create wrote A's stale copy (secret from before, project
  41). Both `update-user` calls reported success (C1, C2).
- F2–F4: with A's create refused, the account was gone from OpenBao and both gateways answered `InvalidAccessKeyId`
  to its current secret. The failed `update-user` (F1) had already deleted it.

The control passed R1–R2, R5–R7, C1–C2, C4 and F1 like the patched build: upstream also keeps UserID and GroupID
through a rotation, and every instance takes the new secret within its cache time once the account exists again.

Run this test again before pinning any build that carries 0004, in leil-rig or leil-cluster.