0004-vault-atomic-update-user.patch makes the Vault IAM backend's UpdateUserAccount -- what admin update-user, and so every key rotation, runs -- one KV v2 write with cas set to the version it read, retried from a fresh read on a check-and-set conflict (up to 5 attempts). Upstream deleted the entry and created it again: in between, another instance on the same KV path whose IAM cache missed answered InvalidAccessKeyId, a failure between the two calls lost the account, and two updates at once lost one of them. Touches only auth/iam_vault.go and a new auth/iam_vault_update_test.go. tests/vault-update/proof.sh proves it on leil-rig against an OpenBao dev server, with two gateways on one KV path and gateway A's OpenBao calls held 3 s by a proxy so upstream's gap can be observed. On leil-test-1: patched 20 passed, 0 failed; 0001-0003 control 13 passed, 7 failed (entry missing 3.00 s, 28 InvalidAccessKeyId on the second gateway, concurrent rotation lost, account gone after a refused write). go test on the node: 35 of 35 packages ok. docs/vault-atomic-update.md has the problem, the change, the proof and its result; README and the multi-version pipeline doc point at it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
74 lines
2.5 KiB
Python
74 lines
2.5 KiB
Python
#!/usr/bin/env python3
|
|
"""A TCP proxy in front of a plain-HTTP OpenBao that holds requests, and can refuse some.
|
|
|
|
vaultproxy.py LISTEN_PORT UPSTREAM_PORT CONTROL_DIR
|
|
|
|
Everything a client sends is held for the number of seconds in CONTROL_DIR/delay
|
|
(none while the file is missing) before it goes on to 127.0.0.1:UPSTREAM_PORT, so
|
|
each Vault call of the gateway behind it takes that much longer, as against a
|
|
distant OpenBao. While CONTROL_DIR/refuse exists, a request that starts with its
|
|
contents (say "POST /v1/s3gw-iam/data/") never reaches OpenBao: the proxy answers
|
|
it 503 and closes the connection. Both files are read at every request, so a test
|
|
can change them while the proxy runs. Python standard library only.
|
|
"""
|
|
|
|
import asyncio
|
|
import sys
|
|
|
|
LISTEN, UPSTREAM, CONTROL = int(sys.argv[1]), int(sys.argv[2]), sys.argv[3]
|
|
|
|
|
|
def control(name, default):
|
|
try:
|
|
with open(f"{CONTROL}/{name}") as f:
|
|
return f.read().strip()
|
|
except FileNotFoundError:
|
|
return default
|
|
|
|
|
|
async def handle(client_r, client_w):
|
|
try:
|
|
server_r, server_w = await asyncio.open_connection("127.0.0.1", UPSTREAM)
|
|
except OSError:
|
|
client_w.close()
|
|
return
|
|
|
|
async def upstream():
|
|
try:
|
|
while data := await client_r.read(65536):
|
|
refuse = control("refuse", "")
|
|
if refuse and data.startswith(refuse.encode()):
|
|
body = b'{"errors":["refused by the proof proxy"]}'
|
|
client_w.write(b"HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\n"
|
|
b"Connection: close\r\nContent-Length: %d\r\n\r\n%s" % (len(body), body))
|
|
await client_w.drain()
|
|
break
|
|
delay = float(control("delay", "0") or 0)
|
|
if delay:
|
|
await asyncio.sleep(delay)
|
|
server_w.write(data)
|
|
await server_w.drain()
|
|
finally:
|
|
server_w.close()
|
|
client_w.close()
|
|
|
|
async def downstream():
|
|
try:
|
|
while data := await server_r.read(65536):
|
|
client_w.write(data)
|
|
await client_w.drain()
|
|
finally:
|
|
client_w.close()
|
|
|
|
await asyncio.gather(upstream(), downstream(), return_exceptions=True)
|
|
|
|
|
|
async def main():
|
|
server = await asyncio.start_server(handle, "127.0.0.1", LISTEN)
|
|
async with server:
|
|
await server.serve_forever()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
asyncio.run(main())
|