s3-gateway/docs/multi-version-pipeline.md

# Multi-version build pipeline

How s3-gateway turns an upstream [versitygw](https://github.com/versity/versitygw)
release into a de-branded build, and how to keep it tracking new
upstream versions. This mirrors the scoutfs-build / scoutfs-notify
model, collapsed into one repo because the patch set is small.

CI builds **one version per run** — triggered by a tag push or a manual
dispatch, never on a schedule.

## The flow

Each build is four steps, run for a single upstream tag:

1. **`ci/clone-source.sh`** — shallow-clones upstream versitygw at a
   tag into `src/`. Tag selection priority:
   1. `VGW_TAG` (the dispatch input, or a manual run),
   2. the pushed tag (`GITHUB_REF` = `refs/tags/...`),
   3. the latest remote `vX.Y.Z` tag.
   The checkout forces `core.autocrlf=false` so the tree is byte-for-
   byte upstream (LF) and the LF patches apply on any host.

2. **`ci/apply-patches.sh`** — resolves the patch directory for the tag
   from `ci/support-matrix.yaml` (`patches:` field, default = tag name)
   and applies `patches/<dir>/*.patch` with `git am --whitespace=nowarn
   --3way`, in filename order. The `--3way` fallback is what lets a
   series rebase across minor upstream drift. A tag the matrix does not
   list is **refused** (exit 1), as is a patch directory that turns out
   to be empty — either would otherwise publish an un-de-branded or
   superseded build.

3. **`ci/build.sh`** — runs `goreleaser release --clean
   --skip=validate,publish` in `src/`, producing binaries + deb/rpm in
   `src/dist/`. Because the patch commits sit on top of the tag, HEAD is
   no longer exactly the tag, so we set `GORELEASER_CURRENT_TAG` to pin
   the stamped version and `--skip=validate` to bypass the dirty/tag
   checks. Upstream's `.goreleaser.yaml` is used unmodified — artifact
   names stay `versitygw`.

4. **`ci/publish-release.sh`** — creates (or reuses) a Gitea release for
   the tag in this repo and uploads `src/dist/*` as assets. Idempotent:
   re-running for the same tag replaces existing assets.

## Patch registry: `ci/support-matrix.yaml`

```yaml
versitygw:
  - tag: v1.11.0
    patches: v1.11.0
```

- `tag` — upstream git tag.
- `patches` — directory under `patches/` to apply (defaults to `tag`).
  Point several tags at one directory if the same series applies to a
  range of upstream releases.

`apply-patches.sh` consults this to find the patch directory for the tag
being built; with the default `patches/<tag>/` naming it doubles as a
human-readable record of which versions are maintained.

It is also an **allowlist**, which is the mechanism that keeps
superseded versions from being rebuilt: a tag absent from the matrix
makes `apply-patches.sh` exit 1. Only `PATCH_DIR` bypasses it, and only
explicitly — that is how an archived series is rebuilt on purpose.

The matrix is read with `yq` when it is installed and with a small
`awk` parse otherwise, so the allowlist still applies in the stock
`golang` CI container, which has no `yq`.

## Adding a new upstream version

When upstream releases `vX.Y.Z`:

1. **Rebase the patch series** onto the new tag in a throwaway clone:

   ```bash
   git clone https://github.com/versity/versitygw vgw && cd vgw
   git -c core.autocrlf=false checkout -b debrand vX.Y.Z
   git config user.email you@example.com && git config user.name you
   git am --3way /path/to/s3-gateway/patches/<previous-tag>/*.patch
   # If a hunk conflicts: fix files, `git add -A`, `git am --continue`.
   mkdir -p /path/to/s3-gateway/patches/vX.Y.Z
   git format-patch --binary vX.Y.Z..HEAD \
     -o /path/to/s3-gateway/patches/vX.Y.Z
   ```

   `--binary` is required because the series deletes PNG logo assets.

2. **Register it** in `ci/support-matrix.yaml`:

   ```yaml
   versitygw:
     - tag: vX.Y.Z
       patches: vX.Y.Z
   ```

   Leave the previous entry in place only if you still want builds of
   it. If the new release supersedes it — a security fix, say — retire
   the old one at the same time (see *Retiring a version* below).

3. **Commit and push** `ci/support-matrix.yaml` + `patches/vX.Y.Z/`.

4. **Build it** — either push a `vX.Y.Z` tag to this repo, or run the
   **build** workflow manually and enter `vX.Y.Z`. CI builds that one
   version and publishes it.

### Sanity-check a series before committing

```bash
VGW_REPO_URL="/path/to/local/versitygw" VGW_TAG=vX.Y.Z bash ci/run.sh
```

A clean `git am` plus a populated `src/dist/` means the web UI patches
are good. `0003-owner-scoped-access.patch` and
`0004-vault-atomic-update-user.patch` change Go code in `auth/`, so a
clean apply is not enough for them: run their unit tests in the patched
tree, on Linux (on Windows several upstream packages fail regardless),

```bash
go test ./auth/ ./s3api/...
```

and each one's proof test on leil-rig — see
[owner-scoped-access.md](owner-scoped-access.md#proving-the-patch-is-in-on-leil-rig)
and
[vault-atomic-update.md](vault-atomic-update.md#proving-the-patch-is-in-on-leil-rig).
If upstream has reworked `auth/access-control.go` or `auth/acl.go`,
re-read 0003's rules against the new code rather than only resolving
the conflict; likewise `auth/iam_vault.go` for 0004 — if upstream's
`UpdateUserAccount` already writes with check-and-set, drop 0004.

`src/` may hold someone's local work tree; `ci/clone-source.sh` deletes
it. Point `VGW_SRC_DIR` somewhere fresh (any `*-src/` name is
gitignored) when in doubt.

## Retiring a version

Drop its `ci/support-matrix.yaml` entry — that alone stops it being
built, since the matrix is the allowlist — and `git mv` its directory
into `patches/archive/<tag>/` rather than deleting it, so the series
stays available for a deliberate rebuild:

```bash
git mv patches/v1.8.0 patches/archive/v1.8.0
# then remove the v1.8.0 entry from ci/support-matrix.yaml
```

Update the table in [`patches/archive/README.md`](../patches/archive/README.md)
and the supported-versions table in the top-level README to match.

Releases already published in this repo are **not** affected: retiring
the patches stops future builds but leaves existing release assets
downloadable until you delete them by hand in Gitea.

To rebuild an archived version deliberately, pass `PATCH_DIR` instead
of re-adding it to the matrix:

```bash
PATCH_DIR=patches/archive/v1.8.0 VGW_TAG=v1.8.0 bash ci/apply-patches.sh
```

## CI configuration

- **Triggers**: a `v*` tag push (builds that tag) or manual dispatch
  with a tag input (one-shot). No cron.
- **Runner**: any Linux runner — `runs-on: ubuntu-latest`. The Go
  toolchain comes from the `golang:latest` container, so the runner
  only needs Docker; nothing version-specific to install.
- **Container**: `golang:latest` — pure-Go build, GoReleaser's bundled
  nfpm makes deb/rpm without rpmbuild/dpkg.
- **Secret `TOKEN_GITEA`**: Gitea PAT with **write:repository**. Used
  both to clone this repo (the workflow does a manual token-rewritten
  clone, since a stock container has no Node for `actions/checkout`) and
  to publish releases.
- **Variable `VGW_REPO_URL`** (optional): override the upstream URL.

## Release tagging

Releases are created in this repo with the **upstream** tag name
(`v1.11.0`). If you need to ship a rebuild of the same upstream version
(e.g. a patch fix), set `RELEASE_TAG` in `ci/publish-release.sh`'s
environment to something like `v1.11.0-acg2` to avoid clobbering.