s3-gateway/docs/multi-version-pipeline.md

Multi-version build pipeline

How s3-gateway turns an upstream versitygw release into a de-branded build, and how to keep it tracking new upstream versions. This mirrors the scoutfs-build / scoutfs-notify model, collapsed into one repo because the patch set is small.

CI builds one version per run — triggered by a tag push or a manual dispatch, never on a schedule.

The flow

Each build is four steps, run for a single upstream tag:

1. ci/clone-source.sh — shallow-clones upstream versitygw at a tag into src/. Tag selection priority:

   1. VGW_TAG (the dispatch input, or a manual run),
   2. the pushed tag (GITHUB_REF = refs/tags/...),
   3. the latest remote vX.Y.Z tag. The checkout forces core.autocrlf=false so the tree is byte-for- byte upstream (LF) and the LF patches apply on any host.

2. ci/apply-patches.sh — resolves the patch directory for the tag from ci/support-matrix.yaml (patches: field, default = tag name) and applies patches/<dir>/*.patch with git am --whitespace=nowarn --3way, in filename order. The --3way fallback is what lets a series rebase across minor upstream drift. A tag the matrix does not list is refused (exit 1), as is a patch directory that turns out to be empty — either would otherwise publish an un-de-branded or superseded build.

3. ci/build.sh — runs goreleaser release --clean --skip=validate,publish in src/, producing binaries + deb/rpm in src/dist/. Because the patch commits sit on top of the tag, HEAD is no longer exactly the tag, so we set GORELEASER_CURRENT_TAG to pin the stamped version and --skip=validate to bypass the dirty/tag checks. Upstream's .goreleaser.yaml is used unmodified — artifact names stay versitygw.

4. ci/publish-release.sh — creates (or reuses) a Gitea release for the tag in this repo and uploads src/dist/* as assets. Idempotent: re-running for the same tag replaces existing assets.

Patch registry: ci/support-matrix.yaml

versitygw:
  - tag: v1.11.0
    patches: v1.11.0

• tag — upstream git tag.
• patches — directory under patches/ to apply (defaults to tag). Point several tags at one directory if the same series applies to a range of upstream releases.

apply-patches.sh consults this to find the patch directory for the tag being built; with the default patches/<tag>/ naming it doubles as a human-readable record of which versions are maintained.

It is also an allowlist, which is the mechanism that keeps superseded versions from being rebuilt: a tag absent from the matrix makes apply-patches.sh exit 1. Only PATCH_DIR bypasses it, and only explicitly — that is how an archived series is rebuilt on purpose.

The matrix is read with yq when it is installed and with a small awk parse otherwise, so the allowlist still applies in the stock golang CI container, which has no yq.

Adding a new upstream version

When upstream releases vX.Y.Z:

1. Rebase the patch series onto the new tag in a throwaway clone:

   git clone https://github.com/versity/versitygw vgw && cd vgw
   git -c core.autocrlf=false checkout -b debrand vX.Y.Z
   git config user.email you@example.com && git config user.name you
   git am --3way /path/to/s3-gateway/patches/<previous-tag>/*.patch
   # If a hunk conflicts: fix files, `git add -A`, `git am --continue`.
   mkdir -p /path/to/s3-gateway/patches/vX.Y.Z
   git format-patch --binary vX.Y.Z..HEAD \
     -o /path/to/s3-gateway/patches/vX.Y.Z
   

   --binary is required because the series deletes PNG logo assets.

2. Register it in ci/support-matrix.yaml:

   versitygw:
     - tag: vX.Y.Z
       patches: vX.Y.Z
   

   Leave the previous entry in place only if you still want builds of it. If the new release supersedes it — a security fix, say — retire the old one at the same time (see Retiring a version below).

3. Commit and push ci/support-matrix.yaml + patches/vX.Y.Z/.

4. Build it — either push a vX.Y.Z tag to this repo, or run the build workflow manually and enter vX.Y.Z. CI builds that one version and publishes it.

Sanity-check a series before committing

VGW_REPO_URL="/path/to/local/versitygw" VGW_TAG=vX.Y.Z bash ci/run.sh

A clean git am plus a populated src/dist/ means the web UI patches are good. 0003-owner-scoped-access.patch and 0004-vault-atomic-update-user.patch change Go code in auth/, so a clean apply is not enough for them: run their unit tests in the patched tree, on Linux (on Windows several upstream packages fail regardless),

go test ./auth/ ./s3api/...

and each one's proof test on leil-rig — see owner-scoped-access.md and vault-atomic-update.md. If upstream has reworked auth/access-control.go or auth/acl.go, re-read 0003's rules against the new code rather than only resolving the conflict; likewise auth/iam_vault.go for 0004 — if upstream's UpdateUserAccount already writes with check-and-set, drop 0004.

src/ may hold someone's local work tree; ci/clone-source.sh deletes it. Point VGW_SRC_DIR somewhere fresh (any *-src/ name is gitignored) when in doubt.

Retiring a version

Drop its ci/support-matrix.yaml entry — that alone stops it being built, since the matrix is the allowlist — and git mv its directory into patches/archive/<tag>/ rather than deleting it, so the series stays available for a deliberate rebuild:

git mv patches/v1.8.0 patches/archive/v1.8.0
# then remove the v1.8.0 entry from ci/support-matrix.yaml

Update the table in patches/archive/README.md and the supported-versions table in the top-level README to match.

Releases already published in this repo are not affected: retiring the patches stops future builds but leaves existing release assets downloadable until you delete them by hand in Gitea.

To rebuild an archived version deliberately, pass PATCH_DIR instead of re-adding it to the matrix:

PATCH_DIR=patches/archive/v1.8.0 VGW_TAG=v1.8.0 bash ci/apply-patches.sh

CI configuration

• Triggers: a v* tag push (builds that tag) or manual dispatch with a tag input (one-shot). No cron.
• Runner: any Linux runner — runs-on: ubuntu-latest. The Go toolchain comes from the golang:latest container, so the runner only needs Docker; nothing version-specific to install.
• Container: golang:latest — pure-Go build, GoReleaser's bundled nfpm makes deb/rpm without rpmbuild/dpkg.
• Secret TOKEN_GITEA: Gitea PAT with write:repository. Used both to clone this repo (the workflow does a manual token-rewritten clone, since a stock container has no Node for actions/checkout) and to publish releases.
• Variable VGW_REPO_URL (optional): override the upstream URL.

Release tagging

Releases are created in this repo with the upstream tag name (v1.11.0). If you need to ship a rebuild of the same upstream version (e.g. a patch fix), set RELEASE_TAG in ci/publish-release.sh's environment to something like v1.11.0-acg2 to avoid clobbering.