From fb92d46e2d21367e794895836051b4bd9bb425a1 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Thu, 6 Aug 2026 15:53:49 -0700 Subject: [PATCH] helm: enterprise license Secret, and a persistent-claim option for master data (#10601) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * helm: mount an enterprise license Secret into every component Running the enterprise image under this chart meant hand-rolling extraVolumes and extraVolumeMounts on every component. Missing one is easy and quiet: a component without the license silently drops to community mode, and on the admin that surfaces only as Data Recovery and Point-in-Time Recovery refusing to enable, with the master looking fine. Add global.seaweedfs.license.existingSecret. The Secret is mounted read-only into master, volume, filer, s3, sftp, admin, worker and all-in-one, and SEAWEED_LICENSE points every process at the file rather than relying on the binary's search paths, which depend on the working directory. The mount is a directory, never a subPath: kubelet refreshes Secret contents in place, but a subPath is resolved once at container start and never updates, which would break license renewal. There is deliberately no checksum annotation on the pod template either — that would roll every pod on renewal, the opposite of what is wanted. Verified on kind: the renewed file reached a running master ~70s after the Secret was patched, same pod UID, restartCount 0. Also documents that only the master re-reads the license on a timer today; the other components pick a renewal up on their next restart. * helm: keep master data on a claim by default The master's -mdir holds its Raft log and snapshots, and with them the cluster's topology UUID — the identity an enterprise license is issued against. It defaulted to a hostPath under /ssd, which does not follow a rescheduled pod: the master came back with an empty data directory, a freshly generated cluster UUID, and a license that no longer matched. With the chart's default of a single master replica there is no peer to recover the identity from either. Default master.data.type to persistentVolumeClaim, sized 1Gi (Raft state is small). hostPath stays available for anyone who wants it. This is breaking for existing releases: volumeClaimTemplates is immutable, so helm upgrade on a release installed with the old default fails with "updates to statefulset spec for fields other than ... are forbidden". Verified on kind, along with both ways out — pinning master.data.type=hostPath upgrades cleanly, and the documented migration (stop the master, pre-seed a claim named after the StatefulSet, upgrade) preserves the cluster UUID. Seeding has to happen while the master is stopped; copying into a live pod loses the state, since the running master rewrites its Raft files before the restart. * helm: mount the license on masters only The master is what reads the license file: it validates it, enforces the capacity limit and binds it to the cluster UUID. Mounting the Secret on volume, filer, s3, sftp, admin and worker put it in six more containers that never look at it, so drop it there and keep master plus all-in-one, which runs `weed server -master`. Two fixes from review while here: - project only the configured key out of the Secret, so an unrelated key in the same Secret is not exposed to the container. Verified the key-scoped projection still updates in place: patching the Secret reached the running master in ~50s, same pod UID, restartCount 0. - drop SEAWEED_LICENSE from merged extraEnvironmentVars while a license Secret is configured. It used to be possible to render the key twice in one container, with the user's value winning over the path the chart actually mounts. CI now pins the scope (master only, all-in-one separately), the key-scoped projection, readOnly, and that SEAWEED_LICENSE renders once. * helm: fix the documented master-data migration The seed pod in the migration never mounted the claim it was supposed to seed, so following the steps verbatim copied the Raft state onto the pod's ephemeral filesystem and threw it away with the pod — landing the reader in exactly the empty-claim, new-cluster-UUID state the section exists to avoid. Give the pod the volume. The names were assembled as -seaweedfs-*, which is wrong whenever the release name already contains the chart name or an override is set; read the StatefulSet name from the cluster instead and derive the claim from it. Also scope the procedure to the chart's single-master default, and create the Secret in the release namespace. Trims the enterprise prose this section had accumulated: this is the OSS chart, and the master-data default is a durability fix that stands on its own. * helm: quote the projected license key, reserve it on the secret env path A secretKey that YAML reads as a non-string (123, yes, no) rendered unquoted into the volume's items, so the projection would not match the Secret's key. Quote both key and path. all-in-one renders secretExtraEnvironmentVars itself, outside the merge helper that already drops SEAWEED_LICENSE, so an entry there could still render the variable twice. Skip it there too while a license Secret is configured. The master template has no such block, so this is the only remaining path. * helm: correct the license helper comments after scoping to masters * helm: keep hostPath as the master data default Defaulting master.data.type to a claim broke every existing release: volumeClaimTemplates is immutable on a StatefulSet, so helm upgrade failed with "updates to statefulset spec for fields other than ... are forbidden" before it changed anything. Keep hostPath as the default and document the claim as the option to choose — for a new install, or for an existing one via the migration already in the README. The chart supported both types all along; only the default moves back. Every immutable field of every rendered StatefulSet is now identical to upstream under default values, so an in-place upgrade cannot trip the API. Verified on kind: install with the unmodified upstream chart, upgrade to this branch (ok), upgrade again turning the license Secret on (ok, volume added in place). A fresh install with master.data.type=persistentVolumeClaim binds its claim as before. The whole PR is additive now: nothing renders differently until a value is set. * helm: scope the migration's StatefulSet lookup to the release * helm: trim the comments added by this change --- .github/workflows/helm_ci.yml | 174 ++++++++++++++++++ k8s/charts/seaweedfs/README.md | 149 +++++++++++++++ .../all-in-one/all-in-one-deployment.yaml | 5 + .../templates/master/master-statefulset.yaml | 3 + .../seaweedfs/templates/shared/_helpers.tpl | 46 +++++ k8s/charts/seaweedfs/values.yaml | 14 ++ 6 files changed, 391 insertions(+) diff --git a/.github/workflows/helm_ci.yml b/.github/workflows/helm_ci.yml index d92c1e355..554f85fbc 100644 --- a/.github/workflows/helm_ci.yml +++ b/.github/workflows/helm_ci.yml @@ -1296,6 +1296,180 @@ jobs: PYEOF echo "NetworkPolicy rendering tests passed" + echo "" + echo "=== Testing enterprise license mount + master persistence ===" + python3 - "$CHART_DIR" <<'PYEOF' + import subprocess, sys, yaml + chart = sys.argv[1] + + def render(values): + args = ["helm", "template", "test", chart] + for k, v in values.items(): + args += ["--set", f"{k}={v}"] + return subprocess.check_output(args, text=True) + + def workloads(manifest): + for d in yaml.safe_load_all(manifest): + if d and d.get("kind") in ("Deployment", "StatefulSet"): + yield d + + ALL_ON = { + "admin.enabled": "true", + "s3.enabled": "true", + "sftp.enabled": "true", + "worker.enabled": "true", + } + failed = [] + + # Case 1: unconfigured renders nothing, so existing installs are unaffected. + out = render(ALL_ON) + if "SEAWEED_LICENSE" in out or "seaweedfs-license" in out: + failed.append("defaults: license plumbing should not render") + else: + print("defaults: no license volume/env (unchanged)") + + # Case 2: only the workloads that run a master carry the license. + LICENSE_PATH = "/etc/seaweedfs/license/seaweed-license.json" + out = render(dict(ALL_ON, **{ + "global.seaweedfs.license.existingSecret": "lic", + })) + carrying = set() + for d in workloads(out): + name = d["metadata"]["name"] + pod = d["spec"]["template"]["spec"] + vols = [v for v in pod.get("volumes", []) if v.get("name") == "seaweedfs-license"] + if not vols: + continue + carrying.add(name) + secret = vols[0].get("secret", {}) + if secret.get("secretName") != "lic": + failed.append(f"{name}: license volume points at {secret.get('secretName')!r}, not the configured Secret") + items = [i.get("key") for i in secret.get("items", [])] + if items != ["seaweed-license.json"]: + failed.append(f"{name}: license volume projects {items}, expected only the license key") + for c in pod.get("containers", []): + mounts = [m for m in c.get("volumeMounts", []) if m["name"] == "seaweedfs-license"] + if not mounts: + failed.append(f"{name}/{c['name']}: license volume not mounted") + continue + # subPath would freeze the file at container start. + if any("subPath" in m for m in mounts): + failed.append(f"{name}/{c['name']}: license mount uses subPath (renewals would not propagate)") + if not all(m.get("readOnly") for m in mounts): + failed.append(f"{name}/{c['name']}: license mount is not readOnly") + env = {e["name"]: e.get("value") for e in c.get("env", [])} + if env.get("SEAWEED_LICENSE") != LICENSE_PATH: + failed.append(f"{name}/{c['name']}: SEAWEED_LICENSE not set to the mounted file") + if carrying != {"test-seaweedfs-master"}: + failed.append(f"license: expected only the master to carry it, got {sorted(carrying)}") + else: + print("license: master only, key-scoped, readOnly, no subPath, SEAWEED_LICENSE set") + + # all-in-one runs `weed server -master`, so it needs the license too. + out = render({ + "allInOne.enabled": "true", + "master.enabled": "false", + "volume.enabled": "false", + "filer.enabled": "false", + "global.seaweedfs.license.existingSecret": "lic", + }) + aio = [d for d in workloads(out) if d["metadata"]["name"].endswith("-all-in-one")] + if len(aio) != 1: + failed.append(f"all-in-one: expected 1 workload, got {len(aio)}") + else: + pod = aio[0]["spec"]["template"]["spec"] + if not any(v.get("name") == "seaweedfs-license" for v in pod.get("volumes", [])): + failed.append("all-in-one: missing the license volume") + else: + print("all-in-one: carries the license") + + # SEAWEED_LICENSE is reserved on both env paths. + out = render({ + "global.seaweedfs.license.existingSecret": "lic", + "global.seaweedfs.extraEnvironmentVars.SEAWEED_LICENSE": "/tmp/bogus.json", + }) + for d in workloads(out): + if not d["metadata"]["name"].endswith("-master"): + continue + for c in d["spec"]["template"]["spec"].get("containers", []): + names = [e["name"] for e in c.get("env", [])] + if names.count("SEAWEED_LICENSE") != 1: + failed.append(f"SEAWEED_LICENSE rendered {names.count('SEAWEED_LICENSE')} times") + else: + value = next(e.get("value") for e in c["env"] if e["name"] == "SEAWEED_LICENSE") + if value != LICENSE_PATH: + failed.append(f"SEAWEED_LICENSE overridden to {value!r} by extraEnvironmentVars") + else: + print("SEAWEED_LICENSE: reserved, rendered once, points at the mount") + + # secretExtraEnvironmentVars is rendered outside the merge helper. + out = render({ + "allInOne.enabled": "true", + "master.enabled": "false", + "volume.enabled": "false", + "filer.enabled": "false", + "global.seaweedfs.license.existingSecret": "lic", + "allInOne.secretExtraEnvironmentVars.SEAWEED_LICENSE.secretKeyRef.name": "x", + "allInOne.secretExtraEnvironmentVars.SEAWEED_LICENSE.secretKeyRef.key": "y", + }) + for d in workloads(out): + if not d["metadata"]["name"].endswith("-all-in-one"): + continue + for c in d["spec"]["template"]["spec"].get("containers", []): + names = [e["name"] for e in c.get("env", [])] + if names.count("SEAWEED_LICENSE") != 1: + failed.append(f"all-in-one: SEAWEED_LICENSE rendered {names.count('SEAWEED_LICENSE')} times via secretExtraEnvironmentVars") + elif next(e.get("value") for e in c["env"] if e["name"] == "SEAWEED_LICENSE") != LICENSE_PATH: + failed.append("all-in-one: secretExtraEnvironmentVars overrode SEAWEED_LICENSE") + else: + print("all-in-one: SEAWEED_LICENSE reserved on the secret env path too") + + # Case 3: the default stays hostPath. volumeClaimTemplates is immutable, + # so growing one here would break helm upgrade on existing releases. + masters = [d for d in workloads(render({})) if d["metadata"]["name"].endswith("-master")] + if len(masters) != 1: + failed.append(f"master: expected 1 StatefulSet by default, got {len(masters)}") + else: + d = masters[0] + claims = [c["metadata"]["name"] for c in d["spec"].get("volumeClaimTemplates", [])] + if any(c.startswith("data-") for c in claims): + failed.append(f"master: default grew a data volumeClaimTemplate ({claims}); " + "that breaks helm upgrade on existing releases") + data_host = [v["name"] for v in d["spec"]["template"]["spec"].get("volumes", []) + if v.get("hostPath") and v["name"].startswith("data-")] + if not data_host: + failed.append("master: default no longer renders a hostPath data volume") + else: + print("master: default still hostPath (upgrade-compatible)") + + # Case 4: opting into a claim renders one, with the requested size. + masters = [d for d in workloads(render({"master.data.type": "persistentVolumeClaim"})) + if d["metadata"]["name"].endswith("-master")] + if len(masters) != 1: + failed.append(f"master.data.type=persistentVolumeClaim: expected 1 StatefulSet, got {len(masters)}") + else: + d = masters[0] + claims = {c["metadata"]["name"]: c["spec"]["resources"]["requests"]["storage"] + for c in d["spec"].get("volumeClaimTemplates", [])} + data = {k: v for k, v in claims.items() if k.startswith("data-")} + if not data: + failed.append(f"master.data.type=persistentVolumeClaim: no data claim rendered ({claims})") + elif set(data.values()) != {"1Gi"}: + failed.append(f"master.data.type=persistentVolumeClaim: unexpected size {data}") + else: + print(f"master.data.type=persistentVolumeClaim: renders {sorted(data)} at 1Gi") + if any(v.get("hostPath") and v["name"].startswith("data-") + for v in d["spec"]["template"]["spec"].get("volumes", [])): + failed.append("master.data.type=persistentVolumeClaim: data still on a hostPath") + + if failed: + print("\nFAIL:", file=sys.stderr) + for f in failed: + print(f" - {f}", file=sys.stderr) + sys.exit(1) + PYEOF + echo "License + master persistence tests passed" + echo "All template rendering tests passed!" - name: Create kind cluster diff --git a/k8s/charts/seaweedfs/README.md b/k8s/charts/seaweedfs/README.md index cf6378bdf..7edde6754 100644 --- a/k8s/charts/seaweedfs/README.md +++ b/k8s/charts/seaweedfs/README.md @@ -107,6 +107,120 @@ https://github.com/rancher/local-path-provisioner you can use ANY storage class you like, just update the correct storage-class for your deployment. +### Master data: hostPath vs a claim + +The master's `-mdir` holds its Raft log and snapshots, and with them the +cluster's identity (its topology UUID). `master.data.type` defaults to +`hostPath`, which does not follow a pod to another node: a master that is +rescheduled comes back with an empty data directory and a brand new cluster +UUID. With the chart's default of a single master replica there is no peer to +recover the identity from either. + +Putting the master's data on a claim avoids that: + +```yaml +master: + data: + type: "persistentVolumeClaim" + size: "1Gi" + storageClass: "" # empty uses the cluster's default StorageClass +``` + +Raft state is small, so a modest claim is enough — sizing matters far more for +volume and filer. + +The default is left at `hostPath` for backward compatibility: +`volumeClaimTemplates` is immutable on a StatefulSet, so flipping the type on a +release that already exists fails, whether the chart changes the default or you +change it yourself: + +```text +StatefulSet.apps "-seaweedfs-master" is invalid: spec: Forbidden: +updates to statefulset spec for fields other than 'replicas', ... are forbidden +``` + +New installs can set the claim from the start. To move an **existing** release +onto a claim without losing the cluster UUID, use the migration below. The +claim has to be seeded while the master is stopped: a running master rewrites +its Raft state, so copying into a live pod is silently undone by the next +restart. + +The steps below are for the chart's default of a single master +(`master.replicas: 1`). With several master replicas, repeat steps 1, 3 and 4 +for every ordinal, or migrate one at a time and let the remaining quorum +re-replicate. + +Take the names from the cluster rather than assembling them — the release +name, `nameOverride` and `fullnameOverride` all feed the chart's fullname +helper, so `-seaweedfs` is not always right: + +```bash +NS=; REL= +# scope by instance as well as component: several releases can share a namespace +STS=$(kubectl -n $NS get sts \ + -l app.kubernetes.io/instance=$REL,app.kubernetes.io/component=master \ + -o jsonpath='{.items[0].metadata.name}') +POD=$STS-0 +# a StatefulSet names its claims