mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-18 21:26:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(g15b): update K8s rerun target for primary alignment

Browse Source
This commit is contained in:
pingqiu
2026-05-03 10:10:27 -07:00
parent 83b953b613
commit 3f01c8b082
2 changed files with 33 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sw-block/design/test/v3-phase-15-g15b-k8s-qa-test-instruction.md
+14 -4
View File
@@ -1,14 +1,14 @@
# V3 Phase 15 G15b Kubernetes Static PV QA Test Instruction
**Date**: 2026-05-03
**Status**: K8s lab instruction for `p15-g15b/k8s-static-pv@eb13105`; M02 re-run pending
**Status**: K8s lab instruction for `p15-g15b/k8s-static-pv@95b7217`; M02 re-run pending
**Scope**: single-node Kubernetes static PV/PVC/pod smoke through real V3 daemons and CSI.
---
## Headline
At `seaweed_block@eb13105`, the G15b lab harness, image build inputs, and M02 DNS/log-preservation fixes are staged to prove:
At `seaweed_block@95b7217`, the G15b lab harness, image build inputs, M02 DNS/log-preservation fixes, CSI node `iscsi_tcp` module loading, and deterministic attachable-primary binding are staged to prove:
```text
blockmaster + product-loop + r1/r2 blockvolume
@@ -39,12 +39,17 @@ Known current local limitation:
- On the current dev workstation, `kubectl` context `rancher-desktop` exists but API server is not reachable. This instruction needs QA or a running K8s lab.
M02 first-run blocker fixed:
M02 first-run blockers fixed:
- `5375add` failed because `hostNetwork: true` blockvolume pods inherited host DNS and could not resolve `blockmaster.kube-system.svc.cluster.local`.
- `eb13105` adds `dnsPolicy: ClusterFirstWithHostNet` to both blockvolume pods.
- `eb13105` also collects daemon logs on every exit before cleanup, so failure evidence is preserved.
M02 second-run blockers fixed:
- `eb13105` reached iSCSI attach but failed when M02 lacked the `iscsi_tcp` kernel module. `95b7217` adds a privileged CSI node init container that runs `modprobe iscsi_tcp || true` and adds `kmod` to the CSI image.
- after manual module load, the lab exposed a primary/listener mismatch: r2 could win authority while only r1 exposed the static loopback iSCSI target. `95b7217` changes the G9F-2 bridge so RF>1 verified placements produce a single deterministic frontend-primary `Bind` ask and pins r1/s1 as the first attachable placement slot.
---
## Commands
@@ -55,6 +60,8 @@ Pre-flight from the code repo:
cd C:\work\seaweed_block_g9c
git checkout p15-g15b/k8s-static-pv
go test ./cmd/blockcsi -run TestG15b_Manifest -count=1 -v
go test ./core/host/master -run 'TestG9F2|TestG15b_ProductLoop' -count=1 -v
go test ./internal/testops ./core/host/master ./cmd/blockcsi -count=1
go test ./core/csi ./cmd/blockcsi ./core/host/volume ./core/host/master ./core/authority ./cmd/blockmaster ./cmd/blockvolume -count=1
```
@@ -72,7 +79,7 @@ G15B_KIND_CLUSTER=<kind-cluster-name> bash scripts/build-g15b-images.sh "$PWD"
Local image build result already verified at `5375add`: PASS, images `sw-block:local` and `sw-block-csi:local` built.
After pulling `eb13105`, rebuild images before rerun:
After pulling `95b7217`, rebuild images before rerun:
```bash
bash scripts/build-g15b-images.sh "$PWD"
@@ -102,8 +109,11 @@ Expected result:
| Manifest attach path requires `ControllerPublish`. | `TestG15b_Manifest_CSIDriverRequiresAttach` |
| Controller deploy uses external-attacher, not provisioner. | `TestG15b_Manifest_ControllerUsesAttacherNotProvisioner` |
| Product stack uses G9G cluster-spec/product-loop and RF=2 r1/r2. | `TestG15b_Manifest_ProductStackSingleNodeLoopbackShape` |
| Product loop emits one deterministic frontend-primary Bind for RF=2 static placement. | `TestG15b_ProductLoop_RF2PlacementEmitsSingleDeterministicBind` |
| Attachable r1/s1 is the first placement slot and only r1 exposes the static iSCSI target. | `TestG15b_Manifest_AttachableReplicaIsFirstPlacementSlot` |
| Static PV does not carry target endpoint truth. | `TestG15b_Manifest_StaticPVDoesNotEmbedTargetFacts` |
| Node plugin has privileged host mount shape. | `TestG15b_Manifest_NodePluginPrivilegedShape` |
| Node plugin loads host `iscsi_tcp` before attach. | `TestG15b_Manifest_NodePluginLoadsISCSITCPModule` |
| Pod write/read checksum path. | `scripts/run-g15b-k8s-static.sh` pod phase + `pod.log` |
| Product logs captured for debug. | `blockmaster.log`, `blockvolume-r1.log`, `blockvolume-r2.log`, `blockcsi-controller.log` |
sw-block/design/v3-phase-15-g15b-k8s-static-pv-mini-plan.md
+19 -4
View File
@@ -1,7 +1,7 @@
# V3 Phase 15 — G15b Kubernetes Static PV Mini-Plan
**Date**: 2026-05-03
**Status**: G15b-1 manifests implemented at `62325c9`; G15b-2 lab harness staged at `32b3a13`; image build inputs added at `5375add`; M02 first run found DNS/harness blockers; fixed at `eb13105`; Kubernetes re-run pending
**Status**: G15b-1 manifests implemented at `62325c9`; G15b-2 lab harness staged at `32b3a13`; image build inputs added at `5375add`; M02 first run found DNS/harness blockers fixed at `eb13105`; M02 second run found `iscsi_tcp` + primary/listener alignment blockers fixed at `95b7217`; Kubernetes re-run pending
**Branch**: `p15-g15b/k8s-static-pv` from `ac49adb`
**Goal**: prove a Kubernetes pod can consume a pre-provisioned V3 block volume through `cmd/blockcsi`, using real Kubernetes CSI control flow and real Linux iSCSI staging.
@@ -162,7 +162,7 @@ Result: PASS on `62325c9`.
### G15b-2 — K8s Lab Harness
Status: **harness staged** at `seaweed_block@32b3a13`; image build inputs added at `seaweed_block@5375add`; DNS/logging fixes at `seaweed_block@eb13105`; real Kubernetes re-run pending.
Status: **harness staged** at `seaweed_block@32b3a13`; image build inputs added at `seaweed_block@5375add`; DNS/logging fixes at `seaweed_block@eb13105`; iSCSI TCP module loading + deterministic attachable-primary bridge fix at `seaweed_block@95b7217`; real Kubernetes re-run pending.
Artifacts:
@@ -200,6 +200,19 @@ Fix at `eb13105`:
- changes `scripts/run-g15b-k8s-static.sh` so daemon logs are collected from the EXIT trap before cleanup;
- adds `.gitattributes` to keep `*.sh` as LF on future checkouts.
M02 second-run findings:
- M02 did not have `iscsi_tcp` loaded, so kubelet/CSI `NodeStage` failed with `iSCSI driver tcp is not loaded`;
- after manual `modprobe iscsi_tcp`, Kubernetes reached `ControllerPublish` and iSCSI login, but the volume still did not materialize because r2 won authority while only r1 exposed the static loopback iSCSI target;
- root cause: the G9F-2 product bridge mapped every RF=2 verified placement slot to a competing `Bind` ask for the same volume; authority's queue keeps latest desired ask per volume, so r2 could overwrite r1 nondeterministically.
Fix at `95b7217`:
- adds a privileged CSI node init container that runs `modprobe iscsi_tcp || true` and mounts host `/lib/modules`;
- adds `kmod` to the `sw-block-csi:local` image;
- changes the G9F-2 verified-placement bridge so RF>1 slots produce one deterministic frontend-primary `Bind` ask (first verified slot) instead of multiple competing `Bind` asks;
- pins the G15b static manifest assumption that the attachable iSCSI replica (`r1/s1`) is the first placement slot and that r2 does not expose a competing static iSCSI target.
Harness responsibilities:
1. Build V3 binaries/images for `blockmaster`, `blockvolume`, and `blockcsi`.
@@ -218,10 +231,12 @@ Pass:
- Pod writes and reads byte-equal data.
- No dangling iSCSI session for the test IQN after cleanup.
Pre-flight verification green at `eb13105`:
Pre-flight verification green at `95b7217`:
```powershell
go test ./cmd/blockcsi -run TestG15b_Manifest -count=1 -v
go test ./core/host/master -run 'TestG9F2|TestG15b_ProductLoop' -count=1 -v
go test ./internal/testops ./core/host/master ./cmd/blockcsi -count=1
go test ./core/csi ./cmd/blockcsi ./core/host/volume ./core/host/master ./core/authority ./cmd/blockmaster ./cmd/blockvolume -count=1
```
@@ -236,7 +251,7 @@ Result: PASS; built `sw-block:local` and `sw-block-csi:local`.
Not yet proven:
- Kubernetes API server availability;
- image load path into the target cluster after rebuilding at `eb13105`;
- image load path into the target cluster after rebuilding at `95b7217`;
- external-attacher calling `ControllerPublish`;
- kubelet calling `NodeStage` / `NodePublish`;
- pod checksum write/read.