Per architect single-sign of G5-5 §close (`seaweedfs@c78116fd2`):
(a) v3-dev-roadmap.md
- §3: G5 line note now mentions G5-5 closed at L3 + G5-5C carry-forward
- §4: G5-5 row → CLOSED (link to seaweedfs@c78116fd2); G5-5C row added
as next active gate with bound pass criterion
- §7: G5-5 close commit appended to recently-closed table
(seaweed_block@5c4718f + seaweedfs@c78116fd2, L3 reached, #4 carry)
(b) v3-phase-15-g5-5c-mini-plan.md (new) v0.1 kickoff
- §1 scope: peer recovery trigger after replica restart; reuse T4d-4
primitives (architect binding); no engine logic change
- §1.A: three trigger source options (A master observation, B periodic
probe, C transport reconnect) with tradeoffs; sw recommends A; final
pick deferred to architect ratification
- §2: 6 acceptance criteria, hardware step is exactly G5-5 #4
(verify_restart_catchup → GREEN with no harness changes)
- §3: 2 new INVs proposed (REPL-PEER-RECOVERY-TRIGGER-001 +
-NO-RETRIGGER-LOOP) + 2 deferred ledger pointers from G5-5 close
- §4: G-1 N/A (new build, no V2 PORT)
- §5: forward-carries from G5-5 §close all addressed
- §6: 5 risks tabled
- §7: sign table awaiting architect §1-§6 ratification including
trigger source pick
Standing by for architect ratification of trigger source binding.
No code starts until §1-§6 signed.
Architect's round-15 hygiene callout: §7 sign table still had
three pre-code 'blocked' rows after the real close-state rows
landed in the prior doc-fix commit. Pure leftover from before
the close-state update overwrote earlier rows but didn't delete
the trailing pre-code rows.
Removed:
- 'Code start (script + Go helper) ... blocked on ratification'
- 'm01 hardware verification run ... blocked'
- '§close append + close sign ... blocked'
Sign table now ends cleanly at the §close architect single-sign
pending row. Ready for sign.
§close summary per v3-batch-process.md §12 template:
Done:
- #1 verify_cluster_ready
- #2 verify_byte_equal — live iSCSI replicated write, byte-equal
verified via storage-aware m01verify (LBA[0]=0xab on cross-host
hardware)
- #3 verify_network_catchup — iptables disconnect+heal, replica
converges to LBA[1]=0xcd byte-equal in 8s via engine-driven
catch-up
- 14 bugs surfaced+fixed across 14 m01 self-iteration rounds
- 5 INV-BIN-WIRING-* invariants in v3-invariant-ledger.md from
G5-4 still load-bearing; G5-5 hardware run is Integration backstop
Not done:
- #4 verify_restart_catchup — kill replica + write while down +
restart: replica's LBA[2]=0xef does NOT converge in 30s. Per
architect ruling 2 (round 13): real recovery-path finding,
surface as G5-5C carry-forward.
- #5 verify_race_stress + #6 verify_full_suite — gated on #4 fix
or test sequencing rework.
Product level reached: L3 (Replicated IO) per v3-architecture.md §13.
Falls short of full L4 (Failure/recovery under IO) — process-restart
recovery is the gap, scoped as G5-5C.
Next gate that makes it usable: G5-5C Peer Recovery Trigger After
Replica Restart — fix engine-driven catch-up re-trigger when a
degraded peer becomes reachable again. After G5-5C: re-run #4#5#6
in this same harness; full L4 reached.
Forward-carries to G5-5C (architect-bound 2026-04-27):
- Reuse existing engine-driven recovery primitives (T4d-4); no
ad-hoc re-ship from replication layer.
- Define trigger source first: observation reappearance, periodic
probe loop, or stream/transport reconnect signal.
- Pass criterion: exactly the failed hardware case from G5-5 #4.
- Seed evidence: seaweed_block@5c4718f primary-fail.log shows the
gate-degraded + stale-barrier-ack pattern.
Forward-carries to opportunistic future hardening:
- Unit test for EnsureStorage→assignment-arrives→first-Open
Identity-latch path (would have caught round-10/11 bug pre-m01).
- Generalize start_cluster() pre-flight stale-state cleanup pattern
for future hardware harnesses.
Forward-carries to G5-6:
- G5-DECISION-001 (Path A vs Path B) — engine-state serializability
pinned in T4d still holds; G5-5 doesn't change posture.
Pending: architect single-sign on §close per v3-batch-process.md §5
+ §8C.2.
Refs: 24 commits in seaweed_block@phase-15 spanning rounds 1-14
(documented in §close.evidence.commits table).
Architect's v0.2 review caught that §1 absorbed the 3 binding revisions
but §2 (the close contract per v3-batch-process.md §2) stayed stale:
- §2 #2 still said "byte-equal on replica's walstore extent"
- §2 had old #4 (race stress) instead of new #4 (process restart)
- §2 #3 didn't name /status/recovery as the R/H source
v0.3 rewrites §2 to match §1, with explicit verifier names:
#1 verify_cluster_ready
#2 verify_byte_equal — m01verify Go helper using walstore.OpenReadOnly
+ storage.LogicalStorage.Read(lba) + SHA-256 (NO raw extent peek)
#3 verify_network_catchup — iptables disconnect + polls
/status/recovery?volume=v1 for R/H; asserts RecoveryDecision="catch_up"
#4 verify_restart_catchup — SIGTERM replica + restart same binary +
same --durable-root; polls /status/recovery same as #3#5 verify_race_stress — 10x -race on G5-4.5 integration test
#6 verify_full_suite — go test ./... clean from m01
#7 v3-dev-roadmap.md updated at gate-close per v3-batch-process.md §8
§1 file map and §5 forward-carry table already match v0.3 numbering
(grep confirmed no stale references). Implementation scope unchanged
from v0.2 (~310 prod LOC + ~30 unit tests).
v3-batch-process.md §2 single-source-of-truth discipline preserved:
§2 acceptance criteria IS the close contract; §1 scope description
stays in sync but is not load-bearing for close evidence.
Addresses 3 architect revision requirements (round 51):
REVISION 1 — process restart distinct from network disconnect:
Split G5-4 #4 forward-carry into TWO scenarios:
§2 #3 network disconnect (iptables) — proves live TCP interrupt
+ recovery without process restart
§2 #4 replica process stop/restart — proves durable reopen +
master resubscribe + recovery reconstruction
G5-4 #4 is now FULLY consumed (was: only network proxy in v0.1).
REVISION 2 — storage-aware byte verifier:
Replace raw walstore .extent peek with storage-abstraction Read(lba):
helper opens replica's walstore via core/storage/walstore (or
equivalent OpenReadOnly path), invokes Read(lba) per LBA in the
range, SHA-256 vs primary's known payload. Raw extent peek
REJECTED — walstore on-disk includes WAL frames + checkpoints
+ sparse regions + potentially-stale-but-valid blocks; only
Read(lba) returns the authoritative current value.
Risk added: if walstore.OpenReadOnly is missing, sw adds it as
part of this batch (small scope expansion contained in
core/storage/walstore; read-only opener for verification only,
NOT a substrate semantic change).
REVISION 3 — named R/H observation source:
/status?volume=v1 returns frontend.Projection (no R/S/H). G5-5
adds /status/recovery?volume=v1 returning engine.ReplicaProjection
(Mode, R, S, H, RecoveryDecision); gated by new --status-recovery
daemon flag (default off; production binaries don't enable).
Loopback-only via existing isLoopbackRemote guard. ~30 prod LOC
+ ~30 unit tests. Engine/adapter logic unchanged — surfaces
already-computed projection through HTTP.
Updated §1 file map, §1.4 truth-domain check, §5 forward-carry
table, §6 risks (3 new rows), §close template unchanged.
Re-submitted for architect §1-§6 ratification. After ratify, sw
codes per §1 file map; estimate ~310 prod LOC + ~30 unit tests.
Codifies the lessons from T4 + G5-4 retrospective:
KEEP — earned its keep on T4:
- G-1 V2 PORT read (saved 5 hidden invariants on T4b-4, probe non-
mutation pin on T4c-2, 3 placement decisions on T4d-3)
- Mini-plan acceptance criteria (single source of truth for close)
- Invariant ledger discipline ("claim without test = wish")
- m01 -race verification (caught 2 engine bugs at T4d-4 part C)
- Architect single-sign at close (caught 4 stale refs at G5-4 close)
DROP — overhead without payoff:
- Separate kickoff PROPOSAL doc (mini-plan §1-§6 = same thing)
- Separate G-1 doc (inline §4 of mini-plan)
- Separate closure report doc (§close section of mini-plan)
- Separate forward-carry checklist (§5 of next-batch mini-plan)
- Separate QA scenario catalogue (write tests directly when ready)
- Multi-version doc churn (v0.1→v0.5)
- Cross-doc invariant restatement (ledger is sole source)
- Mixed T-track + G-N naming for same gate
Compressed sign cycles: was 4-5 architect signs per batch; now 2
(scope ratify + close sign).
Per-batch artifact count: was 5+ (kickoff + mini-plan + G-1 +
closure + checklist + scenario catalogue); now 1 (mini-plan with
§close appended).
Decision rules codified:
§6.1 G-1 yes/no (V2 PORT yes; V3-native no)
§6.2 T-track vs G-N naming (architect picks at kickoff)
§6.3 When to skip mini-plan (1-line hotfix-class)
§8 names the 6 first-order control docs to keep current
(v3-dev-roadmap, v3-phase-15-mvp-scope-gates, v3-invariant-ledger,
v3-block-behavior-contract-index, v3-product-placement-authority-
rationale, v2-v3-contract-bridge-catalogue).
§9 first trial: G5-5 m01 hardware first-light.
§11 honesty principle: documentation that catches bugs is
discipline; documentation that doesn't is ceremony. Drop ceremony,
keep discipline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-atomic with seaweed_block@c820e17 per architect binding round 50
(mini-plan v0.4 §4 #7): ledger inscription required at G5-4 close.
- INV-BIN-WIRING-ROLE-FROM-ASSIGNMENT
- INV-BIN-WIRING-PEER-SET-FROM-ASSIGNMENT-FACT
- INV-BIN-WIRING-LISTENER-LIFECYCLE-LIFO
- INV-BIN-WIRING-ASSIGNMENT-DRIVES-MEMBERPRESENT
- INV-BIN-WIRING-SESSIONID-VIA-ADAPTER
All 5 are ACTIVE with test pointers to
cmd/blockvolume/g5_4_l2_replication_test.go (subprocess integration)
+ source-side checks in cmd/blockvolume/main.go and core/host/volume.
Last verified 2026-04-26 (G5-4 close).
Architect ratification 2026-04-26:
"Role inference, in-process acceptance, G5-DECISION-001 seam, and
sessionID discipline are architecturally correct. G-1 must clarify
replica readiness semantics and confirm ctrl-addr reuse or introduce
repl-addr before code."
2 binding clarifications baked into v0.3:
#1 — §4 #2 acceptance criterion split by role:
- Primary: Healthy=true per existing frontend/write-ready projection
- Replica: replication-ready / listener-bound + ApplyEntry byte-equal
verified — MUST NOT report Healthy=true if existing field implies
frontend-primary-write-ready
- If existing status field is too coarse, G5-4.5 uses precise
assertion names (assertReplicaReplicationReady,
assertPrimaryFrontendReady) instead of unified assertHealthy
#2 — §4 #7 acceptance criterion strengthened:
- Catalogue inscription ALONE insufficient at G5-4 close
- 5 INV-BIN-WIRING-* invariants MUST land in v3-invariant-ledger.md
- Per v3-quality-system.md §6 "an invariant without a test is a wish"
- Ledger updated as PR atomic with code (not after-the-fact)
§7.1 G-1 deliverable extended (G-1-blocking subitems):
- Replica readiness semantics — what existing volume.Status /
ProjectionView field expresses replication-ready (vs Healthy)?
G-1 either proposes new field OR specifies precise assertion names
- --ctrl-addr reuse confirmation — verify NO conflict with NVMe/iSCSI
control-plane traffic on same port. If conflict, G-1 introduces
--repl-addr flag (small scope expansion, contained in this batch)
§3 #4 predicate flipped to ✅ DONE (architect round 50).
§8 sign table updated with explicit ledger requirement at close.
Architect-pre-baked: ratification stays valid; no further mini-plan
revisions needed before G-1.
Sw next: produce G-1 V3-native PORT read deliverable per §7.1
(includes the 2 binding subitems). Code stays blocked until
architect ratifies G-1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses QA's 3 notes + 1 clarification ask:
Note 1 (role inference):
§1.3 rewritten — fact.ReplicaID is master-minted (proto verified
at control.proto:128-148 + mint site at services.go:198-205).
Binary reads `fact.ReplicaID == self.ReplicaID` directly. No
lex-smallest fallback (master always names exactly one bound
replica per volume per line). Removes the binary-side authority
inference that violated the master-authority rule.
Note 2 (acceptance circular):
§4 #2 verifier reframed to G5-4.5 in-process test. m01 hardware
verification belongs to G5-5; G5-4 closes on the in-process pin.
Note 3 (G5-DECISION-001 contradiction):
§5 rewritten — G5-4 ships Path B runtime AND keeps Path A
serializability seam open. T4d-4 part B's RoundTripJSON test
already pins serializability; G5-4 preserves it. G5-6 architect
ratification can promote to Path A by adding persistence on top
of the existing struct, with no engine-state-shape change.
Clarification ask (sessionID minting):
§6 added INV-BIN-WIRING-SESSIONID-VIA-ADAPTER. Adapter mints
unique sessionIDs via process-wide atomic counter at
adapter.go:70; binary inherits for free as long as it dispatches
via the adapter (never via framework shortcuts that hardcode
sessionID=1, which is the known T4c §I + QA G5-1 round 1 SKIP
gap). Pinning this invariant keeps the gap test-side.
Re-submitted for QA re-review per parent kickoff §7 governance loop.
Mirror cmd/blockvolume to T4d-4 part B's WithEngineDrivenRecovery()
framework binding. Single batch (~250 prod + ~150 tests), 5 ordered
subtasks. Design decisions (a-d per kickoff §3 G5-4 row):
(a) Role inference: assignment-driven, no new CLI flag
(b) Peer discovery: AssignmentFact.Peers per T4a-5 P-refined
(c) Listener lifecycle: --ctrl-addr reuse + LIFO Stop in host.Close()
(d) Engine instantiation: one engine per volume, single --volume-id
Pre-merge gates require G-1 V3-native PORT read of cluster.go:357-369
+ V2 lesson check on weed/storage/blockvol/blockvol.go before code.
4 new invariants to inscribe at close (INV-BIN-WIRING-*).
Submitted for QA + architect ratification per parent kickoff §7
governance loop. No code until ratify.
Hand-off doc v0.3 + G5 kickoff v0.2: m01+M02 bring-up smoke surfaced
that cmd/blockvolume binary lacks T4 replication wiring entirely.
Sw-confirmed root cause:
- --t1-readiness HealthyPathExecutor is primary-only by design
- volume.Config.ReplicationVolume slot exists (host.go:73) with godoc
"T4a-5 production wiring sets this" — but T4a-5 only added the
field; the wiring NEVER landed
- T4d-4 part B wired WithEngineDrivenRecovery() for component test
framework (cluster.go:357-369), NOT for the binary
- Result: V3 components compose end-to-end (proven by T4d HARD GATE
#3); the production binary still constructs a primary-only data
plane
Sw confirmed this is real implementation work (150-300 LOC + design),
not a 50-LOC quick patch. Four design decisions needed:
1. Role inference (assignment vs CLI flag vs topology)
2. Peer discovery (from AssignmentFact.Peers)
3. Listener lifecycle (--data-addr reuse + Stop)
4. Engine instantiation (one engine per volume)
G5 kickoff revised to v0.2:
- 5 batches → 6 batches (binary wiring promoted to G5-4)
- G5-1/2/3 are NOT blocked by G5-4 (component framework already
binds T4d-4 part B; QA scenarios + walstore cadence at
component/primary-only scope can run in parallel)
- G5-4 binary wiring: needs full governance loop (kickoff →
architect ratify → mini-plan → architect ratify → G-1 → code).
G-1 source: T4d-4 part B component framework as V3-native PORT
- G5-5 m01 hardware first-light DEPENDS on G5-4 (script can't
drive replica scenarios until binary supports replicas)
- G5-6 G5-DECISION-001 resolution at G5 close (was G5-5 in v0.1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause for "volume not ready" gate: missing
--expected-slots-per-volume 2 flag on blockmaster.
Default is 3; QA's 2-node topology had 2 slots; controller
silently rejected observation snapshot (cmd/blockmaster/main.go:39).
Fix verified locally on Windows (single-node, no m01/M02 needed):
- Add --expected-slots-per-volume 2 to blockmaster command
- Primary reaches Healthy=true with epoch=1
- assignment-received fires; durable storage opens; status
endpoint serves {"Healthy":true}
Lesson learned (process improvement): for V3-internal bring-up
debug, try single-node local reproduction FIRST. The cluster
bring-up gate is V3 logic, not network topology. Reproduces in
seconds locally with full source-code access; m01/M02 only needed
for cross-node-specific scenarios (real network conditions,
iptables, multi-host wire).
Secondary finding: replica r2 sees primary r1's assignment but
records "supersede, not applying to adapter" because T1
HealthyPathExecutor only handles primary case. For G5-4 replica
bring-up, sw needs to wire T4a-T4d ReplicationVolume + ReplicaPeer
+ ReplicaListener stack (not just --t1-readiness flag). This is
the actual next gap for G5-4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: cmd/blockmaster/main.go hardcoded ExpectedSlotsPerVolume=3.
QA's 2-slot topology silently failed validateVolumeTopology in the
controller, so no assignments were minted, no master-log lines,
and volumes timed out at durable open.
Fix landed in seaweed_block@f5de7c5: --expected-slots-per-volume
CLI flag, default 3, set 2 for the 2-node smoke.
QA next: rebuild blockmaster, pass --expected-slots-per-volume 2
in §3.4 of the handoff command sequence; rest unchanged.
Records QA's cross-node smoke attempt 2026-04-26: infrastructure
fully verified READY (m01+M02 reachability, SMB share for binary
distribution, master cross-node listen, network OK), but cluster
bring-up blocked at V3-internal gate.
Symptom: blockvolume on both nodes connects to master but logs
"durable open: frontend: volume not ready" — never reaches steady
state, status endpoint never binds, master log shows no heartbeat
or assignment-mint events.
Hand-off contents:
- §1 specific questions for sw (5 gaps to fill)
- §2 infrastructure verified READY (no action needed)
- §3 copy-pasteable commands sw can run/debug
(build → topology → master → primary → replica → cleanup)
- §4 QA's hypothesis on the gap (assignment-from-master flow)
- §5 debug suggestions for sw (log levels, integration test
references)
- §6 G5-4 script skeleton current state
- §7 QA's next steps once sw answers
Working dirs reproducible:
- Binaries: /mnt/smb/work/share/g5-binaries/{blockmaster,blockvolume}
- Run state: /tmp/g5sm/ on both nodes
- Logs: /tmp/g5sm/logs/{master,primary,replica}.log
Blocks: G5-4 implementation work (script scenario bodies, hardware
first-light scenarios). Does NOT block QA scenario authoring at
component scope (Cluster framework already covers that).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per QA infra-check round 2026-04-26, surfaces real readiness gaps
before architect ratifies G5-4 schedule:
m01 (192.168.1.181 — primary node):
✅ 32-day uptime; sudo password-less; 16 cores; 19 GiB RAM
✅ 177 GiB free disk; Go 1.26.2 installed
✅ iptables / netns / multi-process tools all available
✅ T2 m01 NVMe script template available as pattern reference
M02 (192.168.1.184 — replica node):
✅ Reachable from m01 (0.92ms); same kernel; 178 GiB free disk
❌ Go NOT installed — must scp binaries from m01
Implication for G5-4:
Build binaries on m01, scp to M02. Same cross-node binary pattern
T2 already uses for its iSCSI target deployment. G5-4 skeleton at
seaweed_block/scripts/iterate-m01-replicated-write.sh implements
this build-then-scp flow.
No infrastructure blockers. Architecture ready as soon as G5 mini-plan
ratifies scenario list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two artifacts landing together to close T4 batch series:
1. v3-phase-15-t4d-closure-report.md (NEW)
QA single-sign artifact for T4d batch close per §8C.2; architect
T-end three-sign per §8C.1 (T4d IS final T4 batch — confirmed at
round-48 review). Round-48 + round-49 corrections incorporated:
- Part C commit hash bound to e642ae8 throughout
- CARRY-T4D-LANE-CONTEXT-001 bind point = post-G5 hardening
backlog (not T4e — consistent with "T-end at this close")
- §H Finding #1 reworded — walstore HAS background flusher
(walstore.go:189-190); QA's earlier "caller-driven" was wrong
- §H Finding #3 RESOLVED at a0be6d5 (T2A NVMe race fixed +
m01 -race ×50 PASS)
- 16 invariants pinned (added 2 named for part C bug fixes:
INV-REPL-FAILED-SESSION-KIND-DRIVES-ESCALATION +
INV-REPL-REBUILD-ESCALATION-STICKY-UNTIL-TERMINAL)
- 22/22 packages green under -race on m01 (post-a0be6d5)
2. v3-phase-15-t4d-mini-plan.md (NEW — was uncommitted across
v0.1 → v0.5 evolution)
Final v0.5 incorporates: architect Path B fold; round-47
rebuild path engine-driven HARD GATE expansion; G5-DECISION-001
named decision record; 4-batch shape ratified; T4d-3 G-1 binding.
Active forward-carries (post-G5 hardening backlog):
- CARRY-T4D-LANE-CONTEXT-001 — replace TargetLSN==1 caller shim
with true handler/session-context lane signal
- G5-DECISION-001 — engine recovery state behavior across
primary restart (Path A persist vs Path B rebuild-from-probe)
G5 collective close items (NOT post-G5):
- m01 hardware first-light for replicated write path
- Multi-replica concurrent live + recovery scenarios
- walstore flusher cadence verification + tuning policy
- Minimal metrics/backpressure assessment
- G5-DECISION-001 architect resolution
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Architect sign by pingqiu 2026-04-25:
"T4d v0.2 scope accepted as one batch series; Option C for appliedLSN
source; BlockStore walHead hotfix may land pre-T4d; substrate defense-
in-depth included where practical; 4-batch order approved; T4d-3 G-1
required; T4d-2 no G-1; T-end three-sign at T4d close if T4d remains
final T4 batch."
All open architect-decision points (§2 scope, §2.5 Option/hotfix/
substrate, §3 batch shape, §4 acceptance bar) resolved. §6 open
issues all closed. §8 inscribes the verbatim ratification record.
Sw clearances effective immediately:
- Land BlockStore walHead one-liner as pre-T4d hotfix (single PR with
un-skipped regression test)
- Produce T4d mini-plan (4-batch shape per §3)
- Produce T4d-3 G-1 V2 read on wal_shipper.go runCatchUpTo
- T4d-2 spec is round-43/44 architect text (no G-1 needed)
T-end horizon: §8C.1 T-end three-sign lands at T4d close IF T4d
remains final T4 batch (per architect's criterion #10 wording tweak).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QA single-sign artifact for T4c batch close per §8C.2; architect
acceptance of §B scope deltas signed 2026-04-25 by pingqiu.
Scope deltas accepted:
- T4c closes as mid-T4 batch under §8C.2, not T4 T-end
- L2/L3 mini-plan bar narrowed to muscle-level L2 + component evidence
- L3 m01 first-light deferred to T4d / G5 final close
- Substring "WAL recycled" matching accepted as TEMPORARY, replacement
bound to T4d (preferred) or G5 final sign (latest)
- INV-REPL-CATCHUP-WITHIN-RETENTION-001 downgraded to T4d blocker
(catch-up sender hardcodes ScanLBAs(1); replica's R+1 not threaded)
Doc-hygiene fixes per PM round-2 review (this commit):
- Drop INV-REPL-CATCHUP-DONE-MARKER-EMITTED (non-existent: V2 marker
collapsed into barrier-as-terminator per catchup_sender.go:48,187)
- §B/#2 + #5 reword "green at HEAD" to acknowledge architect Windows
cleanup-only repro failures (tracked as next-batch carry)
- Active formal-INV count 8 -> 6
Forward-carries to T4d (BLOCKERS):
- R+1 catch-up threading (StartCatchUp signature + adapter wire)
- Full engine→adapter→executor recovery wiring
- Structured RecoveryFailureKind replacing substring sentinel
- LastSentMonotonic_AcrossRetries cross-call form scenario
- Windows TempDir cleanup race investigation
Forward-carry to G5 final close:
- m01 hardware first-light for replicated write path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes QA round-2 feedback loop. Three concerns resolved and one L2-blocker
hazard added.
## Q1-Q3 resolution (sw-verifiable per QA concern; V2 source check)
Q1 scope completeness: VERIFIED complete. V2 grep shows sync_all_* are
three test files only — `sync_all_adversarial_test.go`, `sync_all_bug_test.go`,
`sync_all_protocol_test.go`. Zero production files for sync_all / split_brain /
takeover / arbiter. These are cross-entity invariants, not distinct types.
10-entity set stands.
Q2 ReplicaReceiver scope: VERIFIED per-volume, not per-assignment.
`v.replRecv = recv` at `blockvol.go:1515` is the only write site; zero
`replRecv = nil` assignments in codebase. Receiver is constructed-once per
BlockVol instance. L1 §2.3 wording stands.
Q3 RebuildSession/Bitmap durability: VERIFIED no sidecar. Grep
`rebuild_bitmap.go` + `rebuild_session.go` for `os.Open / os.Create /
WriteFile / ReadFile / persist / sidecar` → empty. Recovery is WAL
hydration only (`hydrateBitmapFromRecoveredWAL` at `rebuild_session.go:102`).
L1 §2.10 invariant #3 CORRECTED — earlier draft incorrectly called out a
"sidecar schema" that doesn't exist.
## QA concern #3 resolution: §3.14 new hazard
`AllBlocks()` semantic divergence: V3 `walstore.go:565` and
`smartwal/store.go:367` both call `s.Read(lba)` which reads through the
dirty map (includes unflushed WAL bytes). V2 `rebuild.go:handleExtentStream`
uses `readBlockFromExtent` which BYPASSES dirty map (flushed-only).
Concrete impact: V3 base stream can contain bytes the primary hasn't fsynced.
If primary crashes pre-fsync, replica's copy is "newer" than primary's
recovered state. Epoch fencing + WAL-wins bitmap still prevent corruption,
but the invariant chain is "eventually consistent via epoch churn" instead
of V2's "base stream never contains unflushed bytes". Different contracts,
same end state.
Two L2 options proposed: (a) keep AllBlocks semantics + document non-claim
in §2.7 bridge; (b) add `LogicalStorage.AllBlocksFlushed()` preserving V2
invariant. H5 architect-line decision affects which path is safer.
## QA concern #2 resolution: §3.a locked-pairs section (new)
Documents pre-coupled L2 decisions driven by V3 existing shape:
H6 Option C → H7b locks automatically (Provider intercepts at LogicalStorage
layer; Backend.Write stays host-facing, doesn't carry LSN)
§3.14 + H5 → AllBlocks safety rationale depends on which H5 shape wins
Per BUG-005 documentation-discipline lesson: record coupled pairs explicitly
rather than leaving them as "implied". Saves L2 cycles and gives future
readers visible intent for why Backend.Write excludes LSN.
## QA concern #1 deferred to L2
Volumes map extension (single-map with role discrimination vs two separate
primaryHandles + replicaHandles maps) is a legitimate L2 design concern.
L1 appropriately hedges with "likely needs to grow" (§3.11 Option C); L2
picks shape. QA's BUG-005-adjacent concern (role-discriminated handle
callers forgetting to check role) is the right frame for the L2 decision.
No L1 edit needed; flagged for L2 attention.
## §4 open questions status
Q1-Q3 ✓ resolved
Q4 DistGroupCommit residence → effectively answered by §3.11 C
Q5 protocol-frame wire-compat stance → still architect-line (pairs with H5)
Blocking L2 start now: only H5 + Q5, both architect-line. QA to draft
one-page arch memo per round-2 offer.
## Change log
§5 feedback-round log gains round-3 entry
§6 change log gains full round-3 detail with V2 line citations
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§8C.8 specifies exactly one three-sign per T-boundary — at T-start,
covering the bundled L1+L2+L3 package. I had proposed a separate
L1 three-sign in §5 that isn't in the rule. Architect correctly
pushed back.
§5 rewritten as lightweight cadence:
1. sw V3 pre-scan (~5 min, inline reply, prerequisite to L2 not a
sign gate) — same grep checklist retained, same BUG-005 rationale
2. sw + QA iterate on L2 (catalogue §3 filled) informally
3. sw + QA draft L3 (T4 port plan sketch)
4. T4 T-start three-sign on bundled L1+L2+L3 (only governance event)
Informal feedback-round log hook added so architect/PM inputs are
tracked without per-round sign ceremony.
Change log updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All 5 feedback items accepted; no subsetting.
F1 — RebuildBitmap split into standalone §2.10 entity (10 total,
was 9). Rationale: bitmap has independent on-disk schema (~84 LOC
rebuild_bitmap.go) + independent conflict-resolution invariant
(WAL-wins-over-base). Collapsing into §2.6 RebuildSession at L1
would lose granularity for L2 — bitmap and session may have
different PRESERVE/REBUILD verdicts. §2.6 now explicitly
cross-references §2.10.
F2 — ShipperGroup §2.2 gains "External deps" row: N = RF comes
from master assignment via BlockVol.SetReplicaAddrs, not from
shipper-internal decision. Cross-entity contract (master assignment
↔ ShipperGroup size ↔ ReplicaReceiver expected-connection-count
↔ DistGroupCommit quorum arithmetic) made explicit so L2 split
can't silently drift sync_quorum.
F3 — ReplicaBarrier §2.4 scope rewritten from "per-request
ephemeral" to "per-request call-closure, BUT queue-state shared
per-volume via cond.Wait". Prior wording risked 1:1-porting into
a V3 stateless function, losing multi-watcher cond.Broadcast
semantics.
H5 added to §3 observations — cross-node epoch consistency
observation window for sync_quorum. V2 implicit via ack frame
carrying epoch; V3 L2 must pick "ack frame carries epoch" vs
"primary maintains per-replica epoch cache" before locking.
Different choices → different failover + rebuild-trigger semantics.
H6 added to §3 observations — write-path vs replication-path
concurrency residence. Three L2 options documented:
A) StorageBackend.Write triggers shipper (violates T3a layering)
B) ReplicatedBackend wraps StorageBackend+shipper (clean; +1 entity)
C) Replication inside DurableProvider (extends BUG-005 lesson)
L1 makes no recommendation; L2 LOCKS the decision before L3.
§5 restructured into 5 gated steps; step 1 is a mandatory sw V3
pre-scan of core/frontend/durable/ + core/frontend/*.go for
pre-baked replication-adjacent assumptions. Rationale cited per
architect: BUG-005 latent drift came from implicit V3 convention;
L1 must surface any such convention before L2 verdicts lock.
Concrete grep checklist included so the scan is 5 min, not open-ended.
§2 header + §4 open question #1 updated for 10-entity count.
Scope block references rebuild_bitmap.go explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Predecessor docs for the T3 batch, retained here for audit trail.
The closure report (`v3-phase-15-t3-closure-report.md`), contract
bridge catalogue, and BUG-005/006 artifacts already landed in
commits `4127e5136` + `6e196885e`; this commit fills the docs
those closure artifacts reference back to.
Landed:
v3-phase-15-t3-port-plan-sketch.md T3 umbrella sketch (rev-2.1, three-signed)
v3-phase-15-t3-port-audit.md T3.0 port audit + Addendum A (QA-signed)
v3-phase-15-t3a-mini-plan.md T3a scope + sign-off (CLOSED 0e1595c)
v3-phase-15-t3b-mini-plan.md T3b scope + sign-off (CLOSED 72d0d40)
v3-phase-15-t3c-mini-plan.md T3c scope + sign-off (CLOSED 829c6a9)
Total 1,346 lines of doc; no code impact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Covers 6 areas based on CockroachDB/Ceph/etcd/Longhorn research:
1. Structured logging: zap + JSON + channel model (OPS/STORAGE/REPL/ISCSI/AUDIT/HEALTH)
2. Distributed tracing: OpenTelemetry spans across write/rebuild/failover paths
3. Metrics: 40+ must-have Prometheus metrics with histogram latency buckets
4. Debug tools: debug zip (logs+pprof+state), log merge, live tail
5. Audit logging: every admin mutation with actor/target/operation/result
6. Alert design: 3 tiers (page/ticket/log), anti-patterns to avoid
Identifies existing gaps: no I/O latency histogram, no rebuild duration
metric, no audit trail, no structured logging, no distributed tracing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Covers three personas (developer/operator/platform engineer) with:
- One-command setup: weed server -block (10 seconds to first volume)
- Shell commands: block.list, block.status, block.health, block.create, etc.
- REST API: /block/volumes CRUD, /block/health
- Observability: Prometheus metrics, alerting rules, Grafana dashboard
- Actionable error messages (every error tells you what to do next)
- Dry-run by default for all destructive operations
Competitive comparison: 10s setup vs Ceph 30min, 13.5x write IOPS,
single binary for object + block storage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
P1 feature updated: replace generic "structured results" with concrete
runs.db design (newline-delimited JSON, one line per run). Leverages
existing RunBundle system (manifest.json, result.json already exist).
New CLI commands: list, trend, gc, reindex, diff.
Regression detection via stddev comparison against rolling baseline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
testrunner-roadmap.md: P0-P3 feature plan for multi-version comparison,
Ceph adapter, result tracking, cluster templates, debug mode.
dm-stripe-two-server.yaml: proven Linux dm-stripe across 2 sw-block
volumes on 2 servers. Results: single=42K IOPS → striped=79K IOPS (1.87x).
Data integrity verified via md5. Zero sw-block code changes needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The architect's refactor correctly routes remote rebuild acks through
the shared observation path (pins, watchdog, deferred terminal success).
But requireReplicaSession fails with "sender not found" when the
orchestrator registry is reconciled between installSession and the
first ack arrival.
Fix: when emitTerminal=false (remote path), treat sender-not-found as
non-fatal. The remote coordinator already validated the session — the
sender lookup is for local observation only. Pins and watchdog handle
nil snap gracefully (updateRebuildProgressPin line 296 already checks
snap != nil).
This preserves the architect's design (shared observation + deferred
terminal success) while tolerating the sender registry race that only
affects the remote rebuild path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pingqiu
