refactor: preserve bounded needs_rebuild heartbeat truth

Make the heartbeat/master boundary preserve explicit needs_rebuild truth so primary heartbeat consume no longer collapses that stronger mode into a generic degraded signal. Keep backward compatibility by falling back to the previous heuristic when older heartbeats do not send the field.

Made-with: Cursor
This commit is contained in:
pingqiu
2026-04-04 13:11:42 -07:00
parent 69b41a7f16
commit eb610deb92
11 changed files with 340 additions and 6 deletions
+68
View File
@@ -1272,3 +1272,71 @@ Conclusion:
2. backward compatibility is preserved because older heartbeats without the
explicit field still fall back to the previous address-based heuristic
3. this slice still does not claim broad failover or promotion closure by itself
---
#### `16N` Start Note Rev 1
Date: 2026-04-04
Scope: bounded explicit `needs_rebuild` preservation on the heartbeat/master seam
Why this slice exists:
1. `16M` made replica readiness explicit on the heartbeat/master seam
2. but the primary heartbeat still collapses core `ModeNeedsRebuild` and
`ModeDegraded` into the same `replica_degraded` bit
3. that means master-side outward `VolumeMode` can still lose one important
product-facing distinction even when the core already knows the stronger
`needs_rebuild` truth
Chosen implementation rule:
1. widen the heartbeat wire additively with an explicit `needs_rebuild` field
2. emit it from the current bounded core mode on the core-present path
3. make master-side consume prefer explicit `needs_rebuild` truth and retain the
previous heuristic only as backward-compatible fallback
4. do not broaden this slice into full `VolumeMode` heartbeat ownership
---
#### `16N` Delivery Note Rev 1
Date: 2026-04-04
Scope: bounded explicit `needs_rebuild` preservation on the heartbeat/master seam
What changed:
1. `weed/pb/master.proto`
- added additive optional `needs_rebuild` to `BlockVolumeInfoMessage`
2. `weed/pb/master_pb/master.pb.go`
- regenerated so heartbeat wire presence is represented as `*bool`
3. `weed/storage/blockvol/block_heartbeat.go`
- heartbeat wire struct now carries explicit `NeedsRebuild`
4. `weed/storage/blockvol/block_heartbeat_proto.go`
- heartbeat conversion now writes and reads `NeedsRebuild`
5. `weed/server/volume_server_block.go`
- heartbeat emission now preserves explicit bounded `needs_rebuild` truth
from the current core mode
6. `weed/server/master_block_registry.go`
- registry consume now prefers explicit heartbeat `needs_rebuild` truth and
keeps the older heuristic only when the field is absent
7. focused tests in `block_heartbeat_proto_test.go`,
`volume_server_block_test.go`, and `master_block_registry_test.go`
- now prove explicit `needs_rebuild` preservation and backward-compatible
fallback
Proof / evidence:
1. `go test ./weed/storage/blockvol/ -count=1 -run "TestInfoMessage_(Replica|NeedsRebuild)"`
2. `go test ./weed/server/ -count=1 -timeout 120s -run "Test(BlockService_CollectBlockVolumeHeartbeat_PrimaryNeedsRebuildUsesCoreMode|HeartbeatReplicaDegraded_UsesCoreMode|Registry_UpdateFullHeartbeat_(ConsumesExplicitNeedsRebuildFromPrimaryHeartbeat|NeedsRebuildFallsBackWhenFieldAbsent|ExplicitHealthySuppressesStaleNeedsRebuildHeuristic))"`
3. `go test ./weed/server/ -count=1 -timeout 120s -run "Test(BlockService_ApplyAssignments_|P16B_|P4_|Registry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady)"`
4. result: `PASS`
Conclusion:
1. the heartbeat/master seam no longer collapses explicit core
`needs_rebuild` truth into a generic degraded-only signal
2. backward compatibility is preserved because older heartbeats without the
explicit field still fall back to the previous heuristic
3. this slice still does not claim full `VolumeMode` heartbeat ownership or
broad failover closure by itself
+58
View File
@@ -551,6 +551,60 @@ Evidence:
1. focused working-tree change after `16L` closeout
### `16N`: NeedsRebuild Heartbeat Mode Preservation
Goal:
1. close one bounded failover/publication seam by preserving explicit
`needs_rebuild` truth across the primary heartbeat/master consume boundary
instead of collapsing it into a generic degraded bit
2. keep the slice limited to `needs_rebuild` preservation on the heartbeat wire
and master-registry consume path, not broad `VolumeMode` rebinding
Acceptance object:
1. `BlockVolumeInfoMessage` carries an additive explicit `needs_rebuild` bit on
the heartbeat wire
2. `weed/server` heartbeat emission sets that bit from the current core-owned
mode truth on the core-present path
3. `master_block_registry` consumes explicit heartbeat `needs_rebuild` truth
before the older replica-role / degraded-bit heuristic
4. focused proofs show primary `needs_rebuild` survives heartbeat/master consume
even when the old heuristic would only yield `degraded`
5. this slice still does not yet claim broad `VolumeMode` heartbeat ownership or
broad failover closure
Current chosen path:
1. widen `master.proto` / heartbeat conversion with an additive
`needs_rebuild` field
2. emit that field from `CollectBlockVolumeHeartbeat` using the bounded core
mode on the primary path
3. let master consume prefer explicit `needs_rebuild` truth while retaining the
previous heuristic as backward-compatible fallback
Status:
1. delivered
Delivered result:
1. `BlockVolumeInfoMessage` now carries additive explicit `needs_rebuild`
heartbeat truth on the wire
2. `weed/server` heartbeat emission now preserves explicit bounded
`needs_rebuild` truth from the core-owned mode on the current core-present
path
3. `master_block_registry` now prefers explicit heartbeat `needs_rebuild` truth
over the older collapsed degraded-bit / replica-role heuristic while keeping
the previous heuristic as backward-compatible fallback when the field is
absent
4. focused proofs now show primary `needs_rebuild` survives heartbeat/master
consume as `needs_rebuild` rather than collapsing into generic `degraded`
Evidence:
1. focused working-tree change after `16M` closeout
## Current Checkpoint Review Target
The current review target is the current widened bounded runtime checkpoint
@@ -628,6 +682,10 @@ boundary:
- replica heartbeat/master consume now carries explicit bounded
`ReplicaReady` truth with backward-compatible fallback for older
heartbeats
15. `16N` delivered:
- primary heartbeat/master consume now preserves explicit bounded
`needs_rebuild` truth with backward-compatible fallback for older
heartbeats
After this checkpoint:
+1
View File
@@ -513,6 +513,7 @@ message BlockVolumeInfoMessage {
string nvme_addr = 17;
string nqn = 18;
optional bool replica_ready = 19;
optional bool needs_rebuild = 20;
}
message BlockVolumeShortInfoMessage {
+13 -3
View File
@@ -3912,6 +3912,7 @@ type BlockVolumeInfoMessage struct {
NvmeAddr string `protobuf:"bytes,17,opt,name=nvme_addr,json=nvmeAddr,proto3" json:"nvme_addr,omitempty"`
Nqn string `protobuf:"bytes,18,opt,name=nqn,proto3" json:"nqn,omitempty"`
ReplicaReady *bool `protobuf:"varint,19,opt,name=replica_ready,json=replicaReady,proto3,oneof" json:"replica_ready,omitempty"`
NeedsRebuild *bool `protobuf:"varint,20,opt,name=needs_rebuild,json=needsRebuild,proto3,oneof" json:"needs_rebuild,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -4079,6 +4080,13 @@ func (x *BlockVolumeInfoMessage) GetReplicaReady() bool {
return false
}
func (x *BlockVolumeInfoMessage) GetNeedsRebuild() bool {
if x != nil && x.NeedsRebuild != nil {
return *x.NeedsRebuild
}
return false
}
type BlockVolumeShortInfoMessage struct {
state protoimpl.MessageState `protogen:"open.v1"`
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
@@ -5967,7 +5975,7 @@ const file_weed_pb_master_proto_rawDesc = "" +
"\x0fprevious_leader\x18\x01 \x01(\tR\x0epreviousLeader\x12\x1d\n" +
"\n" +
"new_leader\x18\x02 \x01(\tR\tnewLeader\"\x14\n" +
"\x12VolumeGrowResponse\"\x9e\x05\n" +
"\x12VolumeGrowResponse\"\xda\x05\n" +
"\x16BlockVolumeInfoMessage\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1f\n" +
"\vvolume_size\x18\x02 \x01(\x04R\n" +
@@ -5991,8 +5999,10 @@ const file_weed_pb_master_proto_rawDesc = "" +
"\x0fdurability_mode\x18\x10 \x01(\tR\x0edurabilityMode\x12\x1b\n" +
"\tnvme_addr\x18\x11 \x01(\tR\bnvmeAddr\x12\x10\n" +
"\x03nqn\x18\x12 \x01(\tR\x03nqn\x12(\n" +
"\rreplica_ready\x18\x13 \x01(\bH\x00R\freplicaReady\x88\x01\x01B\x10\n" +
"\x0e_replica_ready\"\x8e\x01\n" +
"\rreplica_ready\x18\x13 \x01(\bH\x00R\freplicaReady\x88\x01\x01\x12(\n" +
"\rneeds_rebuild\x18\x14 \x01(\bH\x01R\fneedsRebuild\x88\x01\x01B\x10\n" +
"\x0e_replica_readyB\x10\n" +
"\x0e_needs_rebuild\"\x8e\x01\n" +
"\x1bBlockVolumeShortInfoMessage\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1f\n" +
"\vvolume_size\x18\x02 \x01(\x04R\n" +
+24 -3
View File
@@ -77,6 +77,8 @@ type BlockVolumeEntry struct {
ReplicaReady bool // all configured replicas are ready for publication
ReplicaDegraded bool // aggregate: transport degraded OR not ready
TransportDegraded bool // primary reports degraded replicas
NeedsRebuild bool // explicit primary heartbeat needs_rebuild truth when present
HasNeedsRebuild bool // whether the current primary heartbeat carried explicit needs_rebuild truth
// CP13-9: Normalized volume mode for external surfaces.
// Computed by recomputeReplicaState from the current entry state.
@@ -178,11 +180,19 @@ func (e *BlockVolumeEntry) computeVolumeMode() string {
return "bootstrap_pending"
}
// Check for NeedsRebuild state on any replica.
for _, ri := range e.Replicas {
if blockvol.RoleFromWire(ri.Role) == blockvol.RoleRebuilding {
// Prefer explicit primary heartbeat needs_rebuild truth when present.
if e.HasNeedsRebuild {
if e.NeedsRebuild {
return "needs_rebuild"
}
} else {
// Backward-compatible fallback: older paths may only surface needs_rebuild
// through replica-side role heuristics.
for _, ri := range e.Replicas {
if blockvol.RoleFromWire(ri.Role) == blockvol.RoleRebuilding {
return "needs_rebuild"
}
}
}
// Replicas exist but not all ready.
@@ -633,6 +643,7 @@ func (r *BlockVolumeRegistry) applyPrimaryHeartbeatObservation(existing *BlockVo
existing.LastLeaseGrant = time.Now()
existing.HealthScore = info.HealthScore
existing.TransportDegraded = info.ReplicaDegraded
existing.NeedsRebuild, existing.HasNeedsRebuild = primaryNeedsRebuildObservedFromHeartbeat(info)
existing.WALHeadLSN = info.WalHeadLsn
// F3: only update DurabilityMode when non-empty (prevents older VS from clearing strict mode).
if info.DurabilityMode != "" {
@@ -720,6 +731,16 @@ func replicaReadyObservedFromHeartbeat(info *master_pb.BlockVolumeInfoMessage) b
return info.ReplicaDataAddr != "" && info.ReplicaCtrlAddr != ""
}
func primaryNeedsRebuildObservedFromHeartbeat(info *master_pb.BlockVolumeInfoMessage) (bool, bool) {
if info == nil {
return false, false
}
if info.NeedsRebuild != nil {
return info.GetNeedsRebuild(), true
}
return false, false
}
// reconcileOnRestart handles the case where a second server reports a volume
// name that already exists in the registry during master restart reconstruction.
// Uses epoch-based tie-breaking to determine who is the real primary.
+105
View File
@@ -2179,3 +2179,108 @@ func TestRegistry_UpdateFullHeartbeat_ReplicaReadyFallsBackToAddressesWhenFieldA
t.Fatalf("expected aggregate replica ready from fallback consume, entry=%+v", entry)
}
}
func TestRegistry_UpdateFullHeartbeat_ConsumesExplicitNeedsRebuildFromPrimaryHeartbeat(t *testing.T) {
r := NewBlockVolumeRegistry()
if err := r.Register(&BlockVolumeEntry{
Name: "vol-master-needs-rebuild",
VolumeServer: "primary-server:8080",
Path: "/blocks/vol-master-needs-rebuild-primary.blk",
Status: StatusActive,
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaFactor: 2,
Replicas: []ReplicaInfo{{
Server: "replica-server:8080",
Path: "/blocks/vol-master-needs-rebuild-replica.blk",
Ready: true,
}},
}); err != nil {
t.Fatalf("register: %v", err)
}
needsRebuild := true
r.UpdateFullHeartbeat("primary-server:8080", []*master_pb.BlockVolumeInfoMessage{{
Path: "/blocks/vol-master-needs-rebuild-primary.blk",
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaDegraded: true,
NeedsRebuild: &needsRebuild,
}}, "")
entry, _ := r.Lookup("vol-master-needs-rebuild")
if !entry.NeedsRebuild || !entry.HasNeedsRebuild {
t.Fatalf("expected explicit needs_rebuild truth on entry, entry=%+v", entry)
}
if entry.VolumeMode != "needs_rebuild" {
t.Fatalf("expected needs_rebuild from explicit primary heartbeat truth, got %q", entry.VolumeMode)
}
}
func TestRegistry_UpdateFullHeartbeat_NeedsRebuildFallsBackWhenFieldAbsent(t *testing.T) {
r := NewBlockVolumeRegistry()
if err := r.Register(&BlockVolumeEntry{
Name: "vol-master-needs-rebuild-fallback",
VolumeServer: "primary-server:8080",
Path: "/blocks/vol-master-needs-rebuild-fallback-primary.blk",
Status: StatusActive,
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaFactor: 2,
Replicas: []ReplicaInfo{{
Server: "replica-server:8080",
Path: "/blocks/vol-master-needs-rebuild-fallback-replica.blk",
Ready: true,
Role: blockvol.RoleToWire(blockvol.RoleRebuilding),
}},
}); err != nil {
t.Fatalf("register: %v", err)
}
r.UpdateFullHeartbeat("primary-server:8080", []*master_pb.BlockVolumeInfoMessage{{
Path: "/blocks/vol-master-needs-rebuild-fallback-primary.blk",
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaDegraded: true,
}}, "")
entry, _ := r.Lookup("vol-master-needs-rebuild-fallback")
if entry.HasNeedsRebuild {
t.Fatalf("did not expect explicit needs_rebuild truth when field absent, entry=%+v", entry)
}
if entry.VolumeMode != "needs_rebuild" {
t.Fatalf("expected fallback needs_rebuild from replica role heuristic, got %q", entry.VolumeMode)
}
}
func TestRegistry_UpdateFullHeartbeat_ExplicitHealthySuppressesStaleNeedsRebuildHeuristic(t *testing.T) {
r := NewBlockVolumeRegistry()
if err := r.Register(&BlockVolumeEntry{
Name: "vol-master-needs-rebuild-explicit-false",
VolumeServer: "primary-server:8080",
Path: "/blocks/vol-master-needs-rebuild-explicit-false-primary.blk",
Status: StatusActive,
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaFactor: 2,
Replicas: []ReplicaInfo{{
Server: "replica-server:8080",
Path: "/blocks/vol-master-needs-rebuild-explicit-false-replica.blk",
Ready: true,
Role: blockvol.RoleToWire(blockvol.RoleRebuilding),
}},
}); err != nil {
t.Fatalf("register: %v", err)
}
needsRebuild := false
r.UpdateFullHeartbeat("primary-server:8080", []*master_pb.BlockVolumeInfoMessage{{
Path: "/blocks/vol-master-needs-rebuild-explicit-false-primary.blk",
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaDegraded: true,
NeedsRebuild: &needsRebuild,
}}, "")
entry, _ := r.Lookup("vol-master-needs-rebuild-explicit-false")
if !entry.HasNeedsRebuild || entry.NeedsRebuild {
t.Fatalf("expected explicit false needs_rebuild truth on entry, entry=%+v", entry)
}
if entry.VolumeMode != "degraded" {
t.Fatalf("expected explicit false to suppress stale needs_rebuild heuristic, got %q", entry.VolumeMode)
}
}
+15
View File
@@ -966,6 +966,7 @@ func (bs *BlockService) CollectBlockVolumeHeartbeat() []blockvol.BlockVolumeInfo
if s, ok := bs.replStates[msgs[i].Path]; ok {
msgs[i].ReplicaDataAddr, msgs[i].ReplicaCtrlAddr = bs.heartbeatReplicaAddrs(msgs[i].Path, s)
msgs[i].ReplicaReady = bs.heartbeatReplicaReady(msgs[i].Path, s)
msgs[i].NeedsRebuild = bs.heartbeatNeedsRebuild(msgs[i].Path, s)
}
msgs[i].ReplicaDegraded = bs.heartbeatReplicaDegraded(msgs[i].Path, msgs[i].ReplicaDegraded)
// NVMe publication: report nvme_addr and nqn if NVMe target is running.
@@ -1021,6 +1022,20 @@ func (bs *BlockService) heartbeatReplicaReady(path string, state *volReplState)
return state.publishHealthy
}
// heartbeatNeedsRebuild returns the explicit needs_rebuild truth that should be
// exposed on the current heartbeat surface. On the core-present path it
// preserves the stronger core mode instead of collapsing it into the degraded
// bit. Older paths keep returning false and rely on previous heuristics.
func (bs *BlockService) heartbeatNeedsRebuild(path string, state *volReplState) bool {
if state == nil {
return false
}
if proj, ok := bs.CoreProjection(path); ok {
return proj.Mode.Name == engine.ModeNeedsRebuild
}
return false
}
// heartbeatReplicaDegraded returns the bounded degraded bit for the current
// heartbeat surface. On the Phase 15 live path it prefers the core mode when
// present, then falls back to the runtime-local status bit.
+33
View File
@@ -1280,6 +1280,39 @@ func TestBlockService_HeartbeatReplicaDegraded_UsesCoreMode(t *testing.T) {
}
}
func TestBlockService_CollectBlockVolumeHeartbeat_PrimaryNeedsRebuildUsesCoreMode(t *testing.T) {
bs := newTestBlockServiceDirect(t)
path := createTestVolDirect(t, bs, "vol-heartbeat-needs-rebuild-primary")
errs := bs.ApplyAssignments([]blockvol.BlockVolumeAssignment{
{
Path: path,
Epoch: 1,
Role: blockvol.RoleToWire(blockvol.RolePrimary),
LeaseTtlMs: 30000,
ReplicaServerID: "vs-2",
ReplicaDataAddr: "10.0.0.2:4260",
ReplicaCtrlAddr: "10.0.0.2:4261",
},
})
if len(errs) != 1 || errs[0] != nil {
t.Fatalf("apply assignment errs=%v", errs)
}
bs.applyCoreEvent(engine.NeedsRebuildObserved{ID: path, ReplicaID: path + "/vs-2", Reason: "gap_too_large"})
msg := findHeartbeatMsg(bs.CollectBlockVolumeHeartbeat(), path)
if msg == nil {
t.Fatal("volume missing from heartbeat")
}
if !msg.NeedsRebuild {
t.Fatalf("expected explicit needs_rebuild truth on heartbeat, msg=%+v", msg)
}
if !msg.ReplicaDegraded {
t.Fatalf("expected degraded bit to remain true on needs_rebuild path, msg=%+v", msg)
}
}
func TestBlockService_HeartbeatIncludesReplicaAddrs(t *testing.T) {
bs := newTestBlockServiceDirect(t)
path := createTestVolDirect(t, bs, "vol1")
+1
View File
@@ -20,6 +20,7 @@ type BlockVolumeInfoMessage struct {
ReplicaDataAddr string // receiver data listen addr (VS reports in heartbeat)
ReplicaCtrlAddr string // receiver ctrl listen addr
ReplicaReady bool // explicit replica readiness truth on the heartbeat seam
NeedsRebuild bool // explicit needs_rebuild truth on the heartbeat seam
HealthScore float64 // CP8-2: 0.0-1.0
ScrubErrors int64 // CP8-2: lifetime scrub error count
LastScrubTime int64 // CP8-2: unix seconds
@@ -19,6 +19,7 @@ func InfoMessageToProto(m BlockVolumeInfoMessage) *master_pb.BlockVolumeInfoMess
ReplicaDataAddr: m.ReplicaDataAddr,
ReplicaCtrlAddr: m.ReplicaCtrlAddr,
ReplicaReady: &m.ReplicaReady,
NeedsRebuild: &m.NeedsRebuild,
HealthScore: m.HealthScore,
ScrubErrors: m.ScrubErrors,
LastScrubTime: m.LastScrubTime,
@@ -47,6 +48,7 @@ func InfoMessageFromProto(p *master_pb.BlockVolumeInfoMessage) BlockVolumeInfoMe
ReplicaDataAddr: p.ReplicaDataAddr,
ReplicaCtrlAddr: p.ReplicaCtrlAddr,
ReplicaReady: p.GetReplicaReady(),
NeedsRebuild: p.GetNeedsRebuild(),
HealthScore: p.HealthScore,
ScrubErrors: p.ScrubErrors,
LastScrubTime: p.LastScrubTime,
@@ -267,6 +267,26 @@ func TestInfoMessage_ReplicaReadyRoundTrip(t *testing.T) {
}
}
func TestInfoMessage_NeedsRebuildRoundTrip(t *testing.T) {
orig := BlockVolumeInfoMessage{
Path: "/data/vol.blk",
Epoch: 1,
NeedsRebuild: true,
}
pb := InfoMessageToProto(orig)
back := InfoMessageFromProto(pb)
if !back.NeedsRebuild {
t.Fatal("NeedsRebuild should be true after round-trip")
}
orig.NeedsRebuild = false
pb = InfoMessageToProto(orig)
back = InfoMessageFromProto(pb)
if back.NeedsRebuild {
t.Fatal("NeedsRebuild should be false after round-trip")
}
}
func TestAssignment_MultiReplicaRoundTrip(t *testing.T) {
orig := BlockVolumeAssignment{
Path: "/data/vol.blk",