mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-21 22:56:55 +00:00
refactor: preserve bounded needs_rebuild heartbeat truth
Make the heartbeat/master boundary preserve explicit needs_rebuild truth so primary heartbeat consume no longer collapses that stronger mode into a generic degraded signal. Keep backward compatibility by falling back to the previous heuristic when older heartbeats do not send the field. Made-with: Cursor
This commit is contained in:
@@ -1272,3 +1272,71 @@ Conclusion:
|
||||
2. backward compatibility is preserved because older heartbeats without the
|
||||
explicit field still fall back to the previous address-based heuristic
|
||||
3. this slice still does not claim broad failover or promotion closure by itself
|
||||
|
||||
---
|
||||
|
||||
#### `16N` Start Note Rev 1
|
||||
|
||||
Date: 2026-04-04
|
||||
Scope: bounded explicit `needs_rebuild` preservation on the heartbeat/master seam
|
||||
|
||||
Why this slice exists:
|
||||
|
||||
1. `16M` made replica readiness explicit on the heartbeat/master seam
|
||||
2. but the primary heartbeat still collapses core `ModeNeedsRebuild` and
|
||||
`ModeDegraded` into the same `replica_degraded` bit
|
||||
3. that means master-side outward `VolumeMode` can still lose one important
|
||||
product-facing distinction even when the core already knows the stronger
|
||||
`needs_rebuild` truth
|
||||
|
||||
Chosen implementation rule:
|
||||
|
||||
1. widen the heartbeat wire additively with an explicit `needs_rebuild` field
|
||||
2. emit it from the current bounded core mode on the core-present path
|
||||
3. make master-side consume prefer explicit `needs_rebuild` truth and retain the
|
||||
previous heuristic only as backward-compatible fallback
|
||||
4. do not broaden this slice into full `VolumeMode` heartbeat ownership
|
||||
|
||||
---
|
||||
|
||||
#### `16N` Delivery Note Rev 1
|
||||
|
||||
Date: 2026-04-04
|
||||
Scope: bounded explicit `needs_rebuild` preservation on the heartbeat/master seam
|
||||
|
||||
What changed:
|
||||
|
||||
1. `weed/pb/master.proto`
|
||||
- added additive optional `needs_rebuild` to `BlockVolumeInfoMessage`
|
||||
2. `weed/pb/master_pb/master.pb.go`
|
||||
- regenerated so heartbeat wire presence is represented as `*bool`
|
||||
3. `weed/storage/blockvol/block_heartbeat.go`
|
||||
- heartbeat wire struct now carries explicit `NeedsRebuild`
|
||||
4. `weed/storage/blockvol/block_heartbeat_proto.go`
|
||||
- heartbeat conversion now writes and reads `NeedsRebuild`
|
||||
5. `weed/server/volume_server_block.go`
|
||||
- heartbeat emission now preserves explicit bounded `needs_rebuild` truth
|
||||
from the current core mode
|
||||
6. `weed/server/master_block_registry.go`
|
||||
- registry consume now prefers explicit heartbeat `needs_rebuild` truth and
|
||||
keeps the older heuristic only when the field is absent
|
||||
7. focused tests in `block_heartbeat_proto_test.go`,
|
||||
`volume_server_block_test.go`, and `master_block_registry_test.go`
|
||||
- now prove explicit `needs_rebuild` preservation and backward-compatible
|
||||
fallback
|
||||
|
||||
Proof / evidence:
|
||||
|
||||
1. `go test ./weed/storage/blockvol/ -count=1 -run "TestInfoMessage_(Replica|NeedsRebuild)"`
|
||||
2. `go test ./weed/server/ -count=1 -timeout 120s -run "Test(BlockService_CollectBlockVolumeHeartbeat_PrimaryNeedsRebuildUsesCoreMode|HeartbeatReplicaDegraded_UsesCoreMode|Registry_UpdateFullHeartbeat_(ConsumesExplicitNeedsRebuildFromPrimaryHeartbeat|NeedsRebuildFallsBackWhenFieldAbsent|ExplicitHealthySuppressesStaleNeedsRebuildHeuristic))"`
|
||||
3. `go test ./weed/server/ -count=1 -timeout 120s -run "Test(BlockService_ApplyAssignments_|P16B_|P4_|Registry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady)"`
|
||||
4. result: `PASS`
|
||||
|
||||
Conclusion:
|
||||
|
||||
1. the heartbeat/master seam no longer collapses explicit core
|
||||
`needs_rebuild` truth into a generic degraded-only signal
|
||||
2. backward compatibility is preserved because older heartbeats without the
|
||||
explicit field still fall back to the previous heuristic
|
||||
3. this slice still does not claim full `VolumeMode` heartbeat ownership or
|
||||
broad failover closure by itself
|
||||
|
||||
@@ -551,6 +551,60 @@ Evidence:
|
||||
|
||||
1. focused working-tree change after `16L` closeout
|
||||
|
||||
### `16N`: NeedsRebuild Heartbeat Mode Preservation
|
||||
|
||||
Goal:
|
||||
|
||||
1. close one bounded failover/publication seam by preserving explicit
|
||||
`needs_rebuild` truth across the primary heartbeat/master consume boundary
|
||||
instead of collapsing it into a generic degraded bit
|
||||
2. keep the slice limited to `needs_rebuild` preservation on the heartbeat wire
|
||||
and master-registry consume path, not broad `VolumeMode` rebinding
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `BlockVolumeInfoMessage` carries an additive explicit `needs_rebuild` bit on
|
||||
the heartbeat wire
|
||||
2. `weed/server` heartbeat emission sets that bit from the current core-owned
|
||||
mode truth on the core-present path
|
||||
3. `master_block_registry` consumes explicit heartbeat `needs_rebuild` truth
|
||||
before the older replica-role / degraded-bit heuristic
|
||||
4. focused proofs show primary `needs_rebuild` survives heartbeat/master consume
|
||||
even when the old heuristic would only yield `degraded`
|
||||
5. this slice still does not yet claim broad `VolumeMode` heartbeat ownership or
|
||||
broad failover closure
|
||||
|
||||
Current chosen path:
|
||||
|
||||
1. widen `master.proto` / heartbeat conversion with an additive
|
||||
`needs_rebuild` field
|
||||
2. emit that field from `CollectBlockVolumeHeartbeat` using the bounded core
|
||||
mode on the primary path
|
||||
3. let master consume prefer explicit `needs_rebuild` truth while retaining the
|
||||
previous heuristic as backward-compatible fallback
|
||||
|
||||
Status:
|
||||
|
||||
1. delivered
|
||||
|
||||
Delivered result:
|
||||
|
||||
1. `BlockVolumeInfoMessage` now carries additive explicit `needs_rebuild`
|
||||
heartbeat truth on the wire
|
||||
2. `weed/server` heartbeat emission now preserves explicit bounded
|
||||
`needs_rebuild` truth from the core-owned mode on the current core-present
|
||||
path
|
||||
3. `master_block_registry` now prefers explicit heartbeat `needs_rebuild` truth
|
||||
over the older collapsed degraded-bit / replica-role heuristic while keeping
|
||||
the previous heuristic as backward-compatible fallback when the field is
|
||||
absent
|
||||
4. focused proofs now show primary `needs_rebuild` survives heartbeat/master
|
||||
consume as `needs_rebuild` rather than collapsing into generic `degraded`
|
||||
|
||||
Evidence:
|
||||
|
||||
1. focused working-tree change after `16M` closeout
|
||||
|
||||
## Current Checkpoint Review Target
|
||||
|
||||
The current review target is the current widened bounded runtime checkpoint
|
||||
@@ -628,6 +682,10 @@ boundary:
|
||||
- replica heartbeat/master consume now carries explicit bounded
|
||||
`ReplicaReady` truth with backward-compatible fallback for older
|
||||
heartbeats
|
||||
15. `16N` delivered:
|
||||
- primary heartbeat/master consume now preserves explicit bounded
|
||||
`needs_rebuild` truth with backward-compatible fallback for older
|
||||
heartbeats
|
||||
|
||||
After this checkpoint:
|
||||
|
||||
|
||||
@@ -513,6 +513,7 @@ message BlockVolumeInfoMessage {
|
||||
string nvme_addr = 17;
|
||||
string nqn = 18;
|
||||
optional bool replica_ready = 19;
|
||||
optional bool needs_rebuild = 20;
|
||||
}
|
||||
|
||||
message BlockVolumeShortInfoMessage {
|
||||
|
||||
@@ -3912,6 +3912,7 @@ type BlockVolumeInfoMessage struct {
|
||||
NvmeAddr string `protobuf:"bytes,17,opt,name=nvme_addr,json=nvmeAddr,proto3" json:"nvme_addr,omitempty"`
|
||||
Nqn string `protobuf:"bytes,18,opt,name=nqn,proto3" json:"nqn,omitempty"`
|
||||
ReplicaReady *bool `protobuf:"varint,19,opt,name=replica_ready,json=replicaReady,proto3,oneof" json:"replica_ready,omitempty"`
|
||||
NeedsRebuild *bool `protobuf:"varint,20,opt,name=needs_rebuild,json=needsRebuild,proto3,oneof" json:"needs_rebuild,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -4079,6 +4080,13 @@ func (x *BlockVolumeInfoMessage) GetReplicaReady() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *BlockVolumeInfoMessage) GetNeedsRebuild() bool {
|
||||
if x != nil && x.NeedsRebuild != nil {
|
||||
return *x.NeedsRebuild
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type BlockVolumeShortInfoMessage struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
|
||||
@@ -5967,7 +5975,7 @@ const file_weed_pb_master_proto_rawDesc = "" +
|
||||
"\x0fprevious_leader\x18\x01 \x01(\tR\x0epreviousLeader\x12\x1d\n" +
|
||||
"\n" +
|
||||
"new_leader\x18\x02 \x01(\tR\tnewLeader\"\x14\n" +
|
||||
"\x12VolumeGrowResponse\"\x9e\x05\n" +
|
||||
"\x12VolumeGrowResponse\"\xda\x05\n" +
|
||||
"\x16BlockVolumeInfoMessage\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1f\n" +
|
||||
"\vvolume_size\x18\x02 \x01(\x04R\n" +
|
||||
@@ -5991,8 +5999,10 @@ const file_weed_pb_master_proto_rawDesc = "" +
|
||||
"\x0fdurability_mode\x18\x10 \x01(\tR\x0edurabilityMode\x12\x1b\n" +
|
||||
"\tnvme_addr\x18\x11 \x01(\tR\bnvmeAddr\x12\x10\n" +
|
||||
"\x03nqn\x18\x12 \x01(\tR\x03nqn\x12(\n" +
|
||||
"\rreplica_ready\x18\x13 \x01(\bH\x00R\freplicaReady\x88\x01\x01B\x10\n" +
|
||||
"\x0e_replica_ready\"\x8e\x01\n" +
|
||||
"\rreplica_ready\x18\x13 \x01(\bH\x00R\freplicaReady\x88\x01\x01\x12(\n" +
|
||||
"\rneeds_rebuild\x18\x14 \x01(\bH\x01R\fneedsRebuild\x88\x01\x01B\x10\n" +
|
||||
"\x0e_replica_readyB\x10\n" +
|
||||
"\x0e_needs_rebuild\"\x8e\x01\n" +
|
||||
"\x1bBlockVolumeShortInfoMessage\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1f\n" +
|
||||
"\vvolume_size\x18\x02 \x01(\x04R\n" +
|
||||
|
||||
@@ -77,6 +77,8 @@ type BlockVolumeEntry struct {
|
||||
ReplicaReady bool // all configured replicas are ready for publication
|
||||
ReplicaDegraded bool // aggregate: transport degraded OR not ready
|
||||
TransportDegraded bool // primary reports degraded replicas
|
||||
NeedsRebuild bool // explicit primary heartbeat needs_rebuild truth when present
|
||||
HasNeedsRebuild bool // whether the current primary heartbeat carried explicit needs_rebuild truth
|
||||
|
||||
// CP13-9: Normalized volume mode for external surfaces.
|
||||
// Computed by recomputeReplicaState from the current entry state.
|
||||
@@ -178,11 +180,19 @@ func (e *BlockVolumeEntry) computeVolumeMode() string {
|
||||
return "bootstrap_pending"
|
||||
}
|
||||
|
||||
// Check for NeedsRebuild state on any replica.
|
||||
for _, ri := range e.Replicas {
|
||||
if blockvol.RoleFromWire(ri.Role) == blockvol.RoleRebuilding {
|
||||
// Prefer explicit primary heartbeat needs_rebuild truth when present.
|
||||
if e.HasNeedsRebuild {
|
||||
if e.NeedsRebuild {
|
||||
return "needs_rebuild"
|
||||
}
|
||||
} else {
|
||||
// Backward-compatible fallback: older paths may only surface needs_rebuild
|
||||
// through replica-side role heuristics.
|
||||
for _, ri := range e.Replicas {
|
||||
if blockvol.RoleFromWire(ri.Role) == blockvol.RoleRebuilding {
|
||||
return "needs_rebuild"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Replicas exist but not all ready.
|
||||
@@ -633,6 +643,7 @@ func (r *BlockVolumeRegistry) applyPrimaryHeartbeatObservation(existing *BlockVo
|
||||
existing.LastLeaseGrant = time.Now()
|
||||
existing.HealthScore = info.HealthScore
|
||||
existing.TransportDegraded = info.ReplicaDegraded
|
||||
existing.NeedsRebuild, existing.HasNeedsRebuild = primaryNeedsRebuildObservedFromHeartbeat(info)
|
||||
existing.WALHeadLSN = info.WalHeadLsn
|
||||
// F3: only update DurabilityMode when non-empty (prevents older VS from clearing strict mode).
|
||||
if info.DurabilityMode != "" {
|
||||
@@ -720,6 +731,16 @@ func replicaReadyObservedFromHeartbeat(info *master_pb.BlockVolumeInfoMessage) b
|
||||
return info.ReplicaDataAddr != "" && info.ReplicaCtrlAddr != ""
|
||||
}
|
||||
|
||||
func primaryNeedsRebuildObservedFromHeartbeat(info *master_pb.BlockVolumeInfoMessage) (bool, bool) {
|
||||
if info == nil {
|
||||
return false, false
|
||||
}
|
||||
if info.NeedsRebuild != nil {
|
||||
return info.GetNeedsRebuild(), true
|
||||
}
|
||||
return false, false
|
||||
}
|
||||
|
||||
// reconcileOnRestart handles the case where a second server reports a volume
|
||||
// name that already exists in the registry during master restart reconstruction.
|
||||
// Uses epoch-based tie-breaking to determine who is the real primary.
|
||||
|
||||
@@ -2179,3 +2179,108 @@ func TestRegistry_UpdateFullHeartbeat_ReplicaReadyFallsBackToAddressesWhenFieldA
|
||||
t.Fatalf("expected aggregate replica ready from fallback consume, entry=%+v", entry)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistry_UpdateFullHeartbeat_ConsumesExplicitNeedsRebuildFromPrimaryHeartbeat(t *testing.T) {
|
||||
r := NewBlockVolumeRegistry()
|
||||
if err := r.Register(&BlockVolumeEntry{
|
||||
Name: "vol-master-needs-rebuild",
|
||||
VolumeServer: "primary-server:8080",
|
||||
Path: "/blocks/vol-master-needs-rebuild-primary.blk",
|
||||
Status: StatusActive,
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaFactor: 2,
|
||||
Replicas: []ReplicaInfo{{
|
||||
Server: "replica-server:8080",
|
||||
Path: "/blocks/vol-master-needs-rebuild-replica.blk",
|
||||
Ready: true,
|
||||
}},
|
||||
}); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
|
||||
needsRebuild := true
|
||||
r.UpdateFullHeartbeat("primary-server:8080", []*master_pb.BlockVolumeInfoMessage{{
|
||||
Path: "/blocks/vol-master-needs-rebuild-primary.blk",
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaDegraded: true,
|
||||
NeedsRebuild: &needsRebuild,
|
||||
}}, "")
|
||||
|
||||
entry, _ := r.Lookup("vol-master-needs-rebuild")
|
||||
if !entry.NeedsRebuild || !entry.HasNeedsRebuild {
|
||||
t.Fatalf("expected explicit needs_rebuild truth on entry, entry=%+v", entry)
|
||||
}
|
||||
if entry.VolumeMode != "needs_rebuild" {
|
||||
t.Fatalf("expected needs_rebuild from explicit primary heartbeat truth, got %q", entry.VolumeMode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistry_UpdateFullHeartbeat_NeedsRebuildFallsBackWhenFieldAbsent(t *testing.T) {
|
||||
r := NewBlockVolumeRegistry()
|
||||
if err := r.Register(&BlockVolumeEntry{
|
||||
Name: "vol-master-needs-rebuild-fallback",
|
||||
VolumeServer: "primary-server:8080",
|
||||
Path: "/blocks/vol-master-needs-rebuild-fallback-primary.blk",
|
||||
Status: StatusActive,
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaFactor: 2,
|
||||
Replicas: []ReplicaInfo{{
|
||||
Server: "replica-server:8080",
|
||||
Path: "/blocks/vol-master-needs-rebuild-fallback-replica.blk",
|
||||
Ready: true,
|
||||
Role: blockvol.RoleToWire(blockvol.RoleRebuilding),
|
||||
}},
|
||||
}); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
|
||||
r.UpdateFullHeartbeat("primary-server:8080", []*master_pb.BlockVolumeInfoMessage{{
|
||||
Path: "/blocks/vol-master-needs-rebuild-fallback-primary.blk",
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaDegraded: true,
|
||||
}}, "")
|
||||
|
||||
entry, _ := r.Lookup("vol-master-needs-rebuild-fallback")
|
||||
if entry.HasNeedsRebuild {
|
||||
t.Fatalf("did not expect explicit needs_rebuild truth when field absent, entry=%+v", entry)
|
||||
}
|
||||
if entry.VolumeMode != "needs_rebuild" {
|
||||
t.Fatalf("expected fallback needs_rebuild from replica role heuristic, got %q", entry.VolumeMode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistry_UpdateFullHeartbeat_ExplicitHealthySuppressesStaleNeedsRebuildHeuristic(t *testing.T) {
|
||||
r := NewBlockVolumeRegistry()
|
||||
if err := r.Register(&BlockVolumeEntry{
|
||||
Name: "vol-master-needs-rebuild-explicit-false",
|
||||
VolumeServer: "primary-server:8080",
|
||||
Path: "/blocks/vol-master-needs-rebuild-explicit-false-primary.blk",
|
||||
Status: StatusActive,
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaFactor: 2,
|
||||
Replicas: []ReplicaInfo{{
|
||||
Server: "replica-server:8080",
|
||||
Path: "/blocks/vol-master-needs-rebuild-explicit-false-replica.blk",
|
||||
Ready: true,
|
||||
Role: blockvol.RoleToWire(blockvol.RoleRebuilding),
|
||||
}},
|
||||
}); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
|
||||
needsRebuild := false
|
||||
r.UpdateFullHeartbeat("primary-server:8080", []*master_pb.BlockVolumeInfoMessage{{
|
||||
Path: "/blocks/vol-master-needs-rebuild-explicit-false-primary.blk",
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaDegraded: true,
|
||||
NeedsRebuild: &needsRebuild,
|
||||
}}, "")
|
||||
|
||||
entry, _ := r.Lookup("vol-master-needs-rebuild-explicit-false")
|
||||
if !entry.HasNeedsRebuild || entry.NeedsRebuild {
|
||||
t.Fatalf("expected explicit false needs_rebuild truth on entry, entry=%+v", entry)
|
||||
}
|
||||
if entry.VolumeMode != "degraded" {
|
||||
t.Fatalf("expected explicit false to suppress stale needs_rebuild heuristic, got %q", entry.VolumeMode)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -966,6 +966,7 @@ func (bs *BlockService) CollectBlockVolumeHeartbeat() []blockvol.BlockVolumeInfo
|
||||
if s, ok := bs.replStates[msgs[i].Path]; ok {
|
||||
msgs[i].ReplicaDataAddr, msgs[i].ReplicaCtrlAddr = bs.heartbeatReplicaAddrs(msgs[i].Path, s)
|
||||
msgs[i].ReplicaReady = bs.heartbeatReplicaReady(msgs[i].Path, s)
|
||||
msgs[i].NeedsRebuild = bs.heartbeatNeedsRebuild(msgs[i].Path, s)
|
||||
}
|
||||
msgs[i].ReplicaDegraded = bs.heartbeatReplicaDegraded(msgs[i].Path, msgs[i].ReplicaDegraded)
|
||||
// NVMe publication: report nvme_addr and nqn if NVMe target is running.
|
||||
@@ -1021,6 +1022,20 @@ func (bs *BlockService) heartbeatReplicaReady(path string, state *volReplState)
|
||||
return state.publishHealthy
|
||||
}
|
||||
|
||||
// heartbeatNeedsRebuild returns the explicit needs_rebuild truth that should be
|
||||
// exposed on the current heartbeat surface. On the core-present path it
|
||||
// preserves the stronger core mode instead of collapsing it into the degraded
|
||||
// bit. Older paths keep returning false and rely on previous heuristics.
|
||||
func (bs *BlockService) heartbeatNeedsRebuild(path string, state *volReplState) bool {
|
||||
if state == nil {
|
||||
return false
|
||||
}
|
||||
if proj, ok := bs.CoreProjection(path); ok {
|
||||
return proj.Mode.Name == engine.ModeNeedsRebuild
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// heartbeatReplicaDegraded returns the bounded degraded bit for the current
|
||||
// heartbeat surface. On the Phase 15 live path it prefers the core mode when
|
||||
// present, then falls back to the runtime-local status bit.
|
||||
|
||||
@@ -1280,6 +1280,39 @@ func TestBlockService_HeartbeatReplicaDegraded_UsesCoreMode(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockService_CollectBlockVolumeHeartbeat_PrimaryNeedsRebuildUsesCoreMode(t *testing.T) {
|
||||
bs := newTestBlockServiceDirect(t)
|
||||
path := createTestVolDirect(t, bs, "vol-heartbeat-needs-rebuild-primary")
|
||||
|
||||
errs := bs.ApplyAssignments([]blockvol.BlockVolumeAssignment{
|
||||
{
|
||||
Path: path,
|
||||
Epoch: 1,
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
LeaseTtlMs: 30000,
|
||||
ReplicaServerID: "vs-2",
|
||||
ReplicaDataAddr: "10.0.0.2:4260",
|
||||
ReplicaCtrlAddr: "10.0.0.2:4261",
|
||||
},
|
||||
})
|
||||
if len(errs) != 1 || errs[0] != nil {
|
||||
t.Fatalf("apply assignment errs=%v", errs)
|
||||
}
|
||||
|
||||
bs.applyCoreEvent(engine.NeedsRebuildObserved{ID: path, ReplicaID: path + "/vs-2", Reason: "gap_too_large"})
|
||||
|
||||
msg := findHeartbeatMsg(bs.CollectBlockVolumeHeartbeat(), path)
|
||||
if msg == nil {
|
||||
t.Fatal("volume missing from heartbeat")
|
||||
}
|
||||
if !msg.NeedsRebuild {
|
||||
t.Fatalf("expected explicit needs_rebuild truth on heartbeat, msg=%+v", msg)
|
||||
}
|
||||
if !msg.ReplicaDegraded {
|
||||
t.Fatalf("expected degraded bit to remain true on needs_rebuild path, msg=%+v", msg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockService_HeartbeatIncludesReplicaAddrs(t *testing.T) {
|
||||
bs := newTestBlockServiceDirect(t)
|
||||
path := createTestVolDirect(t, bs, "vol1")
|
||||
|
||||
@@ -20,6 +20,7 @@ type BlockVolumeInfoMessage struct {
|
||||
ReplicaDataAddr string // receiver data listen addr (VS reports in heartbeat)
|
||||
ReplicaCtrlAddr string // receiver ctrl listen addr
|
||||
ReplicaReady bool // explicit replica readiness truth on the heartbeat seam
|
||||
NeedsRebuild bool // explicit needs_rebuild truth on the heartbeat seam
|
||||
HealthScore float64 // CP8-2: 0.0-1.0
|
||||
ScrubErrors int64 // CP8-2: lifetime scrub error count
|
||||
LastScrubTime int64 // CP8-2: unix seconds
|
||||
|
||||
@@ -19,6 +19,7 @@ func InfoMessageToProto(m BlockVolumeInfoMessage) *master_pb.BlockVolumeInfoMess
|
||||
ReplicaDataAddr: m.ReplicaDataAddr,
|
||||
ReplicaCtrlAddr: m.ReplicaCtrlAddr,
|
||||
ReplicaReady: &m.ReplicaReady,
|
||||
NeedsRebuild: &m.NeedsRebuild,
|
||||
HealthScore: m.HealthScore,
|
||||
ScrubErrors: m.ScrubErrors,
|
||||
LastScrubTime: m.LastScrubTime,
|
||||
@@ -47,6 +48,7 @@ func InfoMessageFromProto(p *master_pb.BlockVolumeInfoMessage) BlockVolumeInfoMe
|
||||
ReplicaDataAddr: p.ReplicaDataAddr,
|
||||
ReplicaCtrlAddr: p.ReplicaCtrlAddr,
|
||||
ReplicaReady: p.GetReplicaReady(),
|
||||
NeedsRebuild: p.GetNeedsRebuild(),
|
||||
HealthScore: p.HealthScore,
|
||||
ScrubErrors: p.ScrubErrors,
|
||||
LastScrubTime: p.LastScrubTime,
|
||||
|
||||
@@ -267,6 +267,26 @@ func TestInfoMessage_ReplicaReadyRoundTrip(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestInfoMessage_NeedsRebuildRoundTrip(t *testing.T) {
|
||||
orig := BlockVolumeInfoMessage{
|
||||
Path: "/data/vol.blk",
|
||||
Epoch: 1,
|
||||
NeedsRebuild: true,
|
||||
}
|
||||
pb := InfoMessageToProto(orig)
|
||||
back := InfoMessageFromProto(pb)
|
||||
if !back.NeedsRebuild {
|
||||
t.Fatal("NeedsRebuild should be true after round-trip")
|
||||
}
|
||||
|
||||
orig.NeedsRebuild = false
|
||||
pb = InfoMessageToProto(orig)
|
||||
back = InfoMessageFromProto(pb)
|
||||
if back.NeedsRebuild {
|
||||
t.Fatal("NeedsRebuild should be false after round-trip")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssignment_MultiReplicaRoundTrip(t *testing.T) {
|
||||
orig := BlockVolumeAssignment{
|
||||
Path: "/data/vol.blk",
|
||||
|
||||
Reference in New Issue
Block a user