refactor: rebind bounded replica-ready heartbeat truth

Make the heartbeat/master boundary carry explicit replica readiness truth so the registry no longer depends only on replica transport-address presence as a readiness proxy. Keep backward compatibility by falling back to the old address heuristic when older heartbeats do not send the field.

Made-with: Cursor
This commit is contained in:
pingqiu
2026-04-04 12:06:53 -07:00
parent 43dbebfa04
commit 69b41a7f16
12 changed files with 710 additions and 346 deletions
+123
View File
@@ -1149,3 +1149,126 @@ Conclusion:
relies on broad volume-wide invalidation
2. this slice still does not claim broad failover/publication closure or full
recovery-loop closure
---
#### `16L` Start Note Rev 1
Date: 2026-04-04
Scope: bounded `PublishHealthy` rebinding at the server boundary
Why this slice exists:
1. `debug` and `core projection` surfaces already treat publication health as
core-owned truth
2. but `ReadinessSnapshot.PublishHealthy` still intentionally stays adapter-local
and `CoreProjectionMismatches` still intentionally excludes it
3. that leaves one visible publication seam at the `weed/server` boundary even
though the publication owner is already the core
Chosen implementation rule:
1. rebind `PublishHealthy` to core `Publication.Healthy` when projection exists
2. update only the call sites that were really using `PublishHealthy` as a proxy
for readiness
3. do not broaden this slice into general failover/publication closure claims
---
#### `16L` Delivery Note Rev 1
Date: 2026-04-04
Scope: bounded `PublishHealthy` rebinding at the server boundary
What changed:
1. `weed/server/volume_server_block.go`
- rebound `ReadinessSnapshot.PublishHealthy` to core
`Publication.Healthy` on the core-present path
- `CoreProjectionMismatches` now includes `publish_healthy`
2. `weed/server/volume_server_block_test.go`
- updated focused readiness/debug proofs to assert the rebound publication
truth and added a proof that core healthy publication overrides stale
adapter-local `publishHealthy`
3. `weed/server/block_heartbeat_loop_test.go`
- changed a readiness-only collector proof to depend on `ReplicaEligible`
rather than publication health
Proof / evidence:
1. `go test ./weed/server -count=1 -timeout 120s -run "Test(BlockService_(ApplyAssignments_UpdatesCoreProjection(Replica|Primary)Path|ReadinessSnapshot_PrefersCoreProjectionPrimaryFields|ReadinessSnapshot_PrefersCorePublicationHealth|ReadinessSnapshot_PrefersCoreProjectionReplicaFields|DebugInfoForVolume_UsesCoreProjection(Primary|Replica)Path|CollectBlockVolumeHeartbeat_(Primary|Replica)UsesCoreReadinessGate|HeartbeatReplicaDegraded_UsesCoreMode)|BlockAssign_CollectorUsesAuthoritativeLifecycle)"`
2. result: `PASS`
Conclusion:
1. the server boundary no longer keeps `PublishHealthy` as a separate
adapter-local publication owner on the core-present path
2. this slice still does not claim broad failover/publication closure by itself
---
#### `16M` Start Note Rev 1
Date: 2026-04-04
Scope: bounded explicit `ReplicaReady` heartbeat truth on the heartbeat/master seam
Why this slice exists:
1. `weed/server` already has a bounded core-owned readiness view on the
core-present path
2. but `master_block_registry` still infers replica readiness from
`replica_data_addr` and `replica_ctrl_addr` presence
3. that keeps one avoidable failover/publication seam alive at the
heartbeat/master boundary because transport endpoint presence is only a proxy
for readiness truth
Chosen implementation rule:
1. widen the heartbeat wire additively with an explicit `replica_ready` field
2. emit the field from the same bounded readiness gate already used at the
server boundary on the core-present path
3. make master-side consume prefer explicit readiness and retain address
presence only as backward-compatible fallback
4. do not broaden this slice into promotion selection or broad failover closure
---
#### `16M` Delivery Note Rev 1
Date: 2026-04-04
Scope: bounded explicit `ReplicaReady` heartbeat truth on the heartbeat/master seam
What changed:
1. `weed/pb/master.proto`
- added additive optional `replica_ready` to `BlockVolumeInfoMessage`
2. `weed/pb/master_pb/master.pb.go`
- regenerated so heartbeat wire presence is represented as `*bool`
3. `weed/storage/blockvol/block_heartbeat.go`
- heartbeat wire struct now carries explicit `ReplicaReady`
4. `weed/storage/blockvol/block_heartbeat_proto.go`
- heartbeat conversion now writes and reads `ReplicaReady`
5. `weed/server/volume_server_block.go`
- heartbeat emission now exports explicit bounded `ReplicaReady` truth from
the current core-owned readiness gate
6. `weed/server/master_block_registry.go`
- registry consume now prefers explicit heartbeat readiness and falls back to
transport-address inference only when the field is absent
7. focused tests in `block_heartbeat_proto_test.go`,
`volume_server_block_test.go`, and `master_block_registry_test.go`
- now prove the explicit heartbeat truth and backward-compatible fallback
Proof / evidence:
1. `go test ./weed/storage/blockvol/ -count=1 -run "TestInfoMessage_Replica"`
2. `go test ./weed/server/ -count=1 -timeout 120s -run "Test(BlockService_CollectBlockVolumeHeartbeat_(Primary|Replica)UsesCoreReadinessGate|Registry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady|Registry_UpdateFullHeartbeat_ReplicaReadyFallsBackToAddressesWhenFieldAbsent)"`
3. `go test ./weed/server/ -count=1 -timeout 120s -run "TestBlockService_ApplyAssignments_|TestP16B_|TestP4_"`
4. result: `PASS`
Conclusion:
1. the heartbeat/master seam no longer depends only on replica transport-address
presence as a proxy for readiness truth
2. backward compatibility is preserved because older heartbeats without the
explicit field still fall back to the previous address-based heuristic
3. this slice still does not claim broad failover or promotion closure by itself
+105 -2
View File
@@ -455,6 +455,102 @@ Evidence:
1. focused working-tree change after `5fd9ec0ed`
### `16L`: PublishHealthy Rebinding
Goal:
1. close one bounded publication seam by making `weed/server`
`PublishHealthy` surfaces reflect the core-owned publication truth rather than
an adapter-local convenience bit
2. keep the slice limited to publication-health rebinding at the current
server/debug/readiness boundary, not broad failover/publication closure
Acceptance object:
1. `ReadinessSnapshot.PublishHealthy` reflects core publication truth when the
core projection exists
2. mismatch/debug/readiness surfaces no longer intentionally exclude
`PublishHealthy` from the core-owned publication owner
3. call sites that really need readiness/eligibility rather than publication
health are updated to use the correct readiness field
4. this slice still does not yet claim broad failover/publication closure
Current chosen path:
1. rebind `PublishHealthy` from adapter-local state to core
`Publication.Healthy` on the core-present path
2. update boundary comments and focused tests to match the rebinding
3. keep adapter-local readiness booleans only for truly local readiness state
Status:
1. delivered
Delivered result:
1. `ReadinessSnapshot.PublishHealthy` now mirrors core
`Publication.Healthy` when the core projection exists
2. `CoreProjectionMismatches` no longer treats `PublishHealthy` as an excluded
publication seam
3. call sites that only needed readiness/eligibility now use readiness fields
rather than publication health as a proxy
Evidence:
1. focused working-tree change after `43dbebfa0`
### `16M`: ReplicaReady Heartbeat Truth Rebinding
Goal:
1. close one bounded failover/publication seam by making replica heartbeat
consume carry an explicit `ReplicaReady` truth rather than forcing the master
registry to infer readiness from replica transport address presence
2. keep the slice limited to the current heartbeat wire and master-registry
consume path, not broad failover/promotion closure
Acceptance object:
1. `BlockVolumeInfoMessage` carries an explicit replica-ready bit on the
heartbeat wire
2. `weed/server` heartbeat emission sets that bit from the same core-owned
readiness truth already used at the server boundary on the core-present path
3. `master_block_registry` consumes explicit replica readiness from heartbeat
first and uses address presence only as a backward-compat fallback
4. focused proofs show master-side `ReplicaReady` and `VolumeMode` follow the
explicit heartbeat truth rather than a transport-address heuristic
5. this slice still does not yet claim broad failover/promotion closure
Current chosen path:
1. widen `master.proto` / heartbeat conversion with an additive
`replica_ready` field
2. emit that field from `CollectBlockVolumeHeartbeat` using the current bounded
core-owned readiness gate
3. update registry consume and focused tests without broadening into unrelated
promotion logic
Status:
1. delivered
Delivered result:
1. `BlockVolumeInfoMessage` now carries additive explicit `replica_ready`
heartbeat truth on the wire
2. `weed/server` heartbeat emission now exports explicit bounded
`ReplicaReady` truth from the same core-owned readiness gate already used at
the server boundary
3. `master_block_registry` now prefers explicit heartbeat `ReplicaReady` when
present and falls back to transport-address inference only for older
heartbeats without the field
4. focused proofs now show master-side `ReplicaReady` and `VolumeMode` follow
explicit heartbeat truth rather than transport-address presence alone
Evidence:
1. focused working-tree change after `16L` closeout
## Current Checkpoint Review Target
The current review target is the current widened bounded runtime checkpoint
@@ -525,11 +621,18 @@ boundary:
12. `16K` delivered:
- replica-scoped recovery invalidation no longer depends on a remaining
volume-wide invalidation seam
13. `16L` delivered:
- `PublishHealthy` is rebound from adapter-local status to the core-owned
publication owner at the server boundary
14. `16M` delivered:
- replica heartbeat/master consume now carries explicit bounded
`ReplicaReady` truth with backward-compatible fallback for older
heartbeats
After this checkpoint:
1. keep `legacy P4` only as a compatibility guard
2. continue closing broader recovery-loop gaps one bounded seam at a time after
replica-scoped invalidation
2. continue closing broader recovery-loop and publication seams one bounded step
at a time after `PublishHealthy` rebinding
3. do not yet claim full recovery-loop closure
4. do not broaden into launch claims
+1
View File
@@ -512,6 +512,7 @@ message BlockVolumeInfoMessage {
string durability_mode = 16;
string nvme_addr = 17;
string nqn = 18;
optional bool replica_ready = 19;
}
message BlockVolumeShortInfoMessage {
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -2,7 +2,7 @@
// versions:
// - protoc-gen-go-grpc v1.6.1
// - protoc v3.21.12
// source: master.proto
// source: weed/pb/master.proto
package master_pb
@@ -1278,5 +1278,5 @@ var Seaweed_ServiceDesc = grpc.ServiceDesc{
ClientStreams: true,
},
},
Metadata: "master.proto",
Metadata: "weed/pb/master.proto",
}
+7 -4
View File
@@ -714,6 +714,9 @@ func replicaReadyObservedFromHeartbeat(info *master_pb.BlockVolumeInfoMessage) b
if info == nil {
return false
}
if info.ReplicaReady != nil {
return info.GetReplicaReady()
}
return info.ReplicaDataAddr != "" && info.ReplicaCtrlAddr != ""
}
@@ -1271,10 +1274,10 @@ type PromotionPreflightResult struct {
// evaluatePromotionLocked evaluates promotion candidates for a volume.
// Caller must hold r.mu (read or write). Returns a preflight result without
// mutating the registry. The four gates:
// 1. Heartbeat freshness (within 2×LeaseTTL)
// 2. WAL LSN recency (within promotionLSNTolerance of primary)
// 3. Role must be RoleReplica (not RoleRebuilding)
// 4. Server must be in blockServers (alive) — fixes B-12
// 1. Heartbeat freshness (within 2×LeaseTTL)
// 2. WAL LSN recency (within promotionLSNTolerance of primary)
// 3. Role must be RoleReplica (not RoleRebuilding)
// 4. Server must be in blockServers (alive) — fixes B-12
func (r *BlockVolumeRegistry) evaluatePromotionLocked(entry *BlockVolumeEntry) PromotionPreflightResult {
result := PromotionPreflightResult{
VolumeName: entry.Name,
+40
View File
@@ -2103,10 +2103,18 @@ func TestRegistry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady(t *test
if hb.ReplicaDataAddr == "" || hb.ReplicaCtrlAddr == "" {
t.Fatalf("expected core-influenced replica addresses on heartbeat, hb=%+v", hb)
}
if !hb.ReplicaReady {
t.Fatalf("expected explicit replica_ready truth on heartbeat, hb=%+v", hb)
}
if hb.ReplicaDegraded {
t.Fatalf("did not expect degraded heartbeat on ready path, hb=%+v", hb)
}
// Prove master-side readiness now follows the explicit heartbeat truth even
// when transport addresses are absent on the consume path.
hb.ReplicaDataAddr = ""
hb.ReplicaCtrlAddr = ""
r := NewBlockVolumeRegistry()
if err := r.Register(&BlockVolumeEntry{
Name: "vol-master-ready",
@@ -2139,3 +2147,35 @@ func TestRegistry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady(t *test
t.Fatalf("expected publish_healthy after ready consume, got %q", entry.VolumeMode)
}
}
func TestRegistry_UpdateFullHeartbeat_ReplicaReadyFallsBackToAddressesWhenFieldAbsent(t *testing.T) {
r := NewBlockVolumeRegistry()
if err := r.Register(&BlockVolumeEntry{
Name: "vol-master-ready-fallback",
VolumeServer: "primary-server:8080",
Path: "/blocks/vol-master-ready-fallback-primary.blk",
Status: StatusActive,
Role: blockvol.RoleToWire(blockvol.RolePrimary),
ReplicaFactor: 2,
Replicas: []ReplicaInfo{{
Server: "replica-server:8080",
Path: "/blocks/vol-master-ready-fallback.blk",
}},
}); err != nil {
t.Fatalf("register: %v", err)
}
r.UpdateFullHeartbeat("replica-server:8080", []*master_pb.BlockVolumeInfoMessage{{
Path: "/blocks/vol-master-ready-fallback.blk",
ReplicaDataAddr: "10.0.0.2:4260",
ReplicaCtrlAddr: "10.0.0.2:4261",
}}, "")
entry, _ := r.Lookup("vol-master-ready-fallback")
if !entry.Replicas[0].Ready {
t.Fatalf("expected backward-compatible ready fallback from addresses, entry=%+v", entry)
}
if !entry.ReplicaReady {
t.Fatalf("expected aggregate replica ready from fallback consume, entry=%+v", entry)
}
}
+28 -9
View File
@@ -38,9 +38,9 @@ type volReplState struct {
// not by blockvol's local storage mechanics.
//
// Important:
// PublishHealthy here is still an adapter-local publication bit used by current
// `weed/server` surfaces. It is NOT the semantic owner for Phase 14 core
// publication health; that owner is `engine.PublicationView`.
// PublishHealthy here is the server-boundary mirror of the core-owned
// publication health when a core projection exists. Adapter-local fallback
// remains only for paths where the core is absent.
type BlockReadinessSnapshot struct {
RoleApplied bool
ReceiverReady bool
@@ -140,8 +140,7 @@ func (bs *BlockService) ExecutedCoreCommands(path string) []string {
}
// CoreProjectionMismatches reports fields that should already agree on the
// narrow Phase 15A path but do not. It intentionally excludes adapter-local
// `PublishHealthy`, which is not yet rebound to the core publication owner.
// current bounded core-present path but do not.
func (bs *BlockService) CoreProjectionMismatches(path string) []string {
proj, ok := bs.CoreProjection(path)
if !ok {
@@ -161,6 +160,9 @@ func (bs *BlockService) CoreProjectionMismatches(path string) []string {
if readiness.ShipperConnected != proj.Readiness.ShipperConnected {
mismatches = append(mismatches, "shipper_connected")
}
if readiness.PublishHealthy != proj.Publication.Healthy {
mismatches = append(mismatches, "publish_healthy")
}
return mismatches
}
@@ -963,6 +965,7 @@ func (bs *BlockService) CollectBlockVolumeHeartbeat() []blockvol.BlockVolumeInfo
for i := range msgs {
if s, ok := bs.replStates[msgs[i].Path]; ok {
msgs[i].ReplicaDataAddr, msgs[i].ReplicaCtrlAddr = bs.heartbeatReplicaAddrs(msgs[i].Path, s)
msgs[i].ReplicaReady = bs.heartbeatReplicaReady(msgs[i].Path, s)
}
msgs[i].ReplicaDegraded = bs.heartbeatReplicaDegraded(msgs[i].Path, msgs[i].ReplicaDegraded)
// NVMe publication: report nvme_addr and nqn if NVMe target is running.
@@ -1004,6 +1007,20 @@ func (bs *BlockService) heartbeatReplicaAddrs(path string, state *volReplState)
return "", ""
}
// heartbeatReplicaReady returns the explicit replica-readiness truth that should
// be exposed on the current heartbeat surface. On the core-present path it
// prefers the core-owned readiness projection; older paths fall back to the
// adapter-local publish gate for compatibility.
func (bs *BlockService) heartbeatReplicaReady(path string, state *volReplState) bool {
if state == nil {
return false
}
if proj, ok := bs.CoreProjection(path); ok {
return proj.Readiness.ReplicaReady
}
return state.publishHealthy
}
// heartbeatReplicaDegraded returns the bounded degraded bit for the current
// heartbeat surface. On the Phase 15 live path it prefers the core mode when
// present, then falls back to the runtime-local status bit.
@@ -1118,10 +1135,10 @@ func (bs *BlockService) markReceiverReady(path, dataAddr, ctrlAddr string) {
state.allReplicas = nil
}
// ReadinessSnapshot reports the service-owned assignment/readiness closure for
// one volume. On the Phase 15 live path it prefers the explicit core projection
// for the aligned readiness subset, while `PublishHealthy` remains adapter-local
// until publication ownership is fully rebound.
// ReadinessSnapshot reports the current server-boundary readiness/publication
// closure for one volume. On the core-present path it prefers the explicit core
// projection for aligned readiness plus publication health, while retaining
// adapter-local fallback only when the core is absent.
func (bs *BlockService) ReadinessSnapshot(path string) BlockReadinessSnapshot {
snap := BlockReadinessSnapshot{}
bs.replMu.RLock()
@@ -1141,6 +1158,7 @@ func (bs *BlockService) ReadinessSnapshot(path string) BlockReadinessSnapshot {
snap.ShipperConfigured = proj.Readiness.ShipperConfigured
snap.ShipperConnected = proj.Readiness.ShipperConnected
snap.ReplicaEligible = proj.Readiness.ReplicaReady
snap.PublishHealthy = proj.Publication.Healthy
}
return snap
}
@@ -1154,6 +1172,7 @@ func (bs *BlockService) ReadinessSnapshot(path string) BlockReadinessSnapshot {
snap.ShipperConfigured = proj.Readiness.ShipperConfigured
snap.ShipperConnected = proj.Readiness.ShipperConnected
snap.ReplicaEligible = proj.Readiness.ReplicaReady
snap.PublishHealthy = proj.Publication.Healthy
}
return snap
}
+50 -4
View File
@@ -937,8 +937,8 @@ func TestBlockService_DebugInfoForVolume_UsesCoreProjectionPrimaryPath(t *testin
}
readiness := bs.ReadinessSnapshot(path)
if !readiness.PublishHealthy {
t.Fatalf("expected adapter-local readiness to still report publish healthy, got %+v", readiness)
if readiness.PublishHealthy {
t.Fatalf("readiness snapshot must follow core publication truth on primary path without durable boundary, got %+v", readiness)
}
vol, ok := bs.blockStore.GetBlockVolume(path)
@@ -1064,6 +1064,9 @@ func TestBlockService_CollectBlockVolumeHeartbeat_PrimaryUsesCoreReadinessGate(t
if msg.ReplicaCtrlAddr != "10.0.0.2:4261" {
t.Fatalf("ReplicaCtrlAddr=%q", msg.ReplicaCtrlAddr)
}
if msg.ReplicaReady {
t.Fatalf("primary heartbeat must not claim replica-ready truth, msg=%+v", msg)
}
}
func TestBlockService_CollectBlockVolumeHeartbeat_ReplicaUsesCoreReadinessGate(t *testing.T) {
@@ -1117,6 +1120,9 @@ func TestBlockService_CollectBlockVolumeHeartbeat_ReplicaUsesCoreReadinessGate(t
if msg.ReplicaCtrlAddr != expectedCtrl {
t.Fatalf("ReplicaCtrlAddr=%q expected=%q", msg.ReplicaCtrlAddr, expectedCtrl)
}
if !msg.ReplicaReady {
t.Fatalf("replica heartbeat should carry explicit replica-ready truth, msg=%+v", msg)
}
}
func TestBlockService_ReadinessSnapshot_PrefersCoreProjectionPrimaryFields(t *testing.T) {
@@ -1157,7 +1163,47 @@ func TestBlockService_ReadinessSnapshot_PrefersCoreProjectionPrimaryFields(t *te
t.Fatalf("expected snapshot shipper_configured from core projection, got %+v", snap)
}
if snap.PublishHealthy {
t.Fatalf("publish_healthy should remain adapter-local on readiness snapshot, got %+v", snap)
t.Fatalf("publish_healthy should follow core publication truth on primary path without durable boundary, got %+v", snap)
}
}
func TestBlockService_ReadinessSnapshot_PrefersCorePublicationHealth(t *testing.T) {
bs := newTestBlockServiceDirect(t)
path := createTestVolDirect(t, bs, "vol-readiness-publish-healthy")
errs := bs.ApplyAssignments([]blockvol.BlockVolumeAssignment{
{
Path: path,
Epoch: 1,
Role: blockvol.RoleToWire(blockvol.RolePrimary),
LeaseTtlMs: 30000,
ReplicaServerID: "vs-2",
ReplicaDataAddr: "10.0.0.2:4260",
ReplicaCtrlAddr: "10.0.0.2:4261",
},
})
if len(errs) != 1 || errs[0] != nil {
t.Fatalf("apply assignment errs=%v", errs)
}
bs.applyCoreEvent(engine.ShipperConnectedObserved{ID: path})
bs.applyCoreEvent(engine.BarrierAccepted{ID: path, FlushedLSN: 12})
bs.replMu.Lock()
state := bs.replStates[path]
if state == nil {
bs.replMu.Unlock()
t.Fatal("missing repl state")
}
state.publishHealthy = false
bs.replMu.Unlock()
snap := bs.ReadinessSnapshot(path)
if !snap.PublishHealthy {
t.Fatalf("publish_healthy should follow core publication truth once healthy, got %+v", snap)
}
if mismatches := bs.CoreProjectionMismatches(path); len(mismatches) != 0 {
t.Fatalf("readiness/core mismatches=%v", mismatches)
}
}
@@ -1198,7 +1244,7 @@ func TestBlockService_ReadinessSnapshot_PrefersCoreProjectionReplicaFields(t *te
t.Fatalf("expected snapshot replica_eligible from core projection, got %+v", snap)
}
if snap.PublishHealthy {
t.Fatalf("publish_healthy should remain adapter-local on readiness snapshot, got %+v", snap)
t.Fatalf("publish_healthy should follow core publication truth on replica path, got %+v", snap)
}
}
+29 -28
View File
@@ -8,22 +8,23 @@ import (
// BlockVolumeInfoMessage is the heartbeat status for one block volume.
// Mirrors the proto message that will be generated from master.proto.
type BlockVolumeInfoMessage struct {
Path string // volume file path (unique ID on this server)
VolumeSize uint64 // logical size in bytes
BlockSize uint32 // block size in bytes
Epoch uint64 // current fencing epoch
Role uint32 // blockvol.Role as uint32 for wire compat
WalHeadLsn uint64 // WAL head LSN
CheckpointLsn uint64 // last flushed LSN
HasLease bool // whether volume holds a valid lease
DiskType string // e.g., "ssd", "hdd"
ReplicaDataAddr string // receiver data listen addr (VS reports in heartbeat)
ReplicaCtrlAddr string // receiver ctrl listen addr
HealthScore float64 // CP8-2: 0.0-1.0
ScrubErrors int64 // CP8-2: lifetime scrub error count
LastScrubTime int64 // CP8-2: unix seconds
ReplicaDegraded bool // CP8-2: true if any replica shipper degraded
DurabilityMode string // CP8-3-1: "best_effort", "sync_all", "sync_quorum"
Path string // volume file path (unique ID on this server)
VolumeSize uint64 // logical size in bytes
BlockSize uint32 // block size in bytes
Epoch uint64 // current fencing epoch
Role uint32 // blockvol.Role as uint32 for wire compat
WalHeadLsn uint64 // WAL head LSN
CheckpointLsn uint64 // last flushed LSN
HasLease bool // whether volume holds a valid lease
DiskType string // e.g., "ssd", "hdd"
ReplicaDataAddr string // receiver data listen addr (VS reports in heartbeat)
ReplicaCtrlAddr string // receiver ctrl listen addr
ReplicaReady bool // explicit replica readiness truth on the heartbeat seam
HealthScore float64 // CP8-2: 0.0-1.0
ScrubErrors int64 // CP8-2: lifetime scrub error count
LastScrubTime int64 // CP8-2: unix seconds
ReplicaDegraded bool // CP8-2: true if any replica shipper degraded
DurabilityMode string // CP8-3-1: "best_effort", "sync_all", "sync_quorum"
NvmeAddr string // NVMe/TCP target address (ip:port), empty if NVMe disabled
NQN string // NVMe subsystem NQN, empty if NVMe disabled
ReplicaShipperStates []ReplicaShipperStatus // CP13-7: per-replica state from primary's shipper group
@@ -60,18 +61,18 @@ func ToBlockVolumeInfoMessage(path, diskType string, vol *BlockVol) BlockVolumeI
status := vol.Status()
hs := vol.HealthStats()
return BlockVolumeInfoMessage{
Path: path,
VolumeSize: info.VolumeSize,
BlockSize: info.BlockSize,
Epoch: status.Epoch,
Role: RoleToWire(status.Role),
WalHeadLsn: status.WALHeadLSN,
CheckpointLsn: status.CheckpointLSN,
HasLease: status.HasLease,
DiskType: diskType,
HealthScore: status.HealthScore,
ScrubErrors: hs.ScrubErrors,
LastScrubTime: hs.LastScrubTime,
Path: path,
VolumeSize: info.VolumeSize,
BlockSize: info.BlockSize,
Epoch: status.Epoch,
Role: RoleToWire(status.Role),
WalHeadLsn: status.WALHeadLSN,
CheckpointLsn: status.CheckpointLSN,
HasLease: status.HasLease,
DiskType: diskType,
HealthScore: status.HealthScore,
ScrubErrors: hs.ScrubErrors,
LastScrubTime: hs.LastScrubTime,
ReplicaDegraded: status.ReplicaDegraded,
DurabilityMode: vol.DurabilityMode().String(),
ReplicaShipperStates: vol.ReplicaShipperStates(),
@@ -18,6 +18,7 @@ func InfoMessageToProto(m BlockVolumeInfoMessage) *master_pb.BlockVolumeInfoMess
DiskType: m.DiskType,
ReplicaDataAddr: m.ReplicaDataAddr,
ReplicaCtrlAddr: m.ReplicaCtrlAddr,
ReplicaReady: &m.ReplicaReady,
HealthScore: m.HealthScore,
ScrubErrors: m.ScrubErrors,
LastScrubTime: m.LastScrubTime,
@@ -45,6 +46,7 @@ func InfoMessageFromProto(p *master_pb.BlockVolumeInfoMessage) BlockVolumeInfoMe
DiskType: p.DiskType,
ReplicaDataAddr: p.ReplicaDataAddr,
ReplicaCtrlAddr: p.ReplicaCtrlAddr,
ReplicaReady: p.GetReplicaReady(),
HealthScore: p.HealthScore,
ScrubErrors: p.ScrubErrors,
LastScrubTime: p.LastScrubTime,
@@ -166,4 +168,3 @@ func AssignmentsFromProto(protos []*master_pb.BlockVolumeAssignment) []BlockVolu
}
return out
}
@@ -205,12 +205,12 @@ func TestNilProtoConversions(t *testing.T) {
func TestInfoMessage_HealthScoreRoundTrip(t *testing.T) {
orig := BlockVolumeInfoMessage{
Path: "/data/vol.blk",
VolumeSize: 1 << 30,
BlockSize: 4096,
Epoch: 5,
HealthScore: 0.85,
ScrubErrors: 3,
Path: "/data/vol.blk",
VolumeSize: 1 << 30,
BlockSize: 4096,
Epoch: 5,
HealthScore: 0.85,
ScrubErrors: 3,
LastScrubTime: 1709000000,
}
pb := InfoMessageToProto(orig)
@@ -247,6 +247,26 @@ func TestInfoMessage_ReplicaDegradedRoundTrip(t *testing.T) {
}
}
func TestInfoMessage_ReplicaReadyRoundTrip(t *testing.T) {
orig := BlockVolumeInfoMessage{
Path: "/data/vol.blk",
Epoch: 1,
ReplicaReady: true,
}
pb := InfoMessageToProto(orig)
back := InfoMessageFromProto(pb)
if !back.ReplicaReady {
t.Fatal("ReplicaReady should be true after round-trip")
}
orig.ReplicaReady = false
pb = InfoMessageToProto(orig)
back = InfoMessageFromProto(pb)
if back.ReplicaReady {
t.Fatal("ReplicaReady should be false after round-trip")
}
}
func TestAssignment_MultiReplicaRoundTrip(t *testing.T) {
orig := BlockVolumeAssignment{
Path: "/data/vol.blk",