mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-21 22:56:55 +00:00
refactor: rebind bounded replica-ready heartbeat truth
Make the heartbeat/master boundary carry explicit replica readiness truth so the registry no longer depends only on replica transport-address presence as a readiness proxy. Keep backward compatibility by falling back to the old address heuristic when older heartbeats do not send the field. Made-with: Cursor
This commit is contained in:
@@ -1149,3 +1149,126 @@ Conclusion:
|
||||
relies on broad volume-wide invalidation
|
||||
2. this slice still does not claim broad failover/publication closure or full
|
||||
recovery-loop closure
|
||||
|
||||
---
|
||||
|
||||
#### `16L` Start Note Rev 1
|
||||
|
||||
Date: 2026-04-04
|
||||
Scope: bounded `PublishHealthy` rebinding at the server boundary
|
||||
|
||||
Why this slice exists:
|
||||
|
||||
1. `debug` and `core projection` surfaces already treat publication health as
|
||||
core-owned truth
|
||||
2. but `ReadinessSnapshot.PublishHealthy` still intentionally stays adapter-local
|
||||
and `CoreProjectionMismatches` still intentionally excludes it
|
||||
3. that leaves one visible publication seam at the `weed/server` boundary even
|
||||
though the publication owner is already the core
|
||||
|
||||
Chosen implementation rule:
|
||||
|
||||
1. rebind `PublishHealthy` to core `Publication.Healthy` when projection exists
|
||||
2. update only the call sites that were really using `PublishHealthy` as a proxy
|
||||
for readiness
|
||||
3. do not broaden this slice into general failover/publication closure claims
|
||||
|
||||
---
|
||||
|
||||
#### `16L` Delivery Note Rev 1
|
||||
|
||||
Date: 2026-04-04
|
||||
Scope: bounded `PublishHealthy` rebinding at the server boundary
|
||||
|
||||
What changed:
|
||||
|
||||
1. `weed/server/volume_server_block.go`
|
||||
- rebound `ReadinessSnapshot.PublishHealthy` to core
|
||||
`Publication.Healthy` on the core-present path
|
||||
- `CoreProjectionMismatches` now includes `publish_healthy`
|
||||
2. `weed/server/volume_server_block_test.go`
|
||||
- updated focused readiness/debug proofs to assert the rebound publication
|
||||
truth and added a proof that core healthy publication overrides stale
|
||||
adapter-local `publishHealthy`
|
||||
3. `weed/server/block_heartbeat_loop_test.go`
|
||||
- changed a readiness-only collector proof to depend on `ReplicaEligible`
|
||||
rather than publication health
|
||||
|
||||
Proof / evidence:
|
||||
|
||||
1. `go test ./weed/server -count=1 -timeout 120s -run "Test(BlockService_(ApplyAssignments_UpdatesCoreProjection(Replica|Primary)Path|ReadinessSnapshot_PrefersCoreProjectionPrimaryFields|ReadinessSnapshot_PrefersCorePublicationHealth|ReadinessSnapshot_PrefersCoreProjectionReplicaFields|DebugInfoForVolume_UsesCoreProjection(Primary|Replica)Path|CollectBlockVolumeHeartbeat_(Primary|Replica)UsesCoreReadinessGate|HeartbeatReplicaDegraded_UsesCoreMode)|BlockAssign_CollectorUsesAuthoritativeLifecycle)"`
|
||||
2. result: `PASS`
|
||||
|
||||
Conclusion:
|
||||
|
||||
1. the server boundary no longer keeps `PublishHealthy` as a separate
|
||||
adapter-local publication owner on the core-present path
|
||||
2. this slice still does not claim broad failover/publication closure by itself
|
||||
|
||||
---
|
||||
|
||||
#### `16M` Start Note Rev 1
|
||||
|
||||
Date: 2026-04-04
|
||||
Scope: bounded explicit `ReplicaReady` heartbeat truth on the heartbeat/master seam
|
||||
|
||||
Why this slice exists:
|
||||
|
||||
1. `weed/server` already has a bounded core-owned readiness view on the
|
||||
core-present path
|
||||
2. but `master_block_registry` still infers replica readiness from
|
||||
`replica_data_addr` and `replica_ctrl_addr` presence
|
||||
3. that keeps one avoidable failover/publication seam alive at the
|
||||
heartbeat/master boundary because transport endpoint presence is only a proxy
|
||||
for readiness truth
|
||||
|
||||
Chosen implementation rule:
|
||||
|
||||
1. widen the heartbeat wire additively with an explicit `replica_ready` field
|
||||
2. emit the field from the same bounded readiness gate already used at the
|
||||
server boundary on the core-present path
|
||||
3. make master-side consume prefer explicit readiness and retain address
|
||||
presence only as backward-compatible fallback
|
||||
4. do not broaden this slice into promotion selection or broad failover closure
|
||||
|
||||
---
|
||||
|
||||
#### `16M` Delivery Note Rev 1
|
||||
|
||||
Date: 2026-04-04
|
||||
Scope: bounded explicit `ReplicaReady` heartbeat truth on the heartbeat/master seam
|
||||
|
||||
What changed:
|
||||
|
||||
1. `weed/pb/master.proto`
|
||||
- added additive optional `replica_ready` to `BlockVolumeInfoMessage`
|
||||
2. `weed/pb/master_pb/master.pb.go`
|
||||
- regenerated so heartbeat wire presence is represented as `*bool`
|
||||
3. `weed/storage/blockvol/block_heartbeat.go`
|
||||
- heartbeat wire struct now carries explicit `ReplicaReady`
|
||||
4. `weed/storage/blockvol/block_heartbeat_proto.go`
|
||||
- heartbeat conversion now writes and reads `ReplicaReady`
|
||||
5. `weed/server/volume_server_block.go`
|
||||
- heartbeat emission now exports explicit bounded `ReplicaReady` truth from
|
||||
the current core-owned readiness gate
|
||||
6. `weed/server/master_block_registry.go`
|
||||
- registry consume now prefers explicit heartbeat readiness and falls back to
|
||||
transport-address inference only when the field is absent
|
||||
7. focused tests in `block_heartbeat_proto_test.go`,
|
||||
`volume_server_block_test.go`, and `master_block_registry_test.go`
|
||||
- now prove the explicit heartbeat truth and backward-compatible fallback
|
||||
|
||||
Proof / evidence:
|
||||
|
||||
1. `go test ./weed/storage/blockvol/ -count=1 -run "TestInfoMessage_Replica"`
|
||||
2. `go test ./weed/server/ -count=1 -timeout 120s -run "Test(BlockService_CollectBlockVolumeHeartbeat_(Primary|Replica)UsesCoreReadinessGate|Registry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady|Registry_UpdateFullHeartbeat_ReplicaReadyFallsBackToAddressesWhenFieldAbsent)"`
|
||||
3. `go test ./weed/server/ -count=1 -timeout 120s -run "TestBlockService_ApplyAssignments_|TestP16B_|TestP4_"`
|
||||
4. result: `PASS`
|
||||
|
||||
Conclusion:
|
||||
|
||||
1. the heartbeat/master seam no longer depends only on replica transport-address
|
||||
presence as a proxy for readiness truth
|
||||
2. backward compatibility is preserved because older heartbeats without the
|
||||
explicit field still fall back to the previous address-based heuristic
|
||||
3. this slice still does not claim broad failover or promotion closure by itself
|
||||
|
||||
@@ -455,6 +455,102 @@ Evidence:
|
||||
|
||||
1. focused working-tree change after `5fd9ec0ed`
|
||||
|
||||
### `16L`: PublishHealthy Rebinding
|
||||
|
||||
Goal:
|
||||
|
||||
1. close one bounded publication seam by making `weed/server`
|
||||
`PublishHealthy` surfaces reflect the core-owned publication truth rather than
|
||||
an adapter-local convenience bit
|
||||
2. keep the slice limited to publication-health rebinding at the current
|
||||
server/debug/readiness boundary, not broad failover/publication closure
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `ReadinessSnapshot.PublishHealthy` reflects core publication truth when the
|
||||
core projection exists
|
||||
2. mismatch/debug/readiness surfaces no longer intentionally exclude
|
||||
`PublishHealthy` from the core-owned publication owner
|
||||
3. call sites that really need readiness/eligibility rather than publication
|
||||
health are updated to use the correct readiness field
|
||||
4. this slice still does not yet claim broad failover/publication closure
|
||||
|
||||
Current chosen path:
|
||||
|
||||
1. rebind `PublishHealthy` from adapter-local state to core
|
||||
`Publication.Healthy` on the core-present path
|
||||
2. update boundary comments and focused tests to match the rebinding
|
||||
3. keep adapter-local readiness booleans only for truly local readiness state
|
||||
|
||||
Status:
|
||||
|
||||
1. delivered
|
||||
|
||||
Delivered result:
|
||||
|
||||
1. `ReadinessSnapshot.PublishHealthy` now mirrors core
|
||||
`Publication.Healthy` when the core projection exists
|
||||
2. `CoreProjectionMismatches` no longer treats `PublishHealthy` as an excluded
|
||||
publication seam
|
||||
3. call sites that only needed readiness/eligibility now use readiness fields
|
||||
rather than publication health as a proxy
|
||||
|
||||
Evidence:
|
||||
|
||||
1. focused working-tree change after `43dbebfa0`
|
||||
|
||||
### `16M`: ReplicaReady Heartbeat Truth Rebinding
|
||||
|
||||
Goal:
|
||||
|
||||
1. close one bounded failover/publication seam by making replica heartbeat
|
||||
consume carry an explicit `ReplicaReady` truth rather than forcing the master
|
||||
registry to infer readiness from replica transport address presence
|
||||
2. keep the slice limited to the current heartbeat wire and master-registry
|
||||
consume path, not broad failover/promotion closure
|
||||
|
||||
Acceptance object:
|
||||
|
||||
1. `BlockVolumeInfoMessage` carries an explicit replica-ready bit on the
|
||||
heartbeat wire
|
||||
2. `weed/server` heartbeat emission sets that bit from the same core-owned
|
||||
readiness truth already used at the server boundary on the core-present path
|
||||
3. `master_block_registry` consumes explicit replica readiness from heartbeat
|
||||
first and uses address presence only as a backward-compat fallback
|
||||
4. focused proofs show master-side `ReplicaReady` and `VolumeMode` follow the
|
||||
explicit heartbeat truth rather than a transport-address heuristic
|
||||
5. this slice still does not yet claim broad failover/promotion closure
|
||||
|
||||
Current chosen path:
|
||||
|
||||
1. widen `master.proto` / heartbeat conversion with an additive
|
||||
`replica_ready` field
|
||||
2. emit that field from `CollectBlockVolumeHeartbeat` using the current bounded
|
||||
core-owned readiness gate
|
||||
3. update registry consume and focused tests without broadening into unrelated
|
||||
promotion logic
|
||||
|
||||
Status:
|
||||
|
||||
1. delivered
|
||||
|
||||
Delivered result:
|
||||
|
||||
1. `BlockVolumeInfoMessage` now carries additive explicit `replica_ready`
|
||||
heartbeat truth on the wire
|
||||
2. `weed/server` heartbeat emission now exports explicit bounded
|
||||
`ReplicaReady` truth from the same core-owned readiness gate already used at
|
||||
the server boundary
|
||||
3. `master_block_registry` now prefers explicit heartbeat `ReplicaReady` when
|
||||
present and falls back to transport-address inference only for older
|
||||
heartbeats without the field
|
||||
4. focused proofs now show master-side `ReplicaReady` and `VolumeMode` follow
|
||||
explicit heartbeat truth rather than transport-address presence alone
|
||||
|
||||
Evidence:
|
||||
|
||||
1. focused working-tree change after `16L` closeout
|
||||
|
||||
## Current Checkpoint Review Target
|
||||
|
||||
The current review target is the current widened bounded runtime checkpoint
|
||||
@@ -525,11 +621,18 @@ boundary:
|
||||
12. `16K` delivered:
|
||||
- replica-scoped recovery invalidation no longer depends on a remaining
|
||||
volume-wide invalidation seam
|
||||
13. `16L` delivered:
|
||||
- `PublishHealthy` is rebound from adapter-local status to the core-owned
|
||||
publication owner at the server boundary
|
||||
14. `16M` delivered:
|
||||
- replica heartbeat/master consume now carries explicit bounded
|
||||
`ReplicaReady` truth with backward-compatible fallback for older
|
||||
heartbeats
|
||||
|
||||
After this checkpoint:
|
||||
|
||||
1. keep `legacy P4` only as a compatibility guard
|
||||
2. continue closing broader recovery-loop gaps one bounded seam at a time after
|
||||
replica-scoped invalidation
|
||||
2. continue closing broader recovery-loop and publication seams one bounded step
|
||||
at a time after `PublishHealthy` rebinding
|
||||
3. do not yet claim full recovery-loop closure
|
||||
4. do not broaden into launch claims
|
||||
|
||||
@@ -512,6 +512,7 @@ message BlockVolumeInfoMessage {
|
||||
string durability_mode = 16;
|
||||
string nvme_addr = 17;
|
||||
string nqn = 18;
|
||||
optional bool replica_ready = 19;
|
||||
}
|
||||
|
||||
message BlockVolumeShortInfoMessage {
|
||||
|
||||
+297
-290
File diff suppressed because it is too large
Load Diff
@@ -2,7 +2,7 @@
|
||||
// versions:
|
||||
// - protoc-gen-go-grpc v1.6.1
|
||||
// - protoc v3.21.12
|
||||
// source: master.proto
|
||||
// source: weed/pb/master.proto
|
||||
|
||||
package master_pb
|
||||
|
||||
@@ -1278,5 +1278,5 @@ var Seaweed_ServiceDesc = grpc.ServiceDesc{
|
||||
ClientStreams: true,
|
||||
},
|
||||
},
|
||||
Metadata: "master.proto",
|
||||
Metadata: "weed/pb/master.proto",
|
||||
}
|
||||
|
||||
@@ -714,6 +714,9 @@ func replicaReadyObservedFromHeartbeat(info *master_pb.BlockVolumeInfoMessage) b
|
||||
if info == nil {
|
||||
return false
|
||||
}
|
||||
if info.ReplicaReady != nil {
|
||||
return info.GetReplicaReady()
|
||||
}
|
||||
return info.ReplicaDataAddr != "" && info.ReplicaCtrlAddr != ""
|
||||
}
|
||||
|
||||
@@ -1271,10 +1274,10 @@ type PromotionPreflightResult struct {
|
||||
// evaluatePromotionLocked evaluates promotion candidates for a volume.
|
||||
// Caller must hold r.mu (read or write). Returns a preflight result without
|
||||
// mutating the registry. The four gates:
|
||||
// 1. Heartbeat freshness (within 2×LeaseTTL)
|
||||
// 2. WAL LSN recency (within promotionLSNTolerance of primary)
|
||||
// 3. Role must be RoleReplica (not RoleRebuilding)
|
||||
// 4. Server must be in blockServers (alive) — fixes B-12
|
||||
// 1. Heartbeat freshness (within 2×LeaseTTL)
|
||||
// 2. WAL LSN recency (within promotionLSNTolerance of primary)
|
||||
// 3. Role must be RoleReplica (not RoleRebuilding)
|
||||
// 4. Server must be in blockServers (alive) — fixes B-12
|
||||
func (r *BlockVolumeRegistry) evaluatePromotionLocked(entry *BlockVolumeEntry) PromotionPreflightResult {
|
||||
result := PromotionPreflightResult{
|
||||
VolumeName: entry.Name,
|
||||
|
||||
@@ -2103,10 +2103,18 @@ func TestRegistry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady(t *test
|
||||
if hb.ReplicaDataAddr == "" || hb.ReplicaCtrlAddr == "" {
|
||||
t.Fatalf("expected core-influenced replica addresses on heartbeat, hb=%+v", hb)
|
||||
}
|
||||
if !hb.ReplicaReady {
|
||||
t.Fatalf("expected explicit replica_ready truth on heartbeat, hb=%+v", hb)
|
||||
}
|
||||
if hb.ReplicaDegraded {
|
||||
t.Fatalf("did not expect degraded heartbeat on ready path, hb=%+v", hb)
|
||||
}
|
||||
|
||||
// Prove master-side readiness now follows the explicit heartbeat truth even
|
||||
// when transport addresses are absent on the consume path.
|
||||
hb.ReplicaDataAddr = ""
|
||||
hb.ReplicaCtrlAddr = ""
|
||||
|
||||
r := NewBlockVolumeRegistry()
|
||||
if err := r.Register(&BlockVolumeEntry{
|
||||
Name: "vol-master-ready",
|
||||
@@ -2139,3 +2147,35 @@ func TestRegistry_UpdateFullHeartbeat_ConsumesCoreInfluencedReplicaReady(t *test
|
||||
t.Fatalf("expected publish_healthy after ready consume, got %q", entry.VolumeMode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistry_UpdateFullHeartbeat_ReplicaReadyFallsBackToAddressesWhenFieldAbsent(t *testing.T) {
|
||||
r := NewBlockVolumeRegistry()
|
||||
if err := r.Register(&BlockVolumeEntry{
|
||||
Name: "vol-master-ready-fallback",
|
||||
VolumeServer: "primary-server:8080",
|
||||
Path: "/blocks/vol-master-ready-fallback-primary.blk",
|
||||
Status: StatusActive,
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
ReplicaFactor: 2,
|
||||
Replicas: []ReplicaInfo{{
|
||||
Server: "replica-server:8080",
|
||||
Path: "/blocks/vol-master-ready-fallback.blk",
|
||||
}},
|
||||
}); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
|
||||
r.UpdateFullHeartbeat("replica-server:8080", []*master_pb.BlockVolumeInfoMessage{{
|
||||
Path: "/blocks/vol-master-ready-fallback.blk",
|
||||
ReplicaDataAddr: "10.0.0.2:4260",
|
||||
ReplicaCtrlAddr: "10.0.0.2:4261",
|
||||
}}, "")
|
||||
|
||||
entry, _ := r.Lookup("vol-master-ready-fallback")
|
||||
if !entry.Replicas[0].Ready {
|
||||
t.Fatalf("expected backward-compatible ready fallback from addresses, entry=%+v", entry)
|
||||
}
|
||||
if !entry.ReplicaReady {
|
||||
t.Fatalf("expected aggregate replica ready from fallback consume, entry=%+v", entry)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,9 +38,9 @@ type volReplState struct {
|
||||
// not by blockvol's local storage mechanics.
|
||||
//
|
||||
// Important:
|
||||
// PublishHealthy here is still an adapter-local publication bit used by current
|
||||
// `weed/server` surfaces. It is NOT the semantic owner for Phase 14 core
|
||||
// publication health; that owner is `engine.PublicationView`.
|
||||
// PublishHealthy here is the server-boundary mirror of the core-owned
|
||||
// publication health when a core projection exists. Adapter-local fallback
|
||||
// remains only for paths where the core is absent.
|
||||
type BlockReadinessSnapshot struct {
|
||||
RoleApplied bool
|
||||
ReceiverReady bool
|
||||
@@ -140,8 +140,7 @@ func (bs *BlockService) ExecutedCoreCommands(path string) []string {
|
||||
}
|
||||
|
||||
// CoreProjectionMismatches reports fields that should already agree on the
|
||||
// narrow Phase 15A path but do not. It intentionally excludes adapter-local
|
||||
// `PublishHealthy`, which is not yet rebound to the core publication owner.
|
||||
// current bounded core-present path but do not.
|
||||
func (bs *BlockService) CoreProjectionMismatches(path string) []string {
|
||||
proj, ok := bs.CoreProjection(path)
|
||||
if !ok {
|
||||
@@ -161,6 +160,9 @@ func (bs *BlockService) CoreProjectionMismatches(path string) []string {
|
||||
if readiness.ShipperConnected != proj.Readiness.ShipperConnected {
|
||||
mismatches = append(mismatches, "shipper_connected")
|
||||
}
|
||||
if readiness.PublishHealthy != proj.Publication.Healthy {
|
||||
mismatches = append(mismatches, "publish_healthy")
|
||||
}
|
||||
return mismatches
|
||||
}
|
||||
|
||||
@@ -963,6 +965,7 @@ func (bs *BlockService) CollectBlockVolumeHeartbeat() []blockvol.BlockVolumeInfo
|
||||
for i := range msgs {
|
||||
if s, ok := bs.replStates[msgs[i].Path]; ok {
|
||||
msgs[i].ReplicaDataAddr, msgs[i].ReplicaCtrlAddr = bs.heartbeatReplicaAddrs(msgs[i].Path, s)
|
||||
msgs[i].ReplicaReady = bs.heartbeatReplicaReady(msgs[i].Path, s)
|
||||
}
|
||||
msgs[i].ReplicaDegraded = bs.heartbeatReplicaDegraded(msgs[i].Path, msgs[i].ReplicaDegraded)
|
||||
// NVMe publication: report nvme_addr and nqn if NVMe target is running.
|
||||
@@ -1004,6 +1007,20 @@ func (bs *BlockService) heartbeatReplicaAddrs(path string, state *volReplState)
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// heartbeatReplicaReady returns the explicit replica-readiness truth that should
|
||||
// be exposed on the current heartbeat surface. On the core-present path it
|
||||
// prefers the core-owned readiness projection; older paths fall back to the
|
||||
// adapter-local publish gate for compatibility.
|
||||
func (bs *BlockService) heartbeatReplicaReady(path string, state *volReplState) bool {
|
||||
if state == nil {
|
||||
return false
|
||||
}
|
||||
if proj, ok := bs.CoreProjection(path); ok {
|
||||
return proj.Readiness.ReplicaReady
|
||||
}
|
||||
return state.publishHealthy
|
||||
}
|
||||
|
||||
// heartbeatReplicaDegraded returns the bounded degraded bit for the current
|
||||
// heartbeat surface. On the Phase 15 live path it prefers the core mode when
|
||||
// present, then falls back to the runtime-local status bit.
|
||||
@@ -1118,10 +1135,10 @@ func (bs *BlockService) markReceiverReady(path, dataAddr, ctrlAddr string) {
|
||||
state.allReplicas = nil
|
||||
}
|
||||
|
||||
// ReadinessSnapshot reports the service-owned assignment/readiness closure for
|
||||
// one volume. On the Phase 15 live path it prefers the explicit core projection
|
||||
// for the aligned readiness subset, while `PublishHealthy` remains adapter-local
|
||||
// until publication ownership is fully rebound.
|
||||
// ReadinessSnapshot reports the current server-boundary readiness/publication
|
||||
// closure for one volume. On the core-present path it prefers the explicit core
|
||||
// projection for aligned readiness plus publication health, while retaining
|
||||
// adapter-local fallback only when the core is absent.
|
||||
func (bs *BlockService) ReadinessSnapshot(path string) BlockReadinessSnapshot {
|
||||
snap := BlockReadinessSnapshot{}
|
||||
bs.replMu.RLock()
|
||||
@@ -1141,6 +1158,7 @@ func (bs *BlockService) ReadinessSnapshot(path string) BlockReadinessSnapshot {
|
||||
snap.ShipperConfigured = proj.Readiness.ShipperConfigured
|
||||
snap.ShipperConnected = proj.Readiness.ShipperConnected
|
||||
snap.ReplicaEligible = proj.Readiness.ReplicaReady
|
||||
snap.PublishHealthy = proj.Publication.Healthy
|
||||
}
|
||||
return snap
|
||||
}
|
||||
@@ -1154,6 +1172,7 @@ func (bs *BlockService) ReadinessSnapshot(path string) BlockReadinessSnapshot {
|
||||
snap.ShipperConfigured = proj.Readiness.ShipperConfigured
|
||||
snap.ShipperConnected = proj.Readiness.ShipperConnected
|
||||
snap.ReplicaEligible = proj.Readiness.ReplicaReady
|
||||
snap.PublishHealthy = proj.Publication.Healthy
|
||||
}
|
||||
return snap
|
||||
}
|
||||
|
||||
@@ -937,8 +937,8 @@ func TestBlockService_DebugInfoForVolume_UsesCoreProjectionPrimaryPath(t *testin
|
||||
}
|
||||
|
||||
readiness := bs.ReadinessSnapshot(path)
|
||||
if !readiness.PublishHealthy {
|
||||
t.Fatalf("expected adapter-local readiness to still report publish healthy, got %+v", readiness)
|
||||
if readiness.PublishHealthy {
|
||||
t.Fatalf("readiness snapshot must follow core publication truth on primary path without durable boundary, got %+v", readiness)
|
||||
}
|
||||
|
||||
vol, ok := bs.blockStore.GetBlockVolume(path)
|
||||
@@ -1064,6 +1064,9 @@ func TestBlockService_CollectBlockVolumeHeartbeat_PrimaryUsesCoreReadinessGate(t
|
||||
if msg.ReplicaCtrlAddr != "10.0.0.2:4261" {
|
||||
t.Fatalf("ReplicaCtrlAddr=%q", msg.ReplicaCtrlAddr)
|
||||
}
|
||||
if msg.ReplicaReady {
|
||||
t.Fatalf("primary heartbeat must not claim replica-ready truth, msg=%+v", msg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockService_CollectBlockVolumeHeartbeat_ReplicaUsesCoreReadinessGate(t *testing.T) {
|
||||
@@ -1117,6 +1120,9 @@ func TestBlockService_CollectBlockVolumeHeartbeat_ReplicaUsesCoreReadinessGate(t
|
||||
if msg.ReplicaCtrlAddr != expectedCtrl {
|
||||
t.Fatalf("ReplicaCtrlAddr=%q expected=%q", msg.ReplicaCtrlAddr, expectedCtrl)
|
||||
}
|
||||
if !msg.ReplicaReady {
|
||||
t.Fatalf("replica heartbeat should carry explicit replica-ready truth, msg=%+v", msg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockService_ReadinessSnapshot_PrefersCoreProjectionPrimaryFields(t *testing.T) {
|
||||
@@ -1157,7 +1163,47 @@ func TestBlockService_ReadinessSnapshot_PrefersCoreProjectionPrimaryFields(t *te
|
||||
t.Fatalf("expected snapshot shipper_configured from core projection, got %+v", snap)
|
||||
}
|
||||
if snap.PublishHealthy {
|
||||
t.Fatalf("publish_healthy should remain adapter-local on readiness snapshot, got %+v", snap)
|
||||
t.Fatalf("publish_healthy should follow core publication truth on primary path without durable boundary, got %+v", snap)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockService_ReadinessSnapshot_PrefersCorePublicationHealth(t *testing.T) {
|
||||
bs := newTestBlockServiceDirect(t)
|
||||
path := createTestVolDirect(t, bs, "vol-readiness-publish-healthy")
|
||||
|
||||
errs := bs.ApplyAssignments([]blockvol.BlockVolumeAssignment{
|
||||
{
|
||||
Path: path,
|
||||
Epoch: 1,
|
||||
Role: blockvol.RoleToWire(blockvol.RolePrimary),
|
||||
LeaseTtlMs: 30000,
|
||||
ReplicaServerID: "vs-2",
|
||||
ReplicaDataAddr: "10.0.0.2:4260",
|
||||
ReplicaCtrlAddr: "10.0.0.2:4261",
|
||||
},
|
||||
})
|
||||
if len(errs) != 1 || errs[0] != nil {
|
||||
t.Fatalf("apply assignment errs=%v", errs)
|
||||
}
|
||||
|
||||
bs.applyCoreEvent(engine.ShipperConnectedObserved{ID: path})
|
||||
bs.applyCoreEvent(engine.BarrierAccepted{ID: path, FlushedLSN: 12})
|
||||
|
||||
bs.replMu.Lock()
|
||||
state := bs.replStates[path]
|
||||
if state == nil {
|
||||
bs.replMu.Unlock()
|
||||
t.Fatal("missing repl state")
|
||||
}
|
||||
state.publishHealthy = false
|
||||
bs.replMu.Unlock()
|
||||
|
||||
snap := bs.ReadinessSnapshot(path)
|
||||
if !snap.PublishHealthy {
|
||||
t.Fatalf("publish_healthy should follow core publication truth once healthy, got %+v", snap)
|
||||
}
|
||||
if mismatches := bs.CoreProjectionMismatches(path); len(mismatches) != 0 {
|
||||
t.Fatalf("readiness/core mismatches=%v", mismatches)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1198,7 +1244,7 @@ func TestBlockService_ReadinessSnapshot_PrefersCoreProjectionReplicaFields(t *te
|
||||
t.Fatalf("expected snapshot replica_eligible from core projection, got %+v", snap)
|
||||
}
|
||||
if snap.PublishHealthy {
|
||||
t.Fatalf("publish_healthy should remain adapter-local on readiness snapshot, got %+v", snap)
|
||||
t.Fatalf("publish_healthy should follow core publication truth on replica path, got %+v", snap)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -8,22 +8,23 @@ import (
|
||||
// BlockVolumeInfoMessage is the heartbeat status for one block volume.
|
||||
// Mirrors the proto message that will be generated from master.proto.
|
||||
type BlockVolumeInfoMessage struct {
|
||||
Path string // volume file path (unique ID on this server)
|
||||
VolumeSize uint64 // logical size in bytes
|
||||
BlockSize uint32 // block size in bytes
|
||||
Epoch uint64 // current fencing epoch
|
||||
Role uint32 // blockvol.Role as uint32 for wire compat
|
||||
WalHeadLsn uint64 // WAL head LSN
|
||||
CheckpointLsn uint64 // last flushed LSN
|
||||
HasLease bool // whether volume holds a valid lease
|
||||
DiskType string // e.g., "ssd", "hdd"
|
||||
ReplicaDataAddr string // receiver data listen addr (VS reports in heartbeat)
|
||||
ReplicaCtrlAddr string // receiver ctrl listen addr
|
||||
HealthScore float64 // CP8-2: 0.0-1.0
|
||||
ScrubErrors int64 // CP8-2: lifetime scrub error count
|
||||
LastScrubTime int64 // CP8-2: unix seconds
|
||||
ReplicaDegraded bool // CP8-2: true if any replica shipper degraded
|
||||
DurabilityMode string // CP8-3-1: "best_effort", "sync_all", "sync_quorum"
|
||||
Path string // volume file path (unique ID on this server)
|
||||
VolumeSize uint64 // logical size in bytes
|
||||
BlockSize uint32 // block size in bytes
|
||||
Epoch uint64 // current fencing epoch
|
||||
Role uint32 // blockvol.Role as uint32 for wire compat
|
||||
WalHeadLsn uint64 // WAL head LSN
|
||||
CheckpointLsn uint64 // last flushed LSN
|
||||
HasLease bool // whether volume holds a valid lease
|
||||
DiskType string // e.g., "ssd", "hdd"
|
||||
ReplicaDataAddr string // receiver data listen addr (VS reports in heartbeat)
|
||||
ReplicaCtrlAddr string // receiver ctrl listen addr
|
||||
ReplicaReady bool // explicit replica readiness truth on the heartbeat seam
|
||||
HealthScore float64 // CP8-2: 0.0-1.0
|
||||
ScrubErrors int64 // CP8-2: lifetime scrub error count
|
||||
LastScrubTime int64 // CP8-2: unix seconds
|
||||
ReplicaDegraded bool // CP8-2: true if any replica shipper degraded
|
||||
DurabilityMode string // CP8-3-1: "best_effort", "sync_all", "sync_quorum"
|
||||
NvmeAddr string // NVMe/TCP target address (ip:port), empty if NVMe disabled
|
||||
NQN string // NVMe subsystem NQN, empty if NVMe disabled
|
||||
ReplicaShipperStates []ReplicaShipperStatus // CP13-7: per-replica state from primary's shipper group
|
||||
@@ -60,18 +61,18 @@ func ToBlockVolumeInfoMessage(path, diskType string, vol *BlockVol) BlockVolumeI
|
||||
status := vol.Status()
|
||||
hs := vol.HealthStats()
|
||||
return BlockVolumeInfoMessage{
|
||||
Path: path,
|
||||
VolumeSize: info.VolumeSize,
|
||||
BlockSize: info.BlockSize,
|
||||
Epoch: status.Epoch,
|
||||
Role: RoleToWire(status.Role),
|
||||
WalHeadLsn: status.WALHeadLSN,
|
||||
CheckpointLsn: status.CheckpointLSN,
|
||||
HasLease: status.HasLease,
|
||||
DiskType: diskType,
|
||||
HealthScore: status.HealthScore,
|
||||
ScrubErrors: hs.ScrubErrors,
|
||||
LastScrubTime: hs.LastScrubTime,
|
||||
Path: path,
|
||||
VolumeSize: info.VolumeSize,
|
||||
BlockSize: info.BlockSize,
|
||||
Epoch: status.Epoch,
|
||||
Role: RoleToWire(status.Role),
|
||||
WalHeadLsn: status.WALHeadLSN,
|
||||
CheckpointLsn: status.CheckpointLSN,
|
||||
HasLease: status.HasLease,
|
||||
DiskType: diskType,
|
||||
HealthScore: status.HealthScore,
|
||||
ScrubErrors: hs.ScrubErrors,
|
||||
LastScrubTime: hs.LastScrubTime,
|
||||
ReplicaDegraded: status.ReplicaDegraded,
|
||||
DurabilityMode: vol.DurabilityMode().String(),
|
||||
ReplicaShipperStates: vol.ReplicaShipperStates(),
|
||||
|
||||
@@ -18,6 +18,7 @@ func InfoMessageToProto(m BlockVolumeInfoMessage) *master_pb.BlockVolumeInfoMess
|
||||
DiskType: m.DiskType,
|
||||
ReplicaDataAddr: m.ReplicaDataAddr,
|
||||
ReplicaCtrlAddr: m.ReplicaCtrlAddr,
|
||||
ReplicaReady: &m.ReplicaReady,
|
||||
HealthScore: m.HealthScore,
|
||||
ScrubErrors: m.ScrubErrors,
|
||||
LastScrubTime: m.LastScrubTime,
|
||||
@@ -45,6 +46,7 @@ func InfoMessageFromProto(p *master_pb.BlockVolumeInfoMessage) BlockVolumeInfoMe
|
||||
DiskType: p.DiskType,
|
||||
ReplicaDataAddr: p.ReplicaDataAddr,
|
||||
ReplicaCtrlAddr: p.ReplicaCtrlAddr,
|
||||
ReplicaReady: p.GetReplicaReady(),
|
||||
HealthScore: p.HealthScore,
|
||||
ScrubErrors: p.ScrubErrors,
|
||||
LastScrubTime: p.LastScrubTime,
|
||||
@@ -166,4 +168,3 @@ func AssignmentsFromProto(protos []*master_pb.BlockVolumeAssignment) []BlockVolu
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
@@ -205,12 +205,12 @@ func TestNilProtoConversions(t *testing.T) {
|
||||
|
||||
func TestInfoMessage_HealthScoreRoundTrip(t *testing.T) {
|
||||
orig := BlockVolumeInfoMessage{
|
||||
Path: "/data/vol.blk",
|
||||
VolumeSize: 1 << 30,
|
||||
BlockSize: 4096,
|
||||
Epoch: 5,
|
||||
HealthScore: 0.85,
|
||||
ScrubErrors: 3,
|
||||
Path: "/data/vol.blk",
|
||||
VolumeSize: 1 << 30,
|
||||
BlockSize: 4096,
|
||||
Epoch: 5,
|
||||
HealthScore: 0.85,
|
||||
ScrubErrors: 3,
|
||||
LastScrubTime: 1709000000,
|
||||
}
|
||||
pb := InfoMessageToProto(orig)
|
||||
@@ -247,6 +247,26 @@ func TestInfoMessage_ReplicaDegradedRoundTrip(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestInfoMessage_ReplicaReadyRoundTrip(t *testing.T) {
|
||||
orig := BlockVolumeInfoMessage{
|
||||
Path: "/data/vol.blk",
|
||||
Epoch: 1,
|
||||
ReplicaReady: true,
|
||||
}
|
||||
pb := InfoMessageToProto(orig)
|
||||
back := InfoMessageFromProto(pb)
|
||||
if !back.ReplicaReady {
|
||||
t.Fatal("ReplicaReady should be true after round-trip")
|
||||
}
|
||||
|
||||
orig.ReplicaReady = false
|
||||
pb = InfoMessageToProto(orig)
|
||||
back = InfoMessageFromProto(pb)
|
||||
if back.ReplicaReady {
|
||||
t.Fatal("ReplicaReady should be false after round-trip")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssignment_MultiReplicaRoundTrip(t *testing.T) {
|
||||
orig := BlockVolumeAssignment{
|
||||
Path: "/data/vol.blk",
|
||||
|
||||
Reference in New Issue
Block a user