Ping QiuandClaude Opus 4.6 8b2b5f6f66 feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests ...

Wire low-level fencing primitives to master/VS control plane and CSI:

- Proto: replica/rebuild address fields on assignment/info/response messages
- Assignment queue: retain-until-confirmed (Peek+Confirm), stale epoch pruning
- VS assignment receiver: processes assignments from HeartbeatResponse
- BlockService replication: ProcessAssignments, deterministic ports (FNV hash)
- Registry replica tracking: SetReplica/ClearReplica/SwapPrimaryReplica
- CreateBlockVolume: primary + replica, enqueues assignments, single-copy mode
- Failover: lease-aware promotion, deferred timers with cancellation on reconnect
- ControllerPublish: returns fresh primary iSCSI address after failover
- Recovery: recoverBlockVolumes drains pendingRebuilds, enqueues Rebuilding
- Real integration tests on M02: failover address switch, rebuild data
  consistency, full lifecycle failover+rebuild (3 tests, all PASS)

Review fixes (12 findings, 5 High, 5 Medium, 2 Low):
- R1-1: AllocateBlockVolume returns replication ports
- R1-2: setupPrimaryReplication starts rebuild server
- R1-3: VS sends periodic block heartbeat for assignment confirmation
- R2-F1: LastLeaseGrant set before Register (no stale-lease race)
- R2-F2: Deferred promotion timers cancelled on VS reconnect
- R2-F3: SwapPrimaryReplica uses RoleToWire instead of uint32(1)
- R2-F4: DeleteBlockVolume deletes replica (best-effort)
- R2-F5: SwapPrimaryReplica computes epoch atomically under lock
- QA: SetReplica removes old replica from byServer index (BUG-QA-CP63-1)

126 CP6-3 tests (67 dev + 48 QA + 8 integration + 3 real).
Cumulative Phase 6: 352 tests. All PASS.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-05 00:52:05 -08:00

..

constants

Nit: use time.Durations instead of constants in seconds. (#7438)

2025-11-04 13:02:22 -08:00

filer_ui

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

master_ui

fix format of internal node URLs in master UI templates

2026-02-22 13:47:29 -08:00

postgres

Move SQL engine and PostgreSQL server to their own binaries (#8417)

2026-02-23 16:27:08 -08:00

volume_server_ui

fix: EC UI template error when viewing shard details (#7955)

2026-01-03 22:45:48 -08:00

block_heartbeat_loop_test.go

feat: Phase 4A CP4b-3 -- assignment processing, 2 bug fixes, 20 QA tests

2026-03-02 11:34:06 -08:00

block_heartbeat_loop.go

feat: Phase 4A CP4b-3 -- assignment processing, 2 bug fixes, 20 QA tests

2026-03-02 11:34:06 -08:00

common_test.go

jwt check the base file id

2019-03-03 10:17:44 -08:00

common.go

fix: Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition (#7635)

2025-12-05 15:59:12 -08:00

filer_grpc_server_admin.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filer_grpc_server_dlm.go

reduce logs

2025-12-26 13:26:25 -08:00

filer_grpc_server_kv.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filer_grpc_server_remote.go

s3: fix remote object not caching (#7790)

2025-12-16 12:41:04 -08:00

filer_grpc_server_rename.go

Prevent bucket renaming in filer, fuse mount, and S3 (#8048)

2026-01-16 19:48:09 -08:00

filer_grpc_server_sub_meta.go

avoid repeated reading disk (#7369)

2025-10-24 01:26:42 -07:00

filer_grpc_server_traverse_meta_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filer_grpc_server_traverse_meta.go

Add error list each entry func (#7485)

2025-11-25 19:35:19 -08:00

filer_grpc_server.go

filer: preserve existing TTL during CreateEntry/UpdateEntry gRPC calls

2026-01-29 14:42:09 -08:00

filer_jwt_test.go

fix Filer startup failure due to JWT on / path #8149 (#8167)

2026-01-29 21:45:15 -08:00

filer_server_handlers_copy.go

use one http client

2025-08-11 16:45:13 -07:00

filer_server_handlers_iam_grpc.go

Implement IAM propagation to S3 servers (#8130)

2026-01-26 22:59:43 -08:00

filer_server_handlers_proxy.go

Changes logging function (#6919)

2025-06-24 08:44:06 -07:00

filer_server_handlers_read_dir.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filer_server_handlers_read.go

S3: set identity to request context, and remove obsolete code (#7523)

2025-11-21 14:46:32 -08:00

filer_server_handlers_tagging.go

Changes logging function (#6919)

2025-06-24 08:44:06 -07:00

filer_server_handlers_write_autochunk.go

filer write request use context without cancellation (#7567)

2025-11-28 11:52:57 -08:00

filer_server_handlers_write_cipher.go

convert error fromating to %w everywhere (#6995)

2025-07-16 23:39:27 -07:00

filer_server_handlers_write_merge.go

S3 API: Add SSE-KMS (#7144)

2025-08-21 08:28:07 -07:00

filer_server_handlers_write_upload.go

fix: multipart upload ETag calculation (#8238)

2026-02-06 21:54:43 -08:00

filer_server_handlers_write.go

Reduce memory allocations in hot paths (#7725)

2025-12-12 12:51:48 -08:00

filer_server_handlers.go

fix Filer startup failure due to JWT on / path #8149 (#8167)

2026-01-29 21:45:15 -08:00

filer_server_rocksdb.go

go fix

2026-02-20 18:42:00 -08:00

filer_server_tus_handlers.go

Add TUS protocol support for resumable uploads (#7592)

2025-12-14 21:56:07 -08:00

filer_server_tus_session.go

Add TUS protocol support for resumable uploads (#7592)

2025-12-14 21:56:07 -08:00

filer_server.go

Fix: filer not yet available in s3.configure (#8198)

2026-02-03 17:43:58 -08:00

integration_block_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_block_assignment_queue_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_block_assignment_queue.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_block_failover_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_block_failover.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_block_registry_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_block_registry.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_grpc_server_admin.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

master_grpc_server_assign.go

Fix Maintenance Task Sorting and Refactor Log Persistence (#8199)

2026-02-04 08:48:55 -08:00

master_grpc_server_block_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_grpc_server_block.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_grpc_server_cluster.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

master_grpc_server_collection.go

move to https://github.com/seaweedfs/seaweedfs

2022-07-29 00:17:28 -07:00

master_grpc_server_raft_test.go

Add cluster.raft.leader.transfer command for graceful leader change (#7819)

2025-12-19 00:15:39 -08:00

master_grpc_server_raft.go

fix: admin does not show all master servers #7999 (#8002)

2026-01-11 12:31:46 -08:00

master_grpc_server_volume.go

Fix imbalance detection disk type grouping and volume grow errors (#8097)

2026-01-23 12:25:11 -08:00

master_grpc_server.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

master_server_handlers_admin.go

fix: generate topology uuid uniformly in single-master mode (#8405)

2026-02-22 23:45:48 -08:00

master_server_handlers_ui.go

hide millseconds in up time (#7553)

2025-11-26 08:01:19 -08:00

master_server_handlers.go

add disable volume_growth flag (#7196)

2025-09-04 05:39:56 -07:00

master_server.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

qa_block_assign_test.go

feat: Phase 4A CP4b-3 -- assignment processing, 2 bug fixes, 20 QA tests

2026-03-02 11:34:06 -08:00

qa_block_cp62_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

qa_block_cp63_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

qa_block_heartbeat_loop_test.go

feat: Phase 4A CP4b-3 -- assignment processing, 2 bug fixes, 20 QA tests

2026-03-02 11:34:06 -08:00

raft_common.go

Prevent split-brain: Persistent ClusterID and Join Validation (#8022)

2026-01-18 14:02:34 -08:00

raft_hashicorp_test.go

Normalize hashicorp raft peer ids (#8253)

2026-02-09 07:46:34 -08:00

raft_hashicorp.go

Normalize hashicorp raft peer ids (#8253)

2026-02-09 07:46:34 -08:00

raft_server_handlers.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

raft_server.go

s3api: add AttachUserPolicy/DetachUserPolicy/ListAttachedUserPolicies (#8379)

2026-02-19 12:26:27 -08:00

volume_grpc_admin.go

Add a version token on RPCs to read/update volume server states. (#8191)

2026-02-06 10:58:43 -08:00

volume_grpc_batch_delete.go

Block RPC write operations on volume servers when maintenance mode is enabled (#8115)

2026-02-02 13:21:02 -08:00

volume_grpc_block_test.go

feat: Phase 6 CP6-2 -- CSI control-plane integration + csi-sanity/k3s validation

2026-03-04 11:01:08 -08:00

volume_grpc_block.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

volume_grpc_client_to_master.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

volume_grpc_copy_incremental.go

move to https://github.com/seaweedfs/seaweedfs

2022-07-29 00:17:28 -07:00

volume_grpc_copy.go

Fix live volume move tail timestamp (#8440)

2026-02-24 20:07:26 -08:00

volume_grpc_erasure_coding_test.go

iceberg: wire pagination for list namespaces/tables REST APIs (#8275)

2026-02-09 21:46:55 -08:00

volume_grpc_erasure_coding.go

Fix EC rebuild shard detection (#8265)

2026-02-09 12:34:38 -08:00

volume_grpc_query.go

move to https://github.com/seaweedfs/seaweedfs

2022-07-29 00:17:28 -07:00

volume_grpc_read_all.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

volume_grpc_read_write.go

Block RPC write operations on volume servers when maintenance mode is enabled (#8115)

2026-02-02 13:21:02 -08:00

volume_grpc_remote.go

Block RPC write operations on volume servers when maintenance mode is enabled (#8115)

2026-02-02 13:21:02 -08:00

volume_grpc_scrub.go

Implement full scrubbing for EC volumes (#8318)

2026-02-16 15:09:01 -08:00

volume_grpc_state.go

Add a version token on RPCs to read/update volume server states. (#8191)

2026-02-06 10:58:43 -08:00

volume_grpc_tail.go

Block RPC write operations on volume servers when maintenance mode is enabled (#8115)

2026-02-02 13:21:02 -08:00

volume_grpc_tier_download.go

avoid load volume file with BytesOffset mismatch (#3841)

2022-10-14 00:18:09 -07:00

volume_grpc_tier_upload.go

Block RPC write operations on volume servers when maintenance mode is enabled (#8115)

2026-02-02 13:21:02 -08:00

volume_grpc_vacuum.go

Block RPC write operations on volume servers when maintenance mode is enabled (#8115)

2026-02-02 13:21:02 -08:00

volume_server_block_test.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

volume_server_block.go

feat: Phase 6 CP6-3 -- failover + rebuild in Kubernetes, 126 tests

2026-03-05 00:52:05 -08:00

volume_server_handlers_admin.go

fix: volume server healthz now checks local conditions only (#7610)

2025-12-02 23:19:14 -08:00

volume_server_handlers_helper.go

directory structure change to work with glide

2016-06-02 18:09:14 -07:00

volume_server_handlers_read.go

fix: Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition (#7635)

2025-12-05 15:59:12 -08:00

volume_server_handlers_ui.go

hide millseconds in up time (#7553)

2025-11-26 08:01:19 -08:00

volume_server_handlers_write.go

fix: multipart upload ETag calculation (#8238)

2026-02-06 21:54:43 -08:00

volume_server_handlers.go

fix: JWT validation failures during replication (#7788) (#7795)

2025-12-16 13:42:18 -08:00

volume_server_test.go

Add a version token on RPCs to read/update volume server states. (#8191)

2026-02-06 10:58:43 -08:00

volume_server.go

feat: Phase 6 CP6-2 -- CSI control-plane integration + csi-sanity/k3s validation

2026-03-04 11:01:08 -08:00

webdav_server.go

mount: improve read throughput with parallel chunk fetching (#7569)

2025-11-29 10:06:11 -08:00

wrapped_webdav_fs.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00