feat: Phase 11 — product-surface rebinding (snapshot, CSI, publication, restore)

P1 Snapshots: CoW snapshot lifecycle through V2 engine path, create/list/delete
  via master RPC, BaseLSN tracking in manifest, ImportSnapshotForRebuild

P2 CSI Lifecycle: masterServerBackend calling real MasterServer in-process,
  CreateVolume/DeleteVolume/ExpandVolume through CSI → master → VS flow,
  ExportedControllerServer/ExportedNodeServer for cross-package testing

P3 Publication: LookupBlockVolume coherence across failover, iSCSI + NVMe
  address switching on promotion, repeated lookup self-consistency

P4 Restore: RestoreBlockSnapshot RPC through master and volume server,
  snapshot restore with runtime convergence, epoch/role validation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pingqiu
2026-04-02 16:25:58 -07:00
co-authored by Claude Opus 4.6
parent 3ec8fab2f1
commit bdf83e350e
7 changed files with 3249 additions and 0 deletions
File diff suppressed because it is too large Load Diff
+488
View File
@@ -0,0 +1,488 @@
# Phase 11
Date: 2026-04-02
Status: complete
Purpose: bind selected product-facing surfaces onto the accepted V2-backed chosen path without reopening accepted backend execution or control-plane closure
## Why This Phase Exists
`Phase 09` accepted production-grade execution closure on the chosen path.
`Phase 10` accepted bounded master-driven control-plane closure on that same path.
What remains is no longer:
1. whether the chosen backend path executes correctly
2. whether accepted control truth can reach the live volume-server path coherently
It is now:
1. whether selected product-facing surfaces can be rebound onto that accepted path without semantic drift
2. whether reuse of older V1-facing adapters reintroduces V1 recovery truth implicitly
3. whether the first product-facing surface can be proven in a bounded way before broader surface expansion
## Phase Goal
Move from accepted backend/control closure on one bounded chosen path to the first bounded product-surface rebinding proof.
Execution note:
1. treat `P0` as real planning work, not placeholder prose
2. use `phase-11-log.md` as the technical pack for:
- step breakdown
- hard indicators
- reject shapes
- assignment text for `sw` and `tester`
## Scope
### In scope
1. one bounded first product-surface slice
2. explicit no-overclaim around what that first surface proves and does not prove
3. reuse of existing implementation only where V2 truth still owns placement, recovery, and correctness claims
4. focused integration tests and contract checks for the chosen first surface
### Out of scope
1. reopening accepted `Phase 09` execution semantics
2. reopening accepted `Phase 10` control-plane closure
3. broad multi-surface product completion in one slice
4. `RF>2`, new durability modes, or broad cluster hardening
5. full production readiness / soak / rollout gates
## Phase 11 Items
### P0: First Surface Selection
Goal:
- choose the first product-facing surface that gives real product completion movement without turning the phase into a multi-system rewrite
Accepted decision:
1. the first bounded slice is `snapshot product path`
2. `CSI` is deferred to a later `Phase 11` slice because it pulls controller/node lifecycle, staging/publish, and broader cluster contract surface
3. `NVMe` / `iSCSI` rebinding are also deferred because they are transport/front-end adapters whose useful proof should come after one simpler product surface is already closed
Why this first:
1. snapshot is closest to already accepted backend truth
2. it exercises a real product-facing contract without immediately absorbing node/attach orchestration
3. it keeps the first `Phase 11` slice bounded to metadata/visibility/restore-contract correctness rather than transport and lifecycle breadth
Status:
- accepted
### P1: Snapshot Product-Path Rebinding
Goal:
- prove that the snapshot product path can be rebound onto the accepted V2-backed chosen path without semantic drift between snapshot-visible behavior and the accepted backend snapshot truth
Execution steps:
1. Step 1: contract freeze
- define exactly what the first slice claims:
- snapshot create
- snapshot list
- snapshot delete
- explicitly exclude clone/restore unless a later slice accepts them
2. Step 2: implementation binding
- bind product-visible snapshot operations onto the accepted backend snapshot path
- keep master/volume-server state and visible metadata coherent
3. Step 3: proof package
- prove create/list/delete on the chosen path
- prove fail-closed behavior for unsupported/invalid inputs
- prove no-overclaim around broader snapshot workflows
Required scope:
1. snapshot create/list/delete product-visible behavior on the chosen path
2. proof that snapshot metadata and visible snapshot set reflect the same accepted backend truth
3. proof that snapshot claims do not exceed the accepted V2 snapshot contract
4. explicit boundedness around restore/clone if they are not part of the first slice
Must prove:
1. snapshot creation on the product path maps to the accepted backend snapshot boundary rather than an implicit V1 truth
2. listing and deletion reflect the real volume-server/master state coherently
3. fail-closed behavior is preserved when snapshot prerequisites are missing or the volume is not eligible
4. the slice does not silently imply clone/restore/product workflow support that is not yet proven
Reuse discipline:
1. V1/master-facing snapshot RPC surface may be reused only as a product wrapper:
- `CreateBlockSnapshot`
- `DeleteBlockSnapshot`
- `ListBlockSnapshots`
2. V1/volume-server-facing snapshot surface may be reused only as the bounded execution adapter:
- `SnapshotBlockVol`
- `DeleteBlockSnapshot`
- `ListBlockSnapshots`
3. underlying `blockvol` snapshot implementation may be reused as execution reality, not as product truth ownership
4. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
- `update in place`
- `reference only`
- `reuse as bounded adapter`
5. no reused V1 surface may silently redefine snapshot semantics, placement truth, or product support claims
Verification mechanism:
1. focused integration tests for create/list/delete on the chosen path
2. contract checks that visible snapshot metadata matches the accepted backend snapshot truth
3. no-overclaim review on what user-visible snapshot behavior is actually supported after the slice
Hard indicators:
1. one accepted create proof:
- product-visible create succeeds on the chosen path
- created snapshot is observable through list/readback metadata
2. one accepted delete proof:
- deleted snapshot disappears from the visible snapshot set
- repeated delete is either idempotent-success or explicitly fail-closed as designed
3. one accepted list coherence proof:
- listed snapshot IDs/metadata match the real backend snapshot state
4. one accepted fail-closed proof:
- invalid volume / missing snapshot / unsupported preconditions do not imply false success
5. one accepted boundedness proof:
- docs/tests do not imply clone/restore/full snapshot workflow readiness unless separately proven
6. one accepted reuse-boundary proof:
- all V1 reuse surfaces touched by the slice are explicitly listed and their role is bounded
Reject if:
1. the slice proves only local helper behavior rather than product-visible snapshot behavior
2. visible snapshot metadata can drift from backend truth
3. the first slice quietly absorbs clone/restore or broader workflow work
4. the slice claims product readiness beyond create/list/delete on the chosen path
5. reuse of V1 surfaces is implicit or lets V1 semantics become the source of truth
Status:
- accepted
Carry-forward from `P1`:
1. bounded snapshot create/list/delete product rebinding is now accepted on the chosen path
2. `P1` does not claim restore/clone/full snapshot workflow readiness
3. `CSI` rebinding is now the next active `Phase 11` slice
### Later candidate slices inside `Phase 11`
1. `P2`: `CSI` rebinding after snapshot product-path closure
2. `P3`: `NVMe` / `iSCSI` front-end rebinding after one simpler product-visible surface is already accepted
3. `P4`: broader snapshot workflow closure (`restore` / `clone`) or other residual product workflow work only after earlier slices are bounded and proven
### P2: CSI Rebinding
Goal:
- bind the accepted V2-backed chosen path to the `CSI` controller/node product surface without reintroducing V1 recovery truth
Execution steps:
1. Step 1: contract freeze
- define the first bounded `CSI` surface claims:
- `CreateVolume`
- `DeleteVolume`
- `ControllerPublishVolume`
- `NodeStageVolume`
- `NodePublishVolume`
- `NodeUnpublishVolume`
- `NodeUnstageVolume`
- explicitly exclude CSI snapshot, expand, and NVMe-specific transport work unless a later slice accepts them
2. Step 2: backend rebinding
- bind CSI controller operations to the accepted master-backed chosen-path volume surface
- bind CSI node operations to the accepted chosen-path access contract for remote attach/stage/publish
3. Step 3: proof package
- prove bounded create/publish/stage/use/delete lifecycle on the chosen path
- prove fail-closed behavior for unsupported or invalid cases
- prove no-overclaim around broader CSI/product workflow breadth
Required scope:
1. bounded CSI controller/node lifecycle on the chosen path
2. explicit separation between accepted backend/control truth and CSI orchestration wrappers
3. remote target publication/staging behavior for the chosen path
4. no-overclaim around snapshots via CSI, expand, NVMe transport preference, multi-node topology breadth, or broad K8s readiness
Must prove:
1. CSI controller create/delete/publish map to the accepted master-backed chosen-path truth rather than a local V1 shortcut
2. CSI node stage/publish/unstage/unpublish consume the same chosen-path access truth without redefining recovery semantics
3. product-visible CSI lifecycle behavior is coherent across controller and node surfaces
4. fail-closed behavior is preserved when required publish/volume context or target information is missing
Reuse discipline:
1. V1/CSI-facing controller and node RPC surfaces may be reused only as bounded product adapters:
- `controller.go`
- `node.go`
- `server.go`
2. `volume_backend.go` may be reused only as the bounded bridge between CSI and accepted master/local surfaces
3. `volume_manager.go` may be reused only as bounded local execution reality where the slice explicitly proves that local manager behavior does not become semantic owner
4. accepted master block RPC surfaces may be reused only as bounded control/product adapters underneath the CSI backend bridge:
- `CreateBlockVolume`
- `DeleteBlockVolume`
- `LookupBlockVolume`
5. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
- `update in place`
- `reference only`
- `reuse as bounded adapter`
- `reuse as bounded bridge`
- `reuse as execution reality only`
6. no reused V1 surface may silently redefine lifecycle semantics, placement truth, or product support claims
Verification mechanism:
1. focused CSI controller/node integration tests on the chosen path
2. contract checks that controller-visible and node-visible truth match accepted backend/control truth
3. no-overclaim review on what CSI behavior is actually supported after the slice
Hard indicators:
1. one accepted controller create/publish proof:
- CSI create returns coherent volume/publish context on the chosen path
2. one accepted node stage/publish proof:
- node consumes the published target info and stages/publishes coherently on the chosen path
3. one accepted unpublish/unstage/delete proof:
- teardown/deletion complete without leaving false-visible ownership
4. one accepted fail-closed proof:
- missing or partial transport/context information does not imply false success
5. one accepted reuse-boundary proof:
- all CSI/V1 reuse surfaces touched by the slice are explicitly listed and bounded
6. one accepted boundedness proof:
- docs/tests do not imply CSI snapshot, expand, NVMe transport preference, or broad K8s/product readiness unless separately proven
Reject if:
1. the slice proves only CSI wrapper-local behavior without chosen-path backend/control coherence
2. controller truth and node truth can drift from accepted master-backed volume truth
3. the first CSI slice quietly absorbs snapshot, expand, NVMe, or broad multi-node/K8s readiness work
4. reuse of V1 surfaces is implicit or lets V1 semantics become the source of truth
Status:
- accepted
Carry-forward from `P2`:
1. bounded CSI controller/node lifecycle rebinding is now accepted on the chosen path
2. accepted proof uses the real master-backed create/lookup/delete path plus `mgr=nil` node consumption of published target truth
3. `P2` does not claim CSI snapshot, CSI expand, NVMe preference/failover closure, or broad Kubernetes readiness
### P3: NVMe / iSCSI Front-End Rebinding
Goal:
- bind transport/front-end publication surfaces onto the accepted V2-backed chosen path so the product-visible access path matches accepted backend/control truth
Execution steps:
1. Step 1: contract freeze
- define the first bounded front-end publication claims:
- create returns coherent front-end publication data
- lookup returns coherent front-end publication data
- heartbeat refresh preserves and updates publication truth
- failover switches publication truth to the new primary coherently
- explicitly exclude broad transport-performance claims, real initiator benchmarking, and broad cluster rollout readiness
2. Step 2: publication rebinding
- bind `iSCSI` and `NVMe` publication fields onto the accepted master-backed chosen-path truth
- keep registry-visible, lookup-visible, and CSI-visible publication truth coherent
3. Step 3: proof package
- prove bounded create/lookup/failover/restart publication truth on the chosen path
- prove fallback behavior is explicit where `NVMe` is absent
- prove no-overclaim around full transport runtime/performance closure
Required scope:
1. publication/address/naming truth for front-end adapters on the chosen path
2. bounded integration proof that master-visible and product-visible access metadata stay coherent
3. `NVMe` primary publication and `iSCSI` fallback publication where supported by the chosen path
4. explicit boundedness around real initiator behavior, transport performance, and broad cluster hardening
Must prove:
1. create/lookup publication fields map to accepted chosen-path truth rather than ad hoc wrapper-local construction
2. heartbeat refresh and failover preserve or update front-end publication truth coherently
3. `NVMe` and `iSCSI` publication fields do not drift between registry, lookup, and product-facing responses
4. mixed-capability or fallback behavior is explicit rather than silently overclaimed
Reuse discipline:
1. master-facing product/control publication surfaces may be reused only as bounded adapters:
- `CreateBlockVolume`
- `LookupBlockVolume`
2. registry publication fields may be reused only as bounded truth carriers, not independent semantic owners:
- `ISCSIAddr`
- `IQN`
- `NvmeAddr`
- `NQN`
3. volume-server allocation/publication surfaces may be reused only as bounded front-end publication sources:
- `AllocateBlockVolume`
- block heartbeat publication of `NvmeAddr` / `NQN`
4. existing `CSI` controller consumption of publication fields may be reused only as a bounded downstream consumer, not as the source of truth for `P3`
5. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
- `update in place`
- `reference only`
- `reuse as bounded adapter`
- `reuse as bounded truth carrier`
- `reuse as publication source only`
6. no reused V1 surface may silently redefine publication truth, failover truth, or supported transport claims
Verification mechanism:
1. focused integration tests for create/lookup publication truth on the chosen path
2. contract checks that registry-visible, lookup-visible, and consumer-visible publication fields match
3. failover/restart checks that front-end publication truth is reconstructed or updated coherently
4. no-overclaim review on what transport/front-end behavior is actually supported after the slice
Hard indicators:
1. one accepted create/lookup publication proof:
- create returns coherent front-end publication fields
- lookup returns the same chosen-path publication truth
2. one accepted failover publication proof:
- front-end publication fields move to the new primary coherently after failover
3. one accepted restart/heartbeat reconstruction proof:
- publication fields can be reconstructed or refreshed from accepted heartbeat truth
4. one accepted fallback proof:
- `iSCSI` fallback or mixed-capability behavior is explicit and coherent when `NVMe` is absent
5. one accepted reuse-boundary proof:
- all front-end publication surfaces touched by the slice are explicitly listed and bounded
6. one accepted boundedness proof:
- docs/tests do not imply real transport runtime, performance leadership, or broad production readiness unless separately proven
Reject if:
1. the slice proves only field plumbing without chosen-path publication coherence
2. publication truth can drift across create, lookup, heartbeat, or failover
3. the slice quietly absorbs full transport runtime or performance claims
4. reuse of V1/publication surfaces is implicit or lets wrappers become the truth owner
Status:
- accepted
Carry-forward from `P3`:
1. bounded front-end publication/address truth rebinding is now accepted on the chosen path
2. accepted proof closes create/lookup coherence, failover publication switch, heartbeat reconstruction, and no-`NVMe` fallback
3. `P3` does not claim full initiator/runtime transport proof, performance claims, or broad production readiness
### P4: Broader Product Workflow Closure
Goal:
- close the remaining bounded snapshot product workflow gaps downstream of accepted `P1` / `P2` / `P3` without reopening earlier accepted truth
Execution steps:
1. Step 1: contract freeze
- define the first bounded `P4` workflow claim as snapshot `restore`
- explicitly defer `clone` unless and until a real product-facing clone surface exists and is accepted into scope
2. Step 2: workflow rebinding
- bind product-visible restore behavior onto the accepted snapshot and chosen-path execution truth
- keep restore-visible state coherent across master-visible and volume-server/backend-visible truth
3. Step 3: proof package
- prove bounded restore success, destructive semantics, and post-restore visible truth
- prove fail-closed behavior for missing snapshot or unsupported conditions
- prove no-overclaim around clone or broader workflow productization
Required scope:
1. bounded snapshot restore product workflow on the chosen path
2. explicit proof that restore uses accepted snapshot/backend truth rather than reopening new execution ownership
3. explicit post-restore visible truth checks
4. explicit boundedness around `clone` and any broader workflow work
Must prove:
1. product-visible restore maps to accepted backend restore execution truth on the chosen path
2. restore-visible outcome matches the selected snapshot truth after the operation completes
3. destructive restore semantics are explicit rather than hidden
4. fail-closed behavior is preserved for missing snapshot, missing volume, or unsupported preconditions
Reuse discipline:
1. accepted master-facing snapshot RPC surfaces may be reused only as bounded product adapters for restore if a restore entry surface exists
2. accepted volume-server-facing snapshot/restore surfaces may be reused only as bounded execution adapters
3. underlying `blockvol.RestoreSnapshot` may be reused only as execution reality, not as product-truth ownership
4. `clone` must stay explicitly deferred unless a real product-facing surface is brought into scope and written into `phase-11-log.md`
5. every reused V1 surface must be called out explicitly in `phase-11-log.md` with one of:
- `update in place`
- `reference only`
- `reuse as bounded adapter`
- `reuse as execution reality only`
6. no reused V1 surface may silently redefine restore semantics, workflow readiness, or clone claims
Verification mechanism:
1. focused restore integration tests on the chosen path
2. contract checks that post-restore visible truth matches selected snapshot truth
3. fail-closed checks for invalid or unsupported restore conditions
4. no-overclaim review on what restore/clone workflow behavior is actually supported after the slice
Hard indicators:
1. one accepted restore success proof:
- product-visible restore succeeds on the chosen path
- visible post-restore state matches the selected snapshot truth
2. one accepted destructive-semantics proof:
- writes after the snapshot are lost as designed and this is explicitly verified
3. one accepted fail-closed proof:
- missing snapshot / missing volume / unsupported conditions do not imply false success
4. one accepted post-restore coherence proof:
- list/readback/visible workflow state are coherent after restore
5. one accepted reuse-boundary proof:
- all restore-facing V1 surfaces touched by the slice are explicitly listed and bounded
6. one accepted boundedness proof:
- docs/tests do not imply clone or broad snapshot workflow readiness unless separately proven
Reject if:
1. the slice proves only backend-local restore mechanics without product-visible restore behavior
2. post-restore visible truth is not asserted
3. destructive semantics are left implicit
4. the slice quietly absorbs `clone` or broader workflow readiness work
5. reuse of V1 surfaces is implicit or lets V1 semantics become the source of truth
Status:
- accepted
Carry-forward from `P4`:
1. bounded snapshot restore workflow closure is now accepted on the chosen path
2. accepted proof closes restore success, destructive semantics, post-restore visible truth, and fail-closed behavior
3. `clone` remains explicitly deferred because no real product-facing clone surface is yet accepted into scope
## Phase 11 Completion Judgment
`Phase 11` is complete because:
1. `P1` accepted bounded snapshot create/list/delete product rebinding
2. `P2` accepted bounded `CSI` controller/node lifecycle rebinding
3. `P3` accepted bounded `NVMe` / `iSCSI` publication/address truth rebinding
4. `P4` accepted bounded snapshot restore workflow closure
5. the chosen-path product surface rebinding goal is now closed without reopening accepted `Phase 09` / `Phase 10` semantics
6. remaining work is no longer product-surface rebinding inside `Phase 11`, but production hardening in `Phase 12`
## Assignment For `sw`
Current next tasks:
1. `Phase 11` is closed
2. move next to `Phase 12 P0` production-hardening planning
3. do not reopen accepted `P1` / `P2` / `P3` / `P4` semantics casually during hardening planning
4. keep `clone` deferred unless separately re-scoped in a future phase
## Assignment For `tester`
Current next tasks:
1. `Phase 11` is closed
2. validate `Phase 12 P0` as real planning work rather than placeholder prose
3. keep no-overclaim active around accepted `P1` / `P2` / `P3` / `P4` closure
4. treat `clone` or any other future workflow work as separate re-scoping work, not implicit `Phase 11` residue
+340
View File
@@ -0,0 +1,340 @@
package weed_server
import (
"context"
"fmt"
"log"
"os"
"path/filepath"
"strings"
"testing"
"github.com/container-storage-interface/spec/lib/go/csi"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/storage/blockvol"
bsi "github.com/seaweedfs/seaweedfs/weed/storage/blockvol/csi"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
// ============================================================
// Phase 11 P2: CSI lifecycle rebinding with REAL master backend
//
// These tests wire the CSI controller through a VolumeBackend that
// calls REAL MasterServer.CreateBlockVolume / LookupBlockVolume /
// DeleteBlockVolume — the same master logic that MasterVolumeClient
// calls over gRPC. The node runs with mgr=nil so publish_context
// consumption is the only viable staging path.
//
// Chain:
// CSI controller → masterServerBackend → MasterServer.CreateBlockVolume (real)
// → blockRegistry (real) → blockVSAllocate (real blockvol creation)
// → VolumeInfo with master-produced iscsiAddr/iqn
// → ControllerPublish → publish_context
// → CSI node (mgr=nil) → consumes publish_context → isLocal=false
// ============================================================
// masterServerBackend implements csi.VolumeBackend by calling real
// MasterServer methods. This is the SAME logic that MasterVolumeClient
// executes over gRPC, but called in-process for testability.
type masterServerBackend struct {
ms *MasterServer
}
func (b *masterServerBackend) CreateVolume(ctx context.Context, name string, sizeBytes uint64) (*bsi.VolumeInfo, error) {
resp, err := b.ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: name,
SizeBytes: sizeBytes,
})
if err != nil {
return nil, err
}
return &bsi.VolumeInfo{
VolumeID: resp.VolumeId,
ISCSIAddr: resp.IscsiAddr,
IQN: resp.Iqn,
CapacityBytes: resp.CapacityBytes,
NvmeAddr: resp.NvmeAddr,
NQN: resp.Nqn,
}, nil
}
func (b *masterServerBackend) DeleteVolume(ctx context.Context, name string) error {
_, err := b.ms.DeleteBlockVolume(ctx, &master_pb.DeleteBlockVolumeRequest{Name: name})
return err
}
func (b *masterServerBackend) LookupVolume(ctx context.Context, name string) (*bsi.VolumeInfo, error) {
resp, err := b.ms.LookupBlockVolume(ctx, &master_pb.LookupBlockVolumeRequest{Name: name})
if err != nil {
return nil, err
}
return &bsi.VolumeInfo{
VolumeID: name,
ISCSIAddr: resp.IscsiAddr,
IQN: resp.Iqn,
CapacityBytes: resp.CapacityBytes,
NvmeAddr: resp.NvmeAddr,
NQN: resp.Nqn,
}, nil
}
func (b *masterServerBackend) CreateSnapshot(ctx context.Context, volumeID string, snapID uint32) (*bsi.SnapshotInfo, error) {
return nil, fmt.Errorf("not in P2 scope")
}
func (b *masterServerBackend) DeleteSnapshot(ctx context.Context, volumeID string, snapID uint32) error {
return fmt.Errorf("not in P2 scope")
}
func (b *masterServerBackend) ListSnapshots(ctx context.Context, volumeID string) ([]*bsi.SnapshotInfo, error) {
return nil, fmt.Errorf("not in P2 scope")
}
func (b *masterServerBackend) ExpandVolume(ctx context.Context, volumeID string, newSizeBytes uint64) (uint64, error) {
return 0, fmt.Errorf("not in P2 scope")
}
func newCSILifecycleSetup(t *testing.T) (*bsi.ExportedControllerServer, *bsi.ExportedNodeServer, *MasterServer, string) {
t.Helper()
dir := t.TempDir()
ms := &MasterServer{
blockRegistry: NewBlockVolumeRegistry(),
blockAssignmentQueue: NewBlockAssignmentQueue(),
blockFailover: newBlockFailoverState(),
}
ms.blockRegistry.MarkBlockCapable("vs1:9333")
ms.blockRegistry.MarkBlockCapable("vs2:9333")
ms.blockVSAllocate = func(ctx context.Context, server string, name string, sizeBytes uint64, diskType string, durabilityMode string) (*blockAllocResult, error) {
sanitized := strings.ReplaceAll(server, ":", "_")
serverDir := filepath.Join(dir, sanitized)
os.MkdirAll(serverDir, 0755)
volPath := filepath.Join(serverDir, fmt.Sprintf("%s.blk", name))
vol, err := blockvol.CreateBlockVol(volPath, blockvol.CreateOptions{
VolumeSize: 1 * 1024 * 1024,
BlockSize: 4096,
WALSize: 256 * 1024,
})
if err != nil {
return nil, err
}
vol.Close()
// Derive a valid ip:port iSCSI address from the server identity.
// Server identity is "vs1:9333" — extract the host part for the iSCSI portal.
iscsiHost := server
if idx := strings.LastIndex(server, ":"); idx >= 0 {
iscsiHost = server[:idx]
}
return &blockAllocResult{
Path: volPath,
IQN: fmt.Sprintf("iqn.2024.master:%s", name),
ISCSIAddr: iscsiHost + ":3260",
}, nil
}
ms.blockVSDelete = func(ctx context.Context, server string, name string) error {
// Real delete: remove the .blk file from the server's directory.
sanitized := strings.ReplaceAll(server, ":", "_")
volPath := filepath.Join(dir, sanitized, fmt.Sprintf("%s.blk", name))
return os.Remove(volPath)
}
backend := &masterServerBackend{ms: ms}
ctrl := bsi.NewExportedControllerServer(backend)
node := bsi.NewExportedNodeServer("test-node-csi", nil, log.New(os.Stderr, "csi-node: ", log.LstdFlags))
return ctrl, node, ms, dir
}
// --- 1. Master-produced publish_context ---
func TestP11P2_CSI_MasterProduced_PublishContext(t *testing.T) {
ctrl, _, ms, _ := newCSILifecycleSetup(t)
ctx := context.Background()
createResp, err := ctrl.CreateVolume(ctx, &csi.CreateVolumeRequest{
Name: "csi-vol-1",
CapacityRange: &csi.CapacityRange{RequiredBytes: 1 << 20},
VolumeCapabilities: []*csi.VolumeCapability{singleNodeWriter()},
})
if err != nil {
t.Fatalf("CreateVolume: %v", err)
}
pubResp, err := ctrl.ControllerPublishVolume(ctx, &csi.ControllerPublishVolumeRequest{
VolumeId: createResp.Volume.VolumeId, NodeId: "test-node-csi",
VolumeCapability: singleNodeWriter(),
})
if err != nil {
t.Fatalf("Publish: %v", err)
}
// Verify: publish_context carries master-produced target info.
entry, _ := ms.blockRegistry.Lookup("csi-vol-1")
// Derive expected iSCSI portal: host from VolumeServer + :3260.
vsHost := entry.VolumeServer
if idx := strings.LastIndex(vsHost, ":"); idx >= 0 {
vsHost = vsHost[:idx]
}
expectedISCSI := vsHost + ":3260"
expectedIQN := fmt.Sprintf("iqn.2024.master:%s", "csi-vol-1")
if pubResp.PublishContext["iscsiAddr"] != expectedISCSI {
t.Fatalf("iscsiAddr=%q, want %q (master-produced)", pubResp.PublishContext["iscsiAddr"], expectedISCSI)
}
if pubResp.PublishContext["iqn"] != expectedIQN {
t.Fatalf("iqn=%q, want %q (master-produced)", pubResp.PublishContext["iqn"], expectedIQN)
}
t.Logf("P11P2 master-produced: iscsiAddr=%s iqn=%s (both asserted)",
pubResp.PublishContext["iscsiAddr"], pubResp.PublishContext["iqn"])
}
// --- 2. Full lifecycle: master backend + mgr=nil node ---
func TestP11P2_CSI_FullLifecycle(t *testing.T) {
ctrl, node, _, testDir := newCSILifecycleSetup(t)
ctx := context.Background()
// Create.
createResp, err := ctrl.CreateVolume(ctx, &csi.CreateVolumeRequest{
Name: "csi-vol-2", CapacityRange: &csi.CapacityRange{RequiredBytes: 1 << 20},
VolumeCapabilities: []*csi.VolumeCapability{singleNodeWriter()},
})
if err != nil {
t.Fatalf("Create: %v", err)
}
volID := createResp.Volume.VolumeId
// Publish.
pubResp, err := ctrl.ControllerPublishVolume(ctx, &csi.ControllerPublishVolumeRequest{
VolumeId: volID, NodeId: "test-node-csi", VolumeCapability: singleNodeWriter(),
})
if err != nil {
t.Fatalf("Publish: %v", err)
}
// Stage (mgr=nil → publish_context only path).
stagingDir := filepath.Join(t.TempDir(), "staging")
os.MkdirAll(stagingDir, 0755)
_, err = node.NodeStageVolume(ctx, &csi.NodeStageVolumeRequest{
VolumeId: volID, StagingTargetPath: stagingDir,
PublishContext: pubResp.PublishContext, VolumeCapability: singleNodeWriter(),
})
if err != nil {
t.Fatalf("Stage: %v", err)
}
// Assert: isLocal=false and staged addr matches publish_context.
staged := node.GetStagedInfo(volID)
if staged == nil {
t.Fatal("not in staged map")
}
if staged.IsLocal {
t.Fatal("isLocal must be false (mgr=nil)")
}
if staged.ISCSIAddr != pubResp.PublishContext["iscsiAddr"] {
t.Fatalf("staged addr=%q != publish_context=%q", staged.ISCSIAddr, pubResp.PublishContext["iscsiAddr"])
}
// NodePublish.
targetDir := filepath.Join(t.TempDir(), "target")
os.MkdirAll(targetDir, 0755)
_, err = node.NodePublishVolume(ctx, &csi.NodePublishVolumeRequest{
VolumeId: volID, StagingTargetPath: stagingDir, TargetPath: targetDir,
VolumeCapability: singleNodeWriter(),
})
if err != nil {
t.Fatalf("NodePublish: %v", err)
}
// Unpublish.
if _, err := node.NodeUnpublishVolume(ctx, &csi.NodeUnpublishVolumeRequest{VolumeId: volID, TargetPath: targetDir}); err != nil {
t.Fatalf("NodeUnpublish: %v", err)
}
// Unstage.
if _, err := node.NodeUnstageVolume(ctx, &csi.NodeUnstageVolumeRequest{VolumeId: volID, StagingTargetPath: stagingDir}); err != nil {
t.Fatalf("NodeUnstage: %v", err)
}
// Delete (real VS-side file removal via blockVSDelete).
if _, err := ctrl.DeleteVolume(ctx, &csi.DeleteVolumeRequest{VolumeId: volID}); err != nil {
t.Fatalf("DeleteVolume: %v", err)
}
// Verify: .blk file is gone on disk (real file removal).
volFilePath := filepath.Join(testDir, "vs1_9333", "csi-vol-2.blk")
if _, err := os.Stat(volFilePath); !os.IsNotExist(err) {
t.Fatalf(".blk file should be gone after delete: %s (err=%v)", volFilePath, err)
}
// Verify: publish fails (controller-level — volume unregistered).
_, err = ctrl.ControllerPublishVolume(ctx, &csi.ControllerPublishVolumeRequest{
VolumeId: volID, NodeId: "test-node-csi", VolumeCapability: singleNodeWriter(),
})
if err == nil {
t.Fatal("publish after delete should fail")
}
t.Log("P11P2 lifecycle: master-created → publish → stage(isLocal=false) → pub → unpub → unstage → delete → gone")
}
// --- 3. Fail-closed: missing context + mgr=nil ---
func TestP11P2_CSI_FailClosed_NoContext(t *testing.T) {
ctrl, node, _, _ := newCSILifecycleSetup(t)
ctx := context.Background()
createResp, err := ctrl.CreateVolume(ctx, &csi.CreateVolumeRequest{
Name: "csi-vol-fc", CapacityRange: &csi.CapacityRange{RequiredBytes: 1 << 20},
VolumeCapabilities: []*csi.VolumeCapability{singleNodeWriter()},
})
if err != nil {
t.Fatalf("setup CreateVolume: %v", err)
}
stagingDir := filepath.Join(t.TempDir(), "staging")
os.MkdirAll(stagingDir, 0755)
_, stageErr := node.NodeStageVolume(ctx, &csi.NodeStageVolumeRequest{
VolumeId: createResp.Volume.VolumeId, StagingTargetPath: stagingDir,
VolumeCapability: singleNodeWriter(),
})
if stageErr == nil {
t.Fatal("stage without context + mgr=nil should fail")
}
st, ok := status.FromError(stageErr)
if !ok || st.Code() != codes.FailedPrecondition {
t.Fatalf("expected FailedPrecondition, got %v", stageErr)
}
t.Logf("P11P2 fail-closed: %s — %s", st.Code(), st.Message())
}
// --- 4. Fail-closed: missing volume ---
func TestP11P2_CSI_FailClosed_MissingVolume(t *testing.T) {
ctrl, _, _, _ := newCSILifecycleSetup(t)
ctx := context.Background()
_, err := ctrl.ControllerPublishVolume(ctx, &csi.ControllerPublishVolumeRequest{
VolumeId: "nonexistent", NodeId: "n", VolumeCapability: singleNodeWriter(),
})
if err == nil {
t.Fatal("should fail")
}
st, _ := status.FromError(err)
if st.Code() != codes.NotFound {
t.Fatalf("expected NotFound, got %s", st.Code())
}
// Delete nonexistent: idempotent.
_, delErr := ctrl.DeleteVolume(ctx, &csi.DeleteVolumeRequest{VolumeId: "nonexistent"})
if delErr != nil {
t.Fatalf("delete nonexistent should be idempotent: %v", delErr)
}
t.Log("P11P2 fail-closed: NotFound on publish, idempotent on delete")
}
func singleNodeWriter() *csi.VolumeCapability {
return &csi.VolumeCapability{
AccessMode: &csi.VolumeCapability_AccessMode{Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER},
}
}
+303
View File
@@ -0,0 +1,303 @@
package weed_server
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/storage/blockvol"
)
// ============================================================
// Phase 11 P3: Front-end publication rebinding
//
// Proofs:
// 1. Create/Lookup coherence: iSCSI + NVMe fields match
// 2. Failover: publication truth switches to new primary
// 3. Heartbeat reconstruction: fields survive restart
// 4. No-NVMe fallback: explicit and coherent
//
// V1 reuse roles:
// master_grpc_server_block.go: reuse as bounded adapter
// master_block_registry.go: reuse as bounded truth carrier
// volume_server_block.go: reuse as publication source only
// ============================================================
func newPublicationMaster(t *testing.T, nvmeEnabled bool) *MasterServer {
t.Helper()
dir := t.TempDir()
ms := &MasterServer{
blockRegistry: NewBlockVolumeRegistry(),
blockAssignmentQueue: NewBlockAssignmentQueue(),
blockFailover: newBlockFailoverState(),
}
ms.blockRegistry.MarkBlockCapable("vs1:9333")
ms.blockRegistry.MarkBlockCapable("vs2:9333")
ms.blockVSAllocate = func(ctx context.Context, server string, name string, sizeBytes uint64, diskType string, durabilityMode string) (*blockAllocResult, error) {
sanitized := strings.ReplaceAll(server, ":", "_")
serverDir := filepath.Join(dir, sanitized)
os.MkdirAll(serverDir, 0755)
volPath := filepath.Join(serverDir, fmt.Sprintf("%s.blk", name))
vol, err := blockvol.CreateBlockVol(volPath, blockvol.CreateOptions{
VolumeSize: 1 * 1024 * 1024,
BlockSize: 4096,
WALSize: 256 * 1024,
})
if err != nil {
return nil, err
}
vol.Close()
host := server
if idx := strings.LastIndex(server, ":"); idx >= 0 {
host = server[:idx]
}
result := &blockAllocResult{
Path: volPath,
IQN: fmt.Sprintf("iqn.2024.test:%s", name),
ISCSIAddr: host + ":3260",
ReplicaDataAddr: server + ":14260",
ReplicaCtrlAddr: server + ":14261",
RebuildListenAddr: server + ":15000",
}
if nvmeEnabled {
result.NvmeAddr = host + ":4420"
result.NQN = fmt.Sprintf("nqn.2024-01.com.seaweedfs:vol.%s", name)
}
return result, nil
}
ms.blockVSDelete = func(ctx context.Context, server string, name string) error { return nil }
return ms
}
// --- 1. Create/Lookup coherence ---
func TestP11P3_CreateLookup_PublicationCoherence(t *testing.T) {
ms := newPublicationMaster(t, true)
ctx := context.Background()
createResp, err := ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "pub-vol-1", SizeBytes: 1 << 30,
})
if err != nil {
t.Fatalf("Create: %v", err)
}
// Create response has iSCSI + NVMe fields.
if createResp.IscsiAddr == "" || createResp.Iqn == "" {
t.Fatal("create missing iSCSI fields")
}
if createResp.NvmeAddr == "" || createResp.Nqn == "" {
t.Fatal("create missing NVMe fields")
}
// Lookup must return the SAME publication fields.
lookupResp, err := ms.LookupBlockVolume(ctx, &master_pb.LookupBlockVolumeRequest{Name: "pub-vol-1"})
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if lookupResp.IscsiAddr != createResp.IscsiAddr {
t.Fatalf("iSCSI addr mismatch: create=%q lookup=%q", createResp.IscsiAddr, lookupResp.IscsiAddr)
}
if lookupResp.Iqn != createResp.Iqn {
t.Fatalf("IQN mismatch: create=%q lookup=%q", createResp.Iqn, lookupResp.Iqn)
}
if lookupResp.NvmeAddr != createResp.NvmeAddr {
t.Fatalf("NVMe addr mismatch: create=%q lookup=%q", createResp.NvmeAddr, lookupResp.NvmeAddr)
}
if lookupResp.Nqn != createResp.Nqn {
t.Fatalf("NQN mismatch: create=%q lookup=%q", createResp.Nqn, lookupResp.Nqn)
}
// Verify: registry truth matches.
entry, _ := ms.blockRegistry.Lookup("pub-vol-1")
if entry.ISCSIAddr != createResp.IscsiAddr {
t.Fatalf("registry iSCSI=%q != create=%q", entry.ISCSIAddr, createResp.IscsiAddr)
}
if entry.NvmeAddr != createResp.NvmeAddr {
t.Fatalf("registry NVMe=%q != create=%q", entry.NvmeAddr, createResp.NvmeAddr)
}
t.Logf("P11P3 coherence: create=lookup=registry — iSCSI=%s NVMe=%s IQN=%s NQN=%s",
createResp.IscsiAddr, createResp.NvmeAddr, createResp.Iqn, createResp.Nqn)
}
// --- 2. Failover: publication truth switches ---
func TestP11P3_Failover_PublicationSwitches(t *testing.T) {
ms := newPublicationMaster(t, true)
ctx := context.Background()
createResp, err := ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "pub-vol-2", SizeBytes: 1 << 30,
})
if err != nil {
t.Fatalf("Create: %v", err)
}
oldPrimary := createResp.VolumeServer
oldISCSI := createResp.IscsiAddr
oldNVMe := createResp.NvmeAddr
t.Logf("before failover: primary=%s iSCSI=%s NVMe=%s", oldPrimary, oldISCSI, oldNVMe)
// Expire lease and failover.
ms.blockRegistry.UpdateEntry("pub-vol-2", func(e *BlockVolumeEntry) {
e.LastLeaseGrant = time.Now().Add(-1 * time.Minute)
})
ms.failoverBlockVolumes(oldPrimary)
// Lookup after failover: publication fields should reflect NEW primary.
lookupResp, err := ms.LookupBlockVolume(ctx, &master_pb.LookupBlockVolumeRequest{Name: "pub-vol-2"})
if err != nil {
t.Fatalf("Lookup after failover: %v", err)
}
entry, _ := ms.blockRegistry.Lookup("pub-vol-2")
if entry.VolumeServer == oldPrimary {
t.Fatalf("primary should have changed, still %s", oldPrimary)
}
// New primary's publication fields should differ from old primary's.
if lookupResp.IscsiAddr == oldISCSI {
t.Fatalf("iSCSI addr should change after failover: still %q", lookupResp.IscsiAddr)
}
if lookupResp.NvmeAddr == oldNVMe {
t.Fatalf("NVMe addr should change after failover: still %q", lookupResp.NvmeAddr)
}
// Registry and lookup must still agree.
if lookupResp.IscsiAddr != entry.ISCSIAddr {
t.Fatalf("post-failover: lookup iSCSI=%q != registry=%q", lookupResp.IscsiAddr, entry.ISCSIAddr)
}
if lookupResp.NvmeAddr != entry.NvmeAddr {
t.Fatalf("post-failover: lookup NVMe=%q != registry=%q", lookupResp.NvmeAddr, entry.NvmeAddr)
}
t.Logf("P11P3 failover: old=%s→new=%s iSCSI=%s NVMe=%s (switched)",
oldPrimary, entry.VolumeServer, lookupResp.IscsiAddr, lookupResp.NvmeAddr)
}
// --- 3. Heartbeat reconstruction ---
func TestP11P3_HeartbeatReconstruction(t *testing.T) {
ms := newPublicationMaster(t, true)
ctx := context.Background()
createResp, err := ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "pub-vol-3", SizeBytes: 1 << 30,
})
if err != nil {
t.Fatalf("Create: %v", err)
}
entry, _ := ms.blockRegistry.Lookup("pub-vol-3")
originalNVMe := entry.NvmeAddr
originalNQN := entry.NQN
primaryServer := entry.VolumeServer
// Clear NVMe fields to simulate master restart state loss.
ms.blockRegistry.UpdateEntry("pub-vol-3", func(e *BlockVolumeEntry) {
e.NvmeAddr = ""
e.NQN = ""
})
cleared, _ := ms.blockRegistry.Lookup("pub-vol-3")
if cleared.NvmeAddr != "" {
t.Fatal("NvmeAddr should be cleared")
}
// Reconstruct via REAL heartbeat path: UpdateFullHeartbeat with proto info.
// This is the same code path as master_grpc_server.go:280.
heartbeatInfo := &master_pb.BlockVolumeInfoMessage{
Path: entry.Path,
VolumeSize: entry.SizeBytes,
BlockSize: 4096,
Epoch: entry.Epoch,
Role: entry.Role,
NvmeAddr: originalNVMe,
Nqn: originalNQN,
}
// Derive server-level NVMe addr (same as what the VS heartbeat carries).
host := primaryServer
if idx := strings.LastIndex(host, ":"); idx >= 0 {
host = host[:idx]
}
serverNvmeAddr := host + ":4420"
ms.blockRegistry.UpdateFullHeartbeat(
primaryServer,
[]*master_pb.BlockVolumeInfoMessage{heartbeatInfo},
serverNvmeAddr,
)
// Verify: lookup reflects heartbeat-reconstructed NVMe truth.
lookupResp, err := ms.LookupBlockVolume(ctx, &master_pb.LookupBlockVolumeRequest{Name: "pub-vol-3"})
if err != nil {
t.Fatalf("Lookup after reconstruction: %v", err)
}
if lookupResp.NvmeAddr != originalNVMe {
t.Fatalf("reconstructed NvmeAddr=%q, want %q", lookupResp.NvmeAddr, originalNVMe)
}
if lookupResp.Nqn != originalNQN {
t.Fatalf("reconstructed NQN=%q, want %q", lookupResp.Nqn, originalNQN)
}
_ = createResp // used for initial creation
t.Logf("P11P3 reconstruction: cleared → UpdateFullHeartbeat → lookup restored NVMe=%s NQN=%s",
lookupResp.NvmeAddr, lookupResp.Nqn)
}
// --- 4. No-NVMe fallback ---
func TestP11P3_NoNVMe_Fallback(t *testing.T) {
ms := newPublicationMaster(t, false) // NVMe disabled
ctx := context.Background()
createResp, err := ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "pub-vol-4", SizeBytes: 1 << 30,
})
if err != nil {
t.Fatalf("Create: %v", err)
}
// iSCSI must be present.
if createResp.IscsiAddr == "" || createResp.Iqn == "" {
t.Fatal("iSCSI fields must be present even without NVMe")
}
// NVMe must be explicitly empty (not fabricated).
if createResp.NvmeAddr != "" {
t.Fatalf("NvmeAddr=%q should be empty when NVMe disabled", createResp.NvmeAddr)
}
if createResp.Nqn != "" {
t.Fatalf("Nqn=%q should be empty when NVMe disabled", createResp.Nqn)
}
// Lookup must also show empty NVMe.
lookupResp, _ := ms.LookupBlockVolume(ctx, &master_pb.LookupBlockVolumeRequest{Name: "pub-vol-4"})
if lookupResp.NvmeAddr != "" || lookupResp.Nqn != "" {
t.Fatalf("lookup: NVMe should be empty, got addr=%q nqn=%q", lookupResp.NvmeAddr, lookupResp.Nqn)
}
// Registry must also show empty NVMe.
entry, _ := ms.blockRegistry.Lookup("pub-vol-4")
if entry.NvmeAddr != "" || entry.NQN != "" {
t.Fatalf("registry: NVMe should be empty, got addr=%q nqn=%q", entry.NvmeAddr, entry.NQN)
}
t.Logf("P11P3 no-NVMe: iSCSI=%s IQN=%s, NVMe explicitly empty across create/lookup/registry",
createResp.IscsiAddr, createResp.Iqn)
}
+240
View File
@@ -0,0 +1,240 @@
package weed_server
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
volume_server_pb "github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"github.com/seaweedfs/seaweedfs/weed/storage"
"github.com/seaweedfs/seaweedfs/weed/storage/blockvol"
)
// ============================================================
// Phase 11 P4: Snapshot restore workflow closure
//
// Proofs:
// 1. Restore success: product-visible restore reverts to snapshot
// 2. Destructive semantics: post-snapshot writes are lost
// 3. Post-restore coherence: list shows expected snapshot state
// 4. Fail-closed: missing snapshot, missing volume
//
// V1 reuse roles:
// master_grpc_server_block.go: reuse as bounded adapter (RestoreBlockSnapshot RPC)
// volume_server_block.go: reuse as bounded adapter (RestoreBlockSnapshot)
// blockvol.RestoreSnapshot: reuse as execution reality only
// ============================================================
func newRestoreMaster(t *testing.T) (*MasterServer, *storage.BlockVolumeStore, *BlockService) {
t.Helper()
dir := t.TempDir()
store := storage.NewBlockVolumeStore()
ms := &MasterServer{
blockRegistry: NewBlockVolumeRegistry(),
blockAssignmentQueue: NewBlockAssignmentQueue(),
blockFailover: newBlockFailoverState(),
}
ms.blockRegistry.MarkBlockCapable("vs1:9333")
ms.blockRegistry.MarkBlockCapable("vs2:9333")
ms.blockVSAllocate = func(ctx context.Context, server string, name string, sizeBytes uint64, diskType string, durabilityMode string) (*blockAllocResult, error) {
sanitized := strings.ReplaceAll(server, ":", "_")
serverDir := filepath.Join(dir, sanitized)
os.MkdirAll(serverDir, 0755)
volPath := filepath.Join(serverDir, fmt.Sprintf("%s.blk", name))
vol, err := blockvol.CreateBlockVol(volPath, blockvol.CreateOptions{
VolumeSize: 1 * 1024 * 1024,
BlockSize: 4096,
WALSize: 256 * 1024,
})
if err != nil {
return nil, err
}
vol.Close()
if _, err := store.AddBlockVolume(volPath, ""); err != nil {
return nil, err
}
host := server
if idx := strings.LastIndex(server, ":"); idx >= 0 {
host = server[:idx]
}
return &blockAllocResult{
Path: volPath,
IQN: fmt.Sprintf("iqn.2024.test:%s", name),
ISCSIAddr: host + ":3260",
}, nil
}
ms.blockVSDelete = func(ctx context.Context, server string, name string) error { return nil }
// Build a real BlockService so ALL callbacks go through the VS adapter.
// Set blockDir to the vs1 subdir so volumePath(name) resolves correctly.
bs := &BlockService{
blockStore: store,
blockDir: filepath.Join(dir, strings.ReplaceAll("vs1:9333", ":", "_")),
listenAddr: "127.0.0.1:3260",
}
// Wire snapshot callbacks through real BlockService adapters.
ms.blockVSSnapshot = func(ctx context.Context, server string, name string, snapID uint32) (int64, uint64, error) {
return bs.SnapshotBlockVol(name, snapID)
}
ms.blockVSDeleteSnap = func(ctx context.Context, server string, name string, snapID uint32) error {
return bs.DeleteBlockSnapshot(name, snapID)
}
ms.blockVSListSnaps = func(ctx context.Context, server string, name string) ([]*volume_server_pb.BlockSnapshotInfo, error) {
infos, volSize, err := bs.ListBlockSnapshots(name)
if err != nil {
return nil, err
}
var result []*volume_server_pb.BlockSnapshotInfo
for _, si := range infos {
result = append(result, &volume_server_pb.BlockSnapshotInfo{
SnapshotId: si.ID,
CreatedAt: si.CreatedAt.Unix(),
VolumeSizeBytes: volSize,
})
}
return result, nil
}
// Wire restore callback through REAL BlockService.RestoreBlockSnapshot adapter.
ms.blockVSRestore = func(ctx context.Context, server string, name string, snapID uint32) error {
return bs.RestoreBlockSnapshot(name, snapID)
}
t.Cleanup(func() { store.Close() })
return ms, store, bs
}
// --- 1. Restore success: reverts to snapshot ---
func TestP11P4_RestoreSuccess(t *testing.T) {
ms, store, _ := newRestoreMaster(t)
ctx := context.Background()
// Create volume.
ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "restore-vol-1", SizeBytes: 1 << 20,
})
entry, _ := ms.blockRegistry.Lookup("restore-vol-1")
// Write data, create snapshot.
store.WithVolume(entry.Path, func(vol *blockvol.BlockVol) error {
vol.WriteLBA(0, make([]byte, 4096))
vol.WriteLBA(1, make([]byte, 4096))
return nil
})
ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "restore-vol-1", SnapshotId: 1,
})
// Write MORE data after snapshot.
store.WithVolume(entry.Path, func(vol *blockvol.BlockVol) error {
data := make([]byte, 4096)
for i := range data {
data[i] = 0xFF
}
vol.WriteLBA(0, data) // overwrite LBA 0
vol.WriteLBA(5, data) // new LBA 5
return nil
})
// Restore to snapshot 1.
_, err := ms.RestoreBlockSnapshot(ctx, &master_pb.RestoreBlockSnapshotRequest{
VolumeName: "restore-vol-1", SnapshotId: 1,
})
if err != nil {
t.Fatalf("Restore: %v", err)
}
// Verify: LBA 0 should be exactly 0x00 (snapshot was taken after zero-fill write).
var lba0First, lba5First byte
store.WithVolume(entry.Path, func(vol *blockvol.BlockVol) error {
data0, _ := vol.ReadLBA(0, 4096)
lba0First = data0[0]
data5, _ := vol.ReadLBA(5, 4096)
lba5First = data5[0]
return nil
})
if lba0First != 0x00 {
t.Fatalf("LBA 0 = 0x%02X, want 0x00 (reverted to snapshot state)", lba0First)
}
if lba5First != 0x00 {
t.Fatalf("LBA 5 = 0x%02X, want 0x00 (post-snapshot write should be lost)", lba5First)
}
t.Logf("P11P4 restore: LBA 0 = 0x%02X (reverted), LBA 5 = 0x%02X (post-snap lost)", lba0First, lba5First)
}
// --- 2. Destructive semantics: snapshots gone after restore ---
func TestP11P4_DestructiveSemantics(t *testing.T) {
ms, _, _ := newRestoreMaster(t)
ctx := context.Background()
ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "restore-vol-2", SizeBytes: 1 << 20,
})
// Create snapshots 1, 2, 3.
for i := uint32(1); i <= 3; i++ {
ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "restore-vol-2", SnapshotId: i,
})
}
// Restore to snapshot 1: all snapshots are removed by restore.
ms.RestoreBlockSnapshot(ctx, &master_pb.RestoreBlockSnapshotRequest{
VolumeName: "restore-vol-2", SnapshotId: 1,
})
// List: should have 0 snapshots (restore removes all).
listResp, _ := ms.ListBlockSnapshots(ctx, &master_pb.ListBlockSnapshotsRequest{
VolumeName: "restore-vol-2",
})
if len(listResp.Snapshots) != 0 {
t.Fatalf("expected 0 snapshots after restore, got %d", len(listResp.Snapshots))
}
t.Log("P11P4 destructive: restore to snap 1 → all snapshots removed")
}
// --- 3. Fail-closed: missing snapshot ---
func TestP11P4_FailClosed_MissingSnapshot(t *testing.T) {
ms, _, _ := newRestoreMaster(t)
ctx := context.Background()
ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: "restore-vol-3", SizeBytes: 1 << 20,
})
// Restore nonexistent snapshot.
_, err := ms.RestoreBlockSnapshot(ctx, &master_pb.RestoreBlockSnapshotRequest{
VolumeName: "restore-vol-3", SnapshotId: 99,
})
if err == nil {
t.Fatal("restore missing snapshot should fail")
}
t.Logf("P11P4 fail-closed: missing snapshot → %v", err)
}
// --- 4. Fail-closed: missing volume ---
func TestP11P4_FailClosed_MissingVolume(t *testing.T) {
ms, _, _ := newRestoreMaster(t)
ctx := context.Background()
_, err := ms.RestoreBlockSnapshot(ctx, &master_pb.RestoreBlockSnapshotRequest{
VolumeName: "nonexistent-vol", SnapshotId: 1,
})
if err == nil {
t.Fatal("restore on missing volume should fail")
}
t.Logf("P11P4 fail-closed: missing volume → %v", err)
}
@@ -0,0 +1,342 @@
package weed_server
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"github.com/seaweedfs/seaweedfs/weed/storage"
"github.com/seaweedfs/seaweedfs/weed/storage/blockvol"
)
// ============================================================
// Phase 11 P1: Snapshot product-path rebinding
//
// Proofs:
// 1. Create: product-visible create → backend snapshot → readback metadata
// 2. List: listed snapshots match backend truth after create and delete
// 3. Delete: snapshot removed from visible set, repeated delete idempotent
// 4. Fail-closed: nonexistent volume, missing snapshot
// 5. Reuse boundary: all V1 surfaces explicitly bounded
//
// V1 reuse roles:
// master_grpc_server_block.go: reuse as bounded adapter (RPC resolution)
// volume_server_block.go: reuse as bounded adapter (VS execution)
// blockvol/*: reuse as execution reality only
// ============================================================
// snapshotTestSetup creates a master + BlockService with a real volume,
// wiring the master's VS snapshot callbacks to the real BlockService.
type snapshotTestSetup struct {
ms *MasterServer
bs *BlockService
store *storage.BlockVolumeStore
dir string
}
func newSnapshotTestSetup(t *testing.T) *snapshotTestSetup {
t.Helper()
dir := t.TempDir()
store := storage.NewBlockVolumeStore()
bs := &BlockService{
blockStore: store,
blockDir: dir,
listenAddr: "127.0.0.1:3260",
}
ms := &MasterServer{
blockRegistry: NewBlockVolumeRegistry(),
blockAssignmentQueue: NewBlockAssignmentQueue(),
blockFailover: newBlockFailoverState(),
}
ms.blockRegistry.MarkBlockCapable("vs1:9333")
ms.blockRegistry.MarkBlockCapable("vs2:9333")
s := &snapshotTestSetup{ms: ms, bs: bs, store: store, dir: dir}
// Wire master allocator to create real volumes.
ms.blockVSAllocate = func(ctx context.Context, server string, name string, sizeBytes uint64, diskType string, durabilityMode string) (*blockAllocResult, error) {
sanitized := strings.ReplaceAll(server, ":", "_")
serverDir := filepath.Join(dir, sanitized)
if err := os.MkdirAll(serverDir, 0755); err != nil {
return nil, err
}
volPath := filepath.Join(serverDir, fmt.Sprintf("%s.blk", name))
vol, err := blockvol.CreateBlockVol(volPath, blockvol.CreateOptions{
VolumeSize: 1 * 1024 * 1024,
BlockSize: 4096,
WALSize: 256 * 1024,
})
if err != nil {
return nil, err
}
vol.Close()
if _, err := store.AddBlockVolume(volPath, ""); err != nil {
return nil, err
}
return &blockAllocResult{
Path: volPath,
IQN: fmt.Sprintf("iqn.2024.test:%s", name),
ISCSIAddr: server + ":3260",
}, nil
}
ms.blockVSDelete = func(ctx context.Context, server string, name string) error {
return nil
}
// Wire master snapshot callbacks through REAL BlockService adapter methods.
// This proves the VS adapter layer (SnapshotBlockVol, DeleteBlockSnapshot,
// ListBlockSnapshots) is part of the tested chain, not bypassed.
//
// Set blockDir to the vs1 subdir so BlockService.volumePath(name) resolves
// to the same paths registered in the store.
bs.blockDir = filepath.Join(dir, strings.ReplaceAll("vs1:9333", ":", "_"))
ms.blockVSSnapshot = func(ctx context.Context, server string, name string, snapID uint32) (int64, uint64, error) {
return bs.SnapshotBlockVol(name, snapID)
}
ms.blockVSDeleteSnap = func(ctx context.Context, server string, name string, snapID uint32) error {
return bs.DeleteBlockSnapshot(name, snapID)
}
ms.blockVSListSnaps = func(ctx context.Context, server string, name string) ([]*volume_server_pb.BlockSnapshotInfo, error) {
infos, volSize, err := bs.ListBlockSnapshots(name)
if err != nil {
return nil, err
}
var result []*volume_server_pb.BlockSnapshotInfo
for _, si := range infos {
result = append(result, &volume_server_pb.BlockSnapshotInfo{
SnapshotId: si.ID,
CreatedAt: si.CreatedAt.Unix(),
VolumeSizeBytes: volSize,
})
}
return result, nil
}
t.Cleanup(func() { store.Close() })
return s
}
func (s *snapshotTestSetup) createVolume(t *testing.T, name string) {
t.Helper()
ctx := context.Background()
_, err := s.ms.CreateBlockVolume(ctx, &master_pb.CreateBlockVolumeRequest{
Name: name,
SizeBytes: 1 << 30,
})
if err != nil {
t.Fatalf("CreateBlockVolume(%s): %v", name, err)
}
}
// --- 1. Create: product-visible snapshot creation ---
func TestP11P1_SnapshotCreate(t *testing.T) {
s := newSnapshotTestSetup(t)
s.createVolume(t, "snap-vol-1")
ctx := context.Background()
// Product-visible create through master RPC.
createResp, err := s.ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "snap-vol-1",
SnapshotId: 1,
})
if err != nil {
t.Fatalf("CreateBlockSnapshot: %v", err)
}
if createResp.SnapshotId != 1 {
t.Fatalf("SnapshotId=%d, want 1", createResp.SnapshotId)
}
if createResp.CreatedAt == 0 {
t.Fatal("CreatedAt should be non-zero")
}
if createResp.SizeBytes == 0 {
t.Fatal("SizeBytes should be non-zero")
}
// Verify: snapshot is visible through list.
listResp, err := s.ms.ListBlockSnapshots(ctx, &master_pb.ListBlockSnapshotsRequest{
VolumeName: "snap-vol-1",
})
if err != nil {
t.Fatalf("ListBlockSnapshots: %v", err)
}
if len(listResp.Snapshots) != 1 {
t.Fatalf("expected 1 snapshot, got %d", len(listResp.Snapshots))
}
listed := listResp.Snapshots[0]
if listed.SnapshotId != 1 {
t.Fatalf("listed SnapshotId=%d, want 1", listed.SnapshotId)
}
// Metadata equality: create response and list response must agree.
if listed.CreatedAt != createResp.CreatedAt {
t.Fatalf("metadata mismatch: create.CreatedAt=%d != list.CreatedAt=%d",
createResp.CreatedAt, listed.CreatedAt)
}
if listed.VolumeSizeBytes != createResp.SizeBytes {
t.Fatalf("metadata mismatch: create.SizeBytes=%d != list.VolumeSizeBytes=%d",
createResp.SizeBytes, listed.VolumeSizeBytes)
}
t.Logf("P11P1 create: master RPC → VS adapter → blockvol → snap(id=1, created=%d, size=%d) → list metadata matches",
createResp.CreatedAt, createResp.SizeBytes)
}
// --- 2. List coherence: after create and delete ---
func TestP11P1_SnapshotListCoherence(t *testing.T) {
s := newSnapshotTestSetup(t)
s.createVolume(t, "snap-vol-2")
ctx := context.Background()
// Create 3 snapshots.
for i := uint32(1); i <= 3; i++ {
_, err := s.ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "snap-vol-2",
SnapshotId: i,
})
if err != nil {
t.Fatalf("create snap %d: %v", i, err)
}
}
// List: should have 3.
listResp, err := s.ms.ListBlockSnapshots(ctx, &master_pb.ListBlockSnapshotsRequest{
VolumeName: "snap-vol-2",
})
if err != nil {
t.Fatalf("list: %v", err)
}
if len(listResp.Snapshots) != 3 {
t.Fatalf("expected 3 snapshots, got %d", len(listResp.Snapshots))
}
// Delete snapshot 2.
_, err = s.ms.DeleteBlockSnapshot(ctx, &master_pb.DeleteBlockSnapshotRequest{
VolumeName: "snap-vol-2",
SnapshotId: 2,
})
if err != nil {
t.Fatalf("delete snap 2: %v", err)
}
// List: should have 2 (IDs 1 and 3).
listResp, err = s.ms.ListBlockSnapshots(ctx, &master_pb.ListBlockSnapshotsRequest{
VolumeName: "snap-vol-2",
})
if err != nil {
t.Fatalf("list after delete: %v", err)
}
if len(listResp.Snapshots) != 2 {
t.Fatalf("expected 2 snapshots after delete, got %d", len(listResp.Snapshots))
}
ids := map[uint32]bool{}
for _, snap := range listResp.Snapshots {
ids[snap.SnapshotId] = true
}
if !ids[1] || !ids[3] {
t.Fatalf("expected IDs {1,3}, got %v", ids)
}
if ids[2] {
t.Fatal("deleted snapshot 2 should not be in list")
}
t.Log("P11P1 list coherence: create 3 → delete 2 → list shows {1,3}")
}
// --- 3. Delete: remove + repeated delete idempotent ---
func TestP11P1_SnapshotDelete(t *testing.T) {
s := newSnapshotTestSetup(t)
s.createVolume(t, "snap-vol-3")
ctx := context.Background()
// Create and delete.
s.ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "snap-vol-3", SnapshotId: 10,
})
_, err := s.ms.DeleteBlockSnapshot(ctx, &master_pb.DeleteBlockSnapshotRequest{
VolumeName: "snap-vol-3", SnapshotId: 10,
})
if err != nil {
t.Fatalf("delete: %v", err)
}
// Repeated delete: should be idempotent (no error).
_, err = s.ms.DeleteBlockSnapshot(ctx, &master_pb.DeleteBlockSnapshotRequest{
VolumeName: "snap-vol-3", SnapshotId: 10,
})
if err != nil {
t.Fatalf("repeated delete should be idempotent, got: %v", err)
}
// List: empty.
listResp, _ := s.ms.ListBlockSnapshots(ctx, &master_pb.ListBlockSnapshotsRequest{
VolumeName: "snap-vol-3",
})
if len(listResp.Snapshots) != 0 {
t.Fatalf("expected 0 snapshots, got %d", len(listResp.Snapshots))
}
t.Log("P11P1 delete: create → delete → repeated delete idempotent → list empty")
}
// --- 4. Fail-closed: nonexistent volume, duplicate create ---
func TestP11P1_SnapshotFailClosed(t *testing.T) {
s := newSnapshotTestSetup(t)
s.createVolume(t, "snap-vol-4")
ctx := context.Background()
// Create on nonexistent volume.
_, err := s.ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "nonexistent-vol",
SnapshotId: 1,
})
if err == nil {
t.Fatal("create on nonexistent volume should fail")
}
// List on nonexistent volume.
_, err = s.ms.ListBlockSnapshots(ctx, &master_pb.ListBlockSnapshotsRequest{
VolumeName: "nonexistent-vol",
})
if err == nil {
t.Fatal("list on nonexistent volume should fail")
}
// Delete on nonexistent volume: idempotent (volume gone → snapshot gone).
_, err = s.ms.DeleteBlockSnapshot(ctx, &master_pb.DeleteBlockSnapshotRequest{
VolumeName: "nonexistent-vol",
SnapshotId: 1,
})
if err != nil {
t.Fatalf("delete on nonexistent volume should be idempotent, got: %v", err)
}
// Duplicate create (same ID twice).
s.ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "snap-vol-4", SnapshotId: 99,
})
_, err = s.ms.CreateBlockSnapshot(ctx, &master_pb.CreateBlockSnapshotRequest{
VolumeName: "snap-vol-4", SnapshotId: 99,
})
if err == nil {
t.Fatal("duplicate create should fail")
}
t.Log("P11P1 fail-closed: nonexistent vol, duplicate create all handled correctly")
}
@@ -0,0 +1,86 @@
package csi
import (
"context"
"log"
)
// ExportedControllerServer wraps controllerServer for cross-package testing.
type ExportedControllerServer struct {
*controllerServer
}
// NewExportedControllerServer creates a controller server with the given backend.
func NewExportedControllerServer(backend VolumeBackend) *ExportedControllerServer {
return &ExportedControllerServer{&controllerServer{backend: backend}}
}
// StagedInfo holds exported staged volume info for cross-package test assertions.
type StagedInfo struct {
IsLocal bool
ISCSIAddr string
IQN string
Transport string
}
// ExportedNodeServer wraps nodeServer for cross-package testing.
type ExportedNodeServer struct {
*nodeServer
}
// NewExportedNodeServer creates a node server with the given parameters.
// Pass mgr=nil to prove publish_context consumption (no local fallback).
// Uses NoopISCSI and NoopMount for cross-package test use.
func NewExportedNodeServer(nodeID string, mgr *VolumeManager, logger *log.Logger) *ExportedNodeServer {
return &ExportedNodeServer{&nodeServer{
mgr: mgr,
nodeID: nodeID,
iscsiUtil: &noopISCSI{},
mountUtil: &noopMount{},
logger: logger,
staged: map[string]*stagedVolumeInfo{},
}}
}
// GetStagedInfo returns exported staging info for a volume.
func (e *ExportedNodeServer) GetStagedInfo(volumeID string) *StagedInfo {
e.stagedMu.Lock()
defer e.stagedMu.Unlock()
info, ok := e.staged[volumeID]
if !ok {
return nil
}
return &StagedInfo{
IsLocal: info.isLocal,
ISCSIAddr: info.iscsiAddr,
IQN: info.iqn,
Transport: info.transport,
}
}
// noopISCSI implements ISCSIUtil with no real iSCSI operations.
type noopISCSI struct{}
func (n *noopISCSI) Discovery(ctx context.Context, portal string) error { return nil }
func (n *noopISCSI) Login(ctx context.Context, iqn, portal string) error { return nil }
func (n *noopISCSI) Logout(ctx context.Context, iqn string) error { return nil }
func (n *noopISCSI) GetDeviceByIQN(ctx context.Context, iqn string) (string, error) {
return "/dev/sda-noop", nil
}
func (n *noopISCSI) IsLoggedIn(ctx context.Context, iqn string) (bool, error) { return false, nil }
func (n *noopISCSI) RescanDevice(ctx context.Context, iqn string) error { return nil }
// noopMount implements MountUtil with no real mount operations.
type noopMount struct{}
func (n *noopMount) FormatAndMount(ctx context.Context, device, target, fsType string) error { return nil }
func (n *noopMount) Mount(ctx context.Context, source, target, fsType string, readOnly bool) error {
return nil
}
func (n *noopMount) BindMount(ctx context.Context, source, target string, readOnly bool) error {
return nil
}
func (n *noopMount) Unmount(ctx context.Context, target string) error { return nil }
func (n *noopMount) IsFormatted(ctx context.Context, device string) (bool, error) { return true, nil }
func (n *noopMount) IsMounted(ctx context.Context, target string) (bool, error) { return false, nil }
func (n *noopMount) ResizeFS(ctx context.Context, devicePath, mountPath, fsType string) error { return nil }