mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-26 01:44:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

ec: scrub a volume through its parity data (#11006)

Browse Source 
* Introduce a new `READS` scrub mode.

`READS` performs a full volume scrub but, unlike `FULL`, it will attempt to
reconstruct data for missing/damaged shard intervals from other shards in the cluster
when necessary.

The goal of this check is to ensure that EC volume contents _are readable by Seaweed_
even on a degraded storage state, by exercising parity data which is not read in `FULL`
mode. This is useful not only to validate data is user-readable, but also to detect potential
parity shard issues which may be difficult to pinpoint otherwise - particularly for older
volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs.

For regular volumes, this operation is equivalent to `FULL`.

Example:

```
> ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply
Live shard topology for volume ID 1 (14 shards):
	0@10.200.18.89:9001
	1@10.200.18.89:9002
	2@10.200.18.89:9003
	3@10.200.18.89:9004
	4@10.200.18.89:9005
	5@10.200.18.89:9006
	6@10.200.18.89:9007
	7@10.200.18.89:9008
	8@10.200.18.89:9009
	9@10.200.18.89:9013
	10@10.200.18.89:9010
	11@10.200.18.89:9011
	12@10.200.18.89:9012
	13@10.200.18.89:9020

Will unmount + delete 3 shard(s):
	0@10.200.18.89:9001
	3@10.200.18.89:9004
	11@10.200.18.89:9011

Unmounting shard 0@10.200.18.89:9001 for volume ID 1...
Deleting shard 0@10.200.18.89:9001 for volume ID 1...
Unmounting shard 3@10.200.18.89:9004 for volume ID 1...
Deleting shard 3@10.200.18.89:9004 for volume ID 1...
Unmounting shard 11@10.200.18.89:9011 for volume ID 1...
Deleting shard 11@10.200.18.89:9011 for volume ID 1...

All done!

> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full
using FULL mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes

Got scrub failures on 1 EC volumes and 1 EC shards :(
Affected volumes: 10.200.18.89:9002:1
Affected shards:  10.200.18.89:9002:1:0

> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads
using READS mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
```

* ec: report the shards a READS scrub had to rebuild

A READS scrub that recovers an interval was recording nothing, so a volume
missing three shards came back clean and nobody repaired it. The unreadable
shard is now recorded before the rebuild is attempted: READS reports the same
broken shards as FULL and differs only in whether the needles themselves
failed, which is the signal worth having - shards are gone, data is still
there.

forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the
RPC guard: it runs the same needle walk as FULL.

Regenerated the proto instead of hand-editing it, so the pancis typo (which
protoc-gen-go-grpc emits into eight other files here) and the header whitespace
stay as generated.

Mirrors into the Rust volume server, which also now honors
force_deleted_needles_check rather than hardcoding it off.

Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq

* ec: answer a deleted needle from a READS rebuild as deleted

#11020 gave the Rust recovery a deleted flag alongside its bytes, and it
answers a deleted needle with no bytes at all. The READS scrub appended that
empty answer, which does not compile against the new signature and, once it
did, would leave the needle short and report the size mismatch as damage.

Zero-fill the interval instead, the way the direct read beside it already
does: the assembled needle then reaches read_bytes as the delete-state
mismatch the walk already tolerates. Go takes the same branch off the flag
its recovery returns, rather than discarding it.

Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq

---------

Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
This commit is contained in:
Chris Lu
2026-08-28 16:42:34 -07:00
committed by GitHub
co-authored by Lisandro Pin
parent ba5b14b457
commit fcc2ea61d3
12 changed files with 375 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
weed/shell/command_ec_scrub.go
+7 -4
View File
@@ -41,10 +41,10 @@ func (c *commandEcVolumeScrub) Do(args []string, commandEnv *CommandEnv, writer
volScrubCommand := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
nodesStr := volScrubCommand.String("node", "", "comma-separated list of volume server <host>:<port> (optional)")
volumeIDsStr := volScrubCommand.String("volumeId", "", "comma-separated EC volume IDs to process (optional)")
mode := volScrubCommand.String("mode", "local", "scrubbing mode (index/local/full/checksum)")
mode := volScrubCommand.String("mode", "local", "scrubbing mode (index/local/full/reads/checksum)")
maxParallelization := volScrubCommand.Int("maxParallelization", DefaultMaxParallelization, "run up to X tasks in parallel, whenever possible")
showDetails := volScrubCommand.Bool("details", false, "display scrub result details, if available")
forceDeletedNeedlesCheck := volScrubCommand.Bool("forceDeletedNeedlesCheck", false, "force strict verification of deleted needles (full mode only); may report false positives when EC indexes disagree")
forceDeletedNeedlesCheck := volScrubCommand.Bool("forceDeletedNeedlesCheck", false, "force strict verification of deleted needles (full and reads modes only); may report false positives when EC indexes disagree")
if err = volScrubCommand.Parse(args); err != nil {
return err
@@ -97,12 +97,15 @@ func (c *commandEcVolumeScrub) Do(args []string, commandEnv *CommandEnv, writer
scrubMode = volume_server_pb.VolumeScrubMode_FULL
case "CHECKSUM":
scrubMode = volume_server_pb.VolumeScrubMode_CHECKSUM
case "READS":
scrubMode = volume_server_pb.VolumeScrubMode_READS
default:
return fmt.Errorf("unsupported scrubbing mode %q", *mode)
}
fmt.Fprintf(writer, "using %s mode\n", scrubMode.String())
if *forceDeletedNeedlesCheck && scrubMode != volume_server_pb.VolumeScrubMode_FULL {
return fmt.Errorf("deleted needle checks are only supported for FULL scrubs")
if *forceDeletedNeedlesCheck &&
scrubMode != volume_server_pb.VolumeScrubMode_FULL && scrubMode != volume_server_pb.VolumeScrubMode_READS {
return fmt.Errorf("deleted needle checks are only supported for FULL and READS scrubs")
}
return ec.ScrubEcVolumes(commandEnv.ecEnv(), writer, volumeServerAddrs, volumeIDs, scrubMode, *forceDeletedNeedlesCheck, *maxParallelization, *showDetails)
weed/shell/command_volume_scrub.go
+4 -1
View File
@@ -50,7 +50,7 @@ func (c *commandVolumeScrub) Do(args []string, commandEnv *CommandEnv, writer io
volScrubCommand := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
nodesStr := volScrubCommand.String("node", "", "comma-separated list of volume server <host>:<port> (optional)")
volumeIDsStr := volScrubCommand.String("volumeId", "", "comma-separated volume IDs to process (optional)")
mode := volScrubCommand.String("mode", "full", "scrubbing mode (index/local/full)")
mode := volScrubCommand.String("mode", "full", "scrubbing mode (index/local/full/reads)")
markBrokenReadonly := volScrubCommand.Bool("markBrokenReadonly", false, "whether to flag volumes with scrub failures as read-only")
maxParallelization := volScrubCommand.Int("maxParallelization", DefaultMaxParallelization, "run up to X tasks in parallel, whenever possible")
showDetails := volScrubCommand.Bool("details", false, "display scrub result details, if available")
@@ -99,6 +99,9 @@ func (c *commandVolumeScrub) Do(args []string, commandEnv *CommandEnv, writer io
c.mode = volume_server_pb.VolumeScrubMode_LOCAL
case "FULL":
c.mode = volume_server_pb.VolumeScrubMode_FULL
case "READS":
// only meaningful for EC volumes; accepted here so one mode name works on both commands
c.mode = volume_server_pb.VolumeScrubMode_READS
default:
return fmt.Errorf("unsupported scrubbing mode %q", *mode)
}