* Introduce a new `READS` scrub mode.
`READS` performs a full volume scrub but, unlike `FULL`, it will attempt to
reconstruct data for missing/damaged shard intervals from other shards in the cluster
when necessary.
The goal of this check is to ensure that EC volume contents _are readable by Seaweed_
even on a degraded storage state, by exercising parity data which is not read in `FULL`
mode. This is useful not only to validate data is user-readable, but also to detect potential
parity shard issues which may be difficult to pinpoint otherwise - particularly for older
volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs.
For regular volumes, this operation is equivalent to `FULL`.
Example:
```
> ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply
Live shard topology for volume ID 1 (14 shards):
0@10.200.18.89:9001
1@10.200.18.89:9002
2@10.200.18.89:9003
3@10.200.18.89:9004
4@10.200.18.89:9005
5@10.200.18.89:9006
6@10.200.18.89:9007
7@10.200.18.89:9008
8@10.200.18.89:9009
9@10.200.18.89:9013
10@10.200.18.89:9010
11@10.200.18.89:9011
12@10.200.18.89:9012
13@10.200.18.89:9020
Will unmount + delete 3 shard(s):
0@10.200.18.89:9001
3@10.200.18.89:9004
11@10.200.18.89:9011
Unmounting shard 0@10.200.18.89:9001 for volume ID 1...
Deleting shard 0@10.200.18.89:9001 for volume ID 1...
Unmounting shard 3@10.200.18.89:9004 for volume ID 1...
Deleting shard 3@10.200.18.89:9004 for volume ID 1...
Unmounting shard 11@10.200.18.89:9011 for volume ID 1...
Deleting shard 11@10.200.18.89:9011 for volume ID 1...
All done!
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full
using FULL mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
Got scrub failures on 1 EC volumes and 1 EC shards :(
Affected volumes: 10.200.18.89:9002:1
Affected shards: 10.200.18.89:9002:1:0
> ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads
using READS mode
Scrubbing 10.200.18.89:9002 (1/1)...
Scrubbed 6 EC files and 1 volumes on 1 nodes
```
* ec: report the shards a READS scrub had to rebuild
A READS scrub that recovers an interval was recording nothing, so a volume
missing three shards came back clean and nobody repaired it. The unreadable
shard is now recorded before the rebuild is attempted: READS reports the same
broken shards as FULL and differs only in whether the needles themselves
failed, which is the signal worth having - shards are gone, data is still
there.
forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the
RPC guard: it runs the same needle walk as FULL.
Regenerated the proto instead of hand-editing it, so the pancis typo (which
protoc-gen-go-grpc emits into eight other files here) and the header whitespace
stay as generated.
Mirrors into the Rust volume server, which also now honors
force_deleted_needles_check rather than hardcoding it off.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
* ec: answer a deleted needle from a READS rebuild as deleted
#11020 gave the Rust recovery a deleted flag alongside its bytes, and it
answers a deleted needle with no bytes at all. The READS scrub appended that
empty answer, which does not compile against the new signature and, once it
did, would leave the needle short and report the size mismatch as damage.
Zero-fill the interval instead, the way the direct read beside it already
does: the assembled needle then reaches read_bytes as the delete-state
mismatch the walk already tolerates. Go takes the same branch off the flag
its recovery returns, rather than discarding it.
Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq
---------
Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>
* shell: move ErrorWaitGroup to weed/util
* shell: remove unused CandidateEcNode and EcRack types
* ec: extract EC orchestration logic from weed/shell into weed/ec
Move the EC node/topology model, balance engine, encode pipeline, decode
pipeline, and rebuild engine into a new weed/ec package so shell commands
and maintenance workers can share the logic. Shell commands keep flag
parsing and delegate through a small ec.Env (dial option, topology fetch,
volume locations, lock check). Tests move along with the code.
* shell: remove unused proportional-rebalance type stubs
* ec: move scrub, replication check, and shard unmount engines into weed/ec
* worker: share the EC generation-aware shard counter from weed/ec
* ec: gofmt
* shell: drop EC aliases with no remaining callers
* ec: guard a missing topology hook and nil disk entries in topology helpers
* ec: drop trailing newlines from decode error strings
* ec: re-check the shell lock before applying shard unmounts
* shell: trim -node entries in ec.scrub
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell
FULL EC scrubs can opt into strict deleted-needle verification via the
-forceDeletedNeedlesCheck shell flag, off by default since it can report
false positives when EC indexes disagree. Rejected for non-FULL modes.
The Rust volume server parses the new field and ignores it: its FULL
scrub verifies shards via RS parity, not per-needle reads.
* volume: require admin auth for ScrubEcVolume
ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the
rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close
the gap so an EC scrub can't be triggered anonymously.
* shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode
Fail in the client before fanning out to every volume server, instead of
erroring halfway through once the servers reject the request.
On volumes failing scrubs, the detail output can get very verbose, which makes
reading results difficult. Most users won't care about this information to
begin with - just whether or not volumes pass scrub tests.
This MR gates the display of scrub result details behind a `--details` flag.
* ec: add EC bitrot checksum protobuf
EcBitrotProtection/EcShardChecksums/ChecksumAlgorithm sidecar messages,
copy_ecsum_file and unsafe_ignore_sidecar fields, and a CHECKSUM scrub mode.
* ec: bitrot checksum sidecar format, validation, and per-volume load
Per-shard CRC32C block checksums in an optional <base>.ecsum sidecar with a
self-integrity header; validation, rolling builder, backfill primitive, and
EcVolume load on mount + removal on destroy.
* ec: capture per-shard checksums at encode; verify-and-exclude on rebuild
WriteEcFilesWithContext returns the protection computed inline during encoding.
generateMissingEcFiles verifies present inputs against the sidecar, excludes
corrupt ones, regenerates in place, and re-verifies; fail-closed unless
unsafe_ignore_sidecar, removing all generated outputs on failure.
* ec: read-only checksum scrub with Reed-Solomon arbiter
ChecksumScrub verifies each local shard against the sidecar and reconstructs
flagged shards from the clean shards so stale-sidecar false positives are not
reported. Wired to the gRPC CHECKSUM mode and ec.scrub -mode checksum.
* ec: server-side bitrot sidecar write, copy, cleanup, and opportunistic backfill
Write .ecsum at fresh encode; propagate it with copy_ecsum_file (tolerant);
remove it on full delete and decode; rebuild honors unsafe_ignore_sidecar and
opportunistically backfills a sidecar when all shards are reachable.
* ec: volume server bitrot config flags
-ec.bitrotChecksum (default on) and -ec.bitrotBlockSizeMB (default 16).
* fix(ec_bitrot): bound -ec.bitrotBlockSizeMB before the int64 multiply
Validate the MiB value is in [1, 1024] before multiplying by 1 MiB, so a huge
flag value cannot overflow int64 and slip past the power-of-two check, and a
block size cannot collapse a sidecar to a few oversized blocks.
* fix(ec_bitrot): distribute the .ecsum sidecar from the worker encode path
The worker EC encode wrote the generation-0 sidecar locally but never added it
to shardFiles, so DistributeEcShards never shipped it and the distributed
holders came up unprotected. Append it to shardFiles and map the ecsum shard
type to its extension in the sender so it travels with the shards.
* fix(ec_bitrot): remove orphaned sidecars when the generation is gone
Gate sidecar removal on existingShardCount==0 alone rather than also requiring a
stray .ecx. A sidecar whose shards have all been deleted is orphaned and must be
removed even when no .ecx remains, or it leaks. .ecx/.ecj/.vif removal stays
gated on hasEcxFile as before.
* fix(ec_bitrot): do not fold checksum blocks scanned into TotalFiles
ChecksumScrub's first return is blocks scanned, not files. Discard it so the
scrub response's TotalFiles (a needle/file count) is not inflated by the block
count for CHECKSUM mode.
* test(ec_bitrot): clean up generated .ecsum sidecars in removeGeneratedFiles
* fix(ec_bitrot): reject an oversized sidecar payload before the uint32 cast
The header stores payload_len as a uint32; bound the payload before the
conversion so a pathological manifest cannot truncate the length field and
corrupt the sidecar. A real manifest is a few KB, so this never trips.
* fix(ec_bitrot): cap -ec.bitrotBlockSizeMB at 64 MiB
The block size becomes the per-shard scratch buffer the scrub/backfill path
allocates, so an over-large value (e.g. 1 GiB) is a memory hazard per concurrent
scrub worker. Lower the upper bound from 1024 to 64 MiB.
* fix(ec_bitrot): add -ecUnsafeIgnoreSidecar to weed tool fix -ecx
The -ecx recovery path reconstructs missing shards via RebuildEcFilesWithContext,
which fails closed on a malformed/stale .ecsum. Without an override flag an
operator could not complete the rebuild without manually deleting the sidecar.
Expose -ecUnsafeIgnoreSidecar (default false) and thread it through.
* fix(ec_bitrot): bound sidecar payload with a direct int constant; drop readFull
Guard len(payload) against a plain int constant (1 GiB) before the allocation
instead of a uint64 MaxUint32 compare, so the allocation-size value is provably
bounded (clears the CodeQL overflow alert) and the math import is no longer
needed. Inline os.File.ReadAt with io.EOF handling in verifyShardFileBlocks and
remove the now-redundant readFull helper (os.File.ReadAt fills the slice or
errors).
* test(ec_bitrot): use slices.Contains instead of a hand-rolled containsU32
* refactor(ec): fold the EcFiles WithContext variants into the base functions
RebuildEcFiles now takes the *ECContext directly (nil => derive from .vif as
before) and WriteEcFiles takes it too (nil => default), removing the parallel
RebuildEcFilesWithContext / WriteEcFilesWithContext names. Callers that had an
explicit context drop the WithContext suffix; the default-context callers pass
nil. No behavior change.
* refactor(ec): pass BackgroundECContext instead of nil to Write/RebuildEcFiles
Add a non-nil BackgroundECContext placeholder (analogous to context.Background())
and have callers with no specific layout pass it instead of a nil *ECContext.
WriteEcFiles resolves a zero/background context to the default ratio and
RebuildEcFiles resolves it from the .vif, so behavior is unchanged.
* fix(ec_bitrot): make BackgroundECContext a func; RebuildEcFiles fails closed on bad .vif
- BackgroundECContext is now a function returning a fresh *ECContext, so callers
cannot mutate a shared singleton or race on it (and it mirrors context.Background,
which is also a function).
- RebuildEcFiles now propagates the MaybeLoadVolumeInfo error: a present-but-
unreadable .vif fails closed instead of silently rebuilding with the default
ratio (which would corrupt a custom-ratio volume). Pass an explicit ctx to override.
