mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 15:04:37 +00:00
ec: uniform shard block layout (#10932)
* ec: uniform shard block layout An EC volume is striped as 1GiB blocks until less than one row remains, then 1MiB blocks, and consecutive blocks land on different shards. With ec.encode's -fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers. New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by the .ecsum manifest). A needle now maps to one shard unless it is larger than the block or straddles a boundary. The chosen size equals the legacy layout's padded shard length for every input, so shard sizes, capacity math, and the shard-size credibility checks are unchanged; only the byte placement moved. Reads, decode, and scrub resolve the block sizes from the volume's .vif; absence keeps the legacy interpretation, so existing EC volumes read exactly as before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the .vif, else the .ecsum sidecar, and with neither de-stripes under both candidate layouts and keeps the one that indexes more valid needles. Same change in the Rust volume server, which now also streams the encode in 256KB sub-batches like Go instead of allocating whole blocks, and computes the large-row count as shardSize/largeBlock to match Go on exact multiples. On a 26MB fixture both encoders produce byte-identical shards, and a Go-written .vif parses in Rust with the block size intact. * ec: resolve the rust ecx rebuild through the recorded layout The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat through a hand-rolled pure-1MiB striping, which was already wrong for legacy volumes with large-block rows and is wrong for any uniform volume with a block past 1MiB. Route the scan through locate_data with the .vif-recorded block size, the same mapping the read path uses. Also seed the new tests' random data instead of the deprecated global math/rand.Read. * ec: fail the Rust ecx rebuild on any shard read error A read error mid-scan published the entries collected so far as a successful .ecx, and read_at's byte count was ignored so a legal short read passed as complete — a truncated or failing shard could produce a silently incomplete recovery index. Exact-read semantics in read_from_data_shards, error propagation in the needle walk, and a truncated-shard regression test. * ec: fail the mount on an unreadable or malformed vif Both servers silently fell back to the legacy layout when an existing .vif could not be read or parsed. Every new encode records a positive uniform block size there, so the fallback mounted the same shards with legacy offset math and could return wrong data. Absent stays legal (legacy volumes predate the sidecar), and a zero-byte stub still reads as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust); a present-but-unreadable or malformed .vif fails the mount instead. * ec: bound the reconstruct fan-out of one needle's intervals A degraded interval fans out a read to every reachable shard location, each with a buffer the size of the interval. Reading a needle's intervals in parallel multiplied that by the interval concurrency: a needle spanning 8 blocks could hold 8 x MaxShardCount remote reads and buffers at once, where the sequential version peaked at MaxShardCount. Give each needle a single reconstruct budget its intervals share, held for the buffer's lifetime, so separate reads stay independent but one read cannot multiply its own fan-out. * ec: drop the duplicated shard-size formula calculateExpectedShardSize reimplemented the padding rule that UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize asserts the two agree for every input — so a change to the rule would have had to be made in both. Defer to the helper, keeping the historic answer for an empty .dat. * ec: resolve the shard block layout from whatever records it Four places still answered the layout question by inference when a record of it was available, or accepted an answer that was not one: - A mount with no .vif defaulted to the legacy layout; the bitrot sidecar records the same config at encode time, so take it when present, as weed fix -ecx already does. The vif itself is now parsed once per mount rather than twice. - The Rust ecx rebuild derived its row count from the padded shard extent, which under the legacy layout reads a shard that is an exact large-block multiple as one row too many. Pass the encode-time .dat size from the .vif and keep the extent as the fallback. - weed fix -ecx read the block size outside the EC-config guard (collapsing the unknown sentinel into a definitive legacy), only wrote the recovered layout back when the .vif was absent rather than unusable, and broke a scan tie by candidate order instead of the documented reach. - The uniform layout tripped writeDatFile's large-block ambiguity guard, which cannot apply when the large and small blocks are the same size. * ec: give the index-recovery tests a parseable vif The fixtures wrote the literal bytes "volinfo" as the source .vif and the recovery copies it verbatim, so the receiving server then mounted the volume from a .vif it could not parse. That used to pass by silently defaulting to the legacy layout; a mount now refuses a vif it cannot read, which is what the tests were exercising all along without meaning to. * ec: validate the layout a vif records, not just its syntax Review follow-ups on the mount-strictness change: - A .vif can parse and still record a block size no encoder could have produced (negative, or not a whole number of small blocks). Both servers took it and mapped every read through it. ValidateBlockSize / the Rust mirror now refuse the mount, the same way an unparseable vif does; 0 stays valid as the legacy two-tier layout. - The bitrot-sidecar fallback accepted parity_shards == 0 and summed the counts in their own width, so values near the ceiling wrapped past the MaxShardCount bound. Require both counts and sum in a wider type. - weed fix -ecx treated a config with only DataShards > 0 as usable, so a half-written .vif suppressed the recovery paths AND survived the rewrite. Require a complete, in-range config before trusting it. - Returning the vif-load error left the .ecx and .ecj descriptors open; repeated mount attempts on malformed metadata could exhaust them. * ec: refuse to act on a layout the metadata does not establish - The worker encode only logged a failed .vif write and skipped it in the distribution set, and treated the .ecsum write as best-effort. A worker whose disk filled after the much larger shards landed could still distribute, mount, verify shard inventory, and delete the source replicas — leaving holders with shards whose geometry nothing records. Both writes and both inclusions are encode success conditions now. - A generation-matching .ecsum that disagreed with the .vif geometry only disabled checksums in Go, and in Rust was not compared at all, so protection stayed On while reads used the other layout. Both files record the layout their generation was encoded with, so a disagreement now fails the mount. * ec: reject an invalid recorded block size in weed fix -ecx A .vif with valid shard counts but a negative or unaligned block size was marked usable: a positive invalid value pinned the scan to a geometry that de-stripes to garbage, and a negative one ran the dual scan but left the invalid .vif in place afterwards. Validate it with the same rule the mount applies, and when it fails leave the layout unknown so the scan recovers it and the file is rewritten. * ec: validate the sidecar layout weed fix -ecx recovers from The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid sidecar carrying the wrong generation, an incomplete ratio, or an unaligned block size would pin the reconstruction to one incorrect uniform-layout candidate instead of letting the dual scan decide. Require generation 0, a complete in-range ratio, and a valid block size; anything less leaves the layout unknown, which is the answer that still recovers by scanning. * ec: let only a genuinely absent sidecar choose the legacy layout With no .vif the bitrot sidecar is the only record of a volume's layout, and the mount fallback read a failed load, an unusable config, or a sidecar stamped for another generation as "assume legacy". A uniform generation-0 volume could therefore mount with legacy or another generation's geometry and answer reads with the wrong bytes. Present-but-unusable now fails the mount; only actual absence keeps the legacy defaults. Shared as EcShardConfigFromSidecar so every caller reads the sidecar the same way. * ec: treat a recorded-but-impossible layout as corruption, not as legacy - A .vif whose ecShardConfig is PRESENT but records an impossible ratio was answered with the default 10+4 and the legacy block layout, in both languages. That reads a uniform volume's shards at the wrong offsets and returns the wrong bytes. Only an entirely absent config still means "this predates the record"; a present one that cannot be true fails the mount. - The shard-count bound summed two uint32 counts as int, which wraps on a 32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site that checked a recorded ratio now goes through it. * ec: rebuild on the geometry the sidecar records, and flag it when it disagrees The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the layout itself — and it resolved a missing or invalid .vif to the default 10+4 with the legacy block size. Two consequences: a 12+4 volume was reconstructed through a 10+4 matrix, which produces wrong bytes and never regenerates shards 14-15; and the chosen geometry then contradicted a valid uniform sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping the input and regenerated-shard checksum checks precisely when the volume had already lost its metadata. The layout now resolves from the bitrot sidecar (found across the server's disks, not just beside the base name) before falling back to the defaults, and a present-but-impossible ratio fails instead of being replaced. A sidecar that contradicts the chosen geometry is BitrotInvalid, which the existing unsafeIgnoreSidecar override still lets an operator push past. * ec: let the Rust rebuild read metadata off a sibling disk read_ec_shard_config searches only the location the rebuild writes into, so a volume whose .vif or generation-0 .ecsum sits on another of the server's disks resolved to the default 10+4 with the legacy block layout — the Rust half of the geometry-guessing the Go rebuild just stopped doing. It then reconstructs a custom-ratio or uniform volume through the wrong Reed-Solomon matrix and de-striping geometry. The rebuild now looks for the .vif in its own location and then each sibling, falls back to the generation-0 sidecar wherever that lives, and only defaults when neither exists anywhere. The encode-time .dat size the ecx rebuild needs is resolved the same way. * ec: resolve a rebuild's vif from every directory that may hold it RebuildEcFiles probed only <data-base>.vif. The caller knows the selected location's index directory and the sibling locations, but passed neither for metadata: additionalDirs carried shard directories only, and were searched for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk holding only shards, therefore resolved a pre-sidecar custom-ratio volume to 10+4 and reconstructed through the wrong matrix — never regenerating shards 14-15. The caller now hands over the index and sibling directories, and the resolver probes the vif across all of them, matching what the Rust resolver already does for both the vif and the sidecar. * ec: make every rebuild consumer agree on the layout it resolved - The post-rebuild bitrot backfill re-derived the geometry from this directory's .vif alone and dropped the block size entirely, so a rebuild that resolved its layout from a sibling, the sidecar, or a uniform vif wrote a manifest describing a DIFFERENT layout — one later mounts reject, or that covers only the default shard count. The layout is resolved once now, through an exported ResolveRebuildECContext, and the rebuild and the backfill share that answer. - The Rust rebuild collected only each location's data directory, so a sibling's INDEX directory — where a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved to 10+4 with the legacy layout. Both directories of every location are carried now, deduped against the rebuild's own. - A shard delivery can bring the checksum manifest with it, but the receive path only writes the file: a server that already had the volume mounted kept its resolved protection state (off) until a remount. The mount RPC re-resolves it once the shards it describes have been added. * ec: cover the rebuild's directory search with tests Reviewers flagged the sibling index directory twice, and the fix that closed it had no test of its own: the assembly sat inline in the rebuild handler, reachable only through a gRPC call against a populated store. Lifting it into rebuildSearchDirs / select_rebuild_location makes the rule assertable — a sibling contributes BOTH its data and its index directory, a shared index directory is listed once, and the rebuild's own data directory never repeats. Writing the Rust cases surfaced that the two implementations do not agree on where the rebuild's own index directory belongs, and both are right: Go's resolver takes a single directory list, so that directory has to be inside it, while Rust's takes the rebuild's data and index directories as their own arguments and would search them twice. The tests now state which contract each side is holding to, so neither drifts into the other's shape. Pure refactor otherwise; no behaviour change. * ec: search the index directory for the layout sidecar The Rust resolver looked for the generation-0 .ecsum in the rebuild's data directory and the sibling list, but not in the rebuild's own index directory — while the .vif lookup directly above it did, and Go's findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx location that directory is where the metadata lives, and callers leave it out of the sibling list precisely because it is passed here separately, so nothing searched it. With no .vif anywhere the sidecar is the only surviving record of the layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy striping — the test added here fails with (10, 4, 0) against the old code — and the rebuild then reconstructs through the wrong matrix and writes .ecx offsets that no reader can follow. * ec: let the rebuild see its own index directory The Rust rebuild takes a single flat directory list — the shape Go's RebuildEcFiles uses — so it cannot be handed the rebuild location's index directory separately the way the layout resolvers are, and the handler was passing the sibling list, which deliberately omits exactly that directory. On a split -dir/-dir.idx location that is where .ecx and .vif live, so the shard and index lookups could not see them. Go has always carried that directory in additionalDirs; this lines the two call sites up. * ec: let a config-free vif fall through to the layout sidecar A .vif that carries no ecShardConfig answers nothing about the layout, so it is no more informative than an absent one — but both trees treated its mere existence as the end of the search. Go went straight to the 10+4 legacy defaults without consulting the sidecar at all; Rust returned whatever ec_shard_config_from could make of a single directory. A 12+4 uniform volume with a legacy config-free vif therefore resolved as 10+4 legacy, and every read landed at the wrong shard offset. The sidecar lookup was also single-directory on both sides, while a split -dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's findBitrotSidecar has always taken both bases; the callers here passed only the data base, and the Rust bitrot resolver derived its path from the data base alone. Rust's layout resolver now takes a candidate directory list — data, index, then any siblings — and searches all of it, which also removes the early return that made the vif's presence decisive. load_vif_info_across_dirs reported `dir` even when load_vif_info had found the vif in `dir_idx`. Nothing reads that field today, so this changes no behaviour; it stops the next caller that resolves the rest of the volume's metadata against the answer from being sent to a disk holding none of it. Absence stays legal throughout: a volume with neither record is genuinely legacy. Present-but-unusable still fails the mount, now in the config-free-vif branch too. * ec: activate a delivered sidecar on every per-disk runtime A vid mounts as one EcVolume per disk, each with its own resolved protection state, but the post-delivery reload used the first-match lookup and so touched exactly one of them. The siblings kept reporting no protection until a remount — and since shard distribution deduplicates the metadata files onto the first target disk for a node, the runtime that got the .ecsum is not necessarily the one the lookup returns. Iterate every runtime instead, via a new FindAllEcVolumes and its Rust mut equivalent. Combined with each runtime now resolving its sidecar against its index directory as well as its data directory, a server sharing one -dir.idx across its disks activates all of them from the single delivered copy. The Rust volume server had no post-mount reload at all; it gets one here, matching Go. * ec: resolve the delivered sidecar across every EC metadata directory Reloading every per-disk runtime, added last round, did not by itself make the delivered manifest reachable. Startup mirroring copies .ecx/.ecj/.vif to every shard-bearing disk so each mounts self-contained, but deliberately not .ecsum, and a repair delivers exactly one copy. Each runtime was resolving against its own two directories, so every sibling of the disk that received the file kept reporting no protection however often it reloaded. Resolve one authoritative copy across every EC metadata directory instead of duplicating the file. Mirroring .ecsum would have to keep pace with a file that is rewritten as shards are repaired, and would not help the reported case at all: the delivery happens at runtime, and mirroring only runs at startup. The regression test pins both halves — a reload restricted to the volume's own directories still finds nothing, and the same reload given the server's metadata directories turns protection on. * ec: ask every directory before writing a TOFU baseline After a rebuild the opportunistic backfill asks whether this volume already has a checksum manifest, and answered from the data base alone. A split -dir/-dir.idx layout keeps the sidecar with the index, and a multi-disk server may keep it on a sibling, so an existing manifest read as absent. The consequence is worse than a missed read. On a false "no" the backfill writes a fresh sidecar at the data base from whatever the shards say right now — and the data base is the first candidate every resolver checks, so that TOFU baseline shadows the real manifest rather than sitting beside it. A shard that was silently corrupt gets blessed, and the record that would have caught it stops being consulted. FindBitrotSidecar exports the search the package already used internally, so the question is asked of the data base, the index base and the sibling disks — the same candidates the rebuild resolves its layout from. * ec: refuse a shard block size no encoder could have produced weed fix -ecx derived one from the raw shard extent, so a truncated or partially copied shard wrote a .vif that NewEcVolume then permanently refuses — the volume the tool was run to rescue could never mount again. An extent that is not a whole number of small blocks cannot have come from a uniform encode, so it is no longer offered as a candidate, and nothing unvalidated reaches the .vif. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: derive the .vif's dat size and block size from one measurement VolumeEcShardsGenerate stat'ed the .dat before the encode while WriteEcFiles stat'ed it again to size the blocks. A write landing between the two produced a .vif whose own two fields describe different files. WriteEcFiles now leaves both on the context, and fills a placeholder context in place so the caller can read them back. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: keep the source volume until every holder serves its shard layout The uniform layout rides in a .vif field older volume servers never knew: they discard it, mount the shards as legacy and return wrong bytes with nothing erroring, and the shard files are the same length either way so no other check notices. The upgrade order lived only in the release note. VolumeEcShardsInfo now reports the block size the holder actually serves, in both the Go and Rust servers, and the pre-delete verification refuses to drop the source unless every reachable holder echoes the one the shards were encoded with — while a rollback still exists. A server that predates the field answers 0, which is the negative answer. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: drop the rebuild's dead block-size parameters generateMissingEcFiles never reads largeBlockSize/smallBlockSize — Reed-Solomon reconstruction is layout-agnostic — so passing the legacy constants only advertised a layout the rebuild does not use. Also move UniformBlockSize's doc off ValidateBlockSize. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: warn about EC defaults only when the mount used them The "vif file not found, using defaults" warning fired even after the bitrot sidecar supplied a non-default layout, sending anyone triaging wrong bytes after the legacy layout the volume never mounted on. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: stat the distributed bitrot sidecar once The strict check re-stat'ed the file immediately before the stat that already gates inclusion, and a failed sidecar write now fails the encode outright, so the first could only fire on a deletion between the two lines. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: say what the reconstruct budget actually bounds A shard's buffer stays in bufs until its interval reconstructs, which is after the read that filled it released its permit, so the semaphore bounds round trips in flight and not retained bytes. Peak memory is the intervals reconstructing at once times the shards each reaches times the interval size. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * test: let the fake volume server report its delivered EC layout The pre-delete verification now asks each holder which shard block layout it serves, and a fake that always answered "unset" looked exactly like a volume server too old to know the field. Distribution ships the .vif to every holder alongside its shards, so read the layout back out of it as a real holder does. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7
This commit is contained in:
Chris Lu
+167
-26
@@ -327,9 +327,15 @@ func doFixEcxFromShards(basePath, baseFileName, collection string, volumeId int6
|
||||
// Priority: explicit flags > existing .vif > defaults (10+4).
|
||||
vifName := base + ".vif"
|
||||
vifExists := util.FileExists(vifName)
|
||||
// Whether the on-disk .vif actually answers the layout question. An empty
|
||||
// stub (MaybeLoadVolumeInfo reads it as absent) or one with no EC config
|
||||
// exists but tells us nothing, and the recovered layout must still be
|
||||
// written back over it.
|
||||
vifUsable := false
|
||||
dataShards := erasure_coding.DataShardsCount
|
||||
parityShards := erasure_coding.ParityShardsCount
|
||||
var datFileSize int64
|
||||
blockSize := int64(-1) // the shard block layout; 0 legacy, >0 uniform, <0 unknown
|
||||
if vifExists {
|
||||
// MaybeLoadVolumeInfo returns a non-nil error when the .vif exists but
|
||||
// cannot be read or unmarshalled; fail loudly rather than silently
|
||||
@@ -338,13 +344,66 @@ func doFixEcxFromShards(basePath, baseFileName, collection string, volumeId int6
|
||||
fail(fmt.Errorf("volume %d: read %s: %w", volumeId, vifName, loadErr))
|
||||
return
|
||||
} else if found && vi != nil {
|
||||
if cfg := vi.GetEcShardConfig(); cfg != nil && cfg.GetDataShards() > 0 {
|
||||
// A partial config (parity 0, or counts past the shard ceiling)
|
||||
// describes no layout: leave the sentinel unknown and let the
|
||||
// sidecar / dual scan below answer, and rewrite the .vif at the end
|
||||
// rather than trusting it.
|
||||
cfg := vi.GetEcShardConfig()
|
||||
if cfg != nil && erasure_coding.ValidEcShardCounts(cfg.GetDataShards(), cfg.GetParityShards()) {
|
||||
dataShards = int(cfg.GetDataShards())
|
||||
parityShards = int(cfg.GetParityShards())
|
||||
// Only an EC config answers the layout question. Reading 0 off a
|
||||
// vif with no config would assert "legacy" and suppress both the
|
||||
// .ecsum fallback and the dual-layout scan below; leave the
|
||||
// sentinel at -1 (unknown) instead.
|
||||
// A recorded block size no encoder could have produced is not
|
||||
// an answer: a positive one would pin the scan to a geometry
|
||||
// that de-stripes to garbage, and a negative one would leave
|
||||
// the invalid .vif in place after the dual scan recovers the
|
||||
// real layout. Leave the sentinel unknown and rewrite the file
|
||||
// at the end.
|
||||
if bs := cfg.GetBlockSize(); erasure_coding.ValidateBlockSize(bs) == nil {
|
||||
blockSize = bs
|
||||
vifUsable = true
|
||||
} else {
|
||||
glog.Warningf("volume %d: %s records an invalid shard block size %d; recovering the layout by scan and rewriting it",
|
||||
volumeId, vifName, bs)
|
||||
}
|
||||
}
|
||||
datFileSize = vi.GetDatFileSize()
|
||||
}
|
||||
}
|
||||
// The bitrot sidecar records the same EC config at encode time; use it when
|
||||
// the .vif is gone. A uniform-layout volume cannot be de-striped correctly
|
||||
// without its block size.
|
||||
if blockSize < 0 {
|
||||
sidecarPath := erasure_coding.BitrotSidecarPath(base, 0)
|
||||
if prot, serr := erasure_coding.LoadBitrotSidecar(sidecarPath); serr == nil {
|
||||
// The sidecar is about to pin the reconstruction to one geometry
|
||||
// instead of letting the dual scan decide, so every field it
|
||||
// contributes has to hold up: it must describe generation 0 (the
|
||||
// shards being read), a complete in-range ratio, and a block size
|
||||
// an encoder could have produced. Anything less leaves the layout
|
||||
// unknown, which is the answer that still recovers by scanning.
|
||||
cfg := prot.GetEcShardConfig()
|
||||
ds, ps := int(cfg.GetDataShards()), int(cfg.GetParityShards())
|
||||
switch {
|
||||
case prot.GetGeneration() != 0:
|
||||
glog.Warningf("volume %d: %s records generation %d, not the generation-0 shards; ignoring it",
|
||||
volumeId, sidecarPath, prot.GetGeneration())
|
||||
case !erasure_coding.ValidEcShardCounts(cfg.GetDataShards(), cfg.GetParityShards()):
|
||||
glog.Warningf("volume %d: %s records invalid shard counts %d+%d; ignoring it",
|
||||
volumeId, sidecarPath, ds, ps)
|
||||
case erasure_coding.ValidateBlockSize(cfg.GetBlockSize()) != nil:
|
||||
glog.Warningf("volume %d: %s records an invalid shard block size %d; ignoring it",
|
||||
volumeId, sidecarPath, cfg.GetBlockSize())
|
||||
default:
|
||||
dataShards = ds
|
||||
parityShards = ps
|
||||
blockSize = cfg.GetBlockSize()
|
||||
}
|
||||
}
|
||||
}
|
||||
if *fixEcDataShards > 0 {
|
||||
dataShards = *fixEcDataShards
|
||||
}
|
||||
@@ -381,6 +440,9 @@ func doFixEcxFromShards(basePath, baseFileName, collection string, volumeId int6
|
||||
}
|
||||
if !dataComplete {
|
||||
ctx := &erasure_coding.ECContext{DataShards: dataShards, ParityShards: parityShards}
|
||||
if blockSize > 0 {
|
||||
ctx.BlockSize = blockSize
|
||||
}
|
||||
glog.Infof("volume %d: %d/%d shards present; reconstructing missing shards (%s) before index rebuild", volumeId, presentCount, dataShards+parityShards, ctx.String())
|
||||
if _, err := erasure_coding.RebuildEcFiles(base, ctx, *fixEcUnsafeIgnoreSidecar); err != nil {
|
||||
fail(fmt.Errorf("volume %d: reconstruct missing shards from %d survivors: %w", volumeId, presentCount, err))
|
||||
@@ -407,37 +469,114 @@ func doFixEcxFromShards(basePath, baseFileName, collection string, volumeId int6
|
||||
glog.V(0).Infof("volume %d: no .dat size in .vif; reconstructing padded .dat (%d bytes) from %d data shards", volumeId, reconstructSize, dataShards)
|
||||
}
|
||||
|
||||
// De-stripe the data shards into a temporary .dat next to the shards.
|
||||
// De-stripe the data shards into a temporary .dat next to the shards and
|
||||
// scan it into a fresh .ecx. With the layout unknown, de-stripe under both
|
||||
// candidate layouts and keep the one whose needle chain scans furthest.
|
||||
type layoutCandidate struct {
|
||||
name string
|
||||
large, small int64
|
||||
}
|
||||
var candidates []layoutCandidate
|
||||
switch {
|
||||
case blockSize > 0:
|
||||
candidates = []layoutCandidate{{"uniform", blockSize, blockSize}}
|
||||
case blockSize == 0:
|
||||
candidates = []layoutCandidate{{"legacy", erasure_coding.ErasureCodingLargeBlockSize, erasure_coding.ErasureCodingSmallBlockSize}}
|
||||
default:
|
||||
candidates = []layoutCandidate{
|
||||
{"legacy", erasure_coding.ErasureCodingLargeBlockSize, erasure_coding.ErasureCodingSmallBlockSize},
|
||||
}
|
||||
// A uniform-layout shard is exactly one block long, and every block
|
||||
// size an encoder can produce is a whole number of small blocks. An
|
||||
// extent that is not — a truncated or partially copied shard — could
|
||||
// not have come from a uniform encode, and recording it would write a
|
||||
// .vif that ValidateBlockSize refuses on the next mount: the volume
|
||||
// this tool was run to rescue would never open again.
|
||||
if erasure_coding.ValidateBlockSize(shardSize) == nil && shardSize > 0 {
|
||||
candidates = append(candidates, layoutCandidate{"uniform", shardSize, shardSize})
|
||||
glog.Infof("volume %d: no .vif or .ecsum records the shard block layout; trying both", volumeId)
|
||||
} else {
|
||||
glog.Infof("volume %d: no .vif or .ecsum records the shard block layout, and the %d-byte shard extent is not a whole number of %d-byte blocks; trying the legacy layout only",
|
||||
volumeId, shardSize, erasure_coding.ErasureCodingSmallBlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
tmpBase := base + ".ecxrecover"
|
||||
tmpDat := tmpBase + ".dat"
|
||||
if err := erasure_coding.WriteDatFile(tmpBase, reconstructSize, reconstructSize, shardFileNames); err != nil {
|
||||
os.Remove(tmpDat)
|
||||
fail(fmt.Errorf("volume %d: reconstruct .dat from data shards: %w", volumeId, err))
|
||||
defer os.Remove(tmpDat)
|
||||
bestIdx := -1
|
||||
var realDatSize, bestNeedles int64
|
||||
var version needle.Version
|
||||
var candErr error
|
||||
for i, cand := range candidates {
|
||||
if err := erasure_coding.WriteDatFile(tmpBase, reconstructSize, reconstructSize, shardFileNames, cand.large, cand.small); err != nil {
|
||||
candErr = fmt.Errorf("volume %d: reconstruct .dat (%s layout): %w", volumeId, cand.name, err)
|
||||
continue
|
||||
}
|
||||
size, needles, ver, err := writeEcxFromDat(tmpDat, ecxName+"."+cand.name)
|
||||
if err != nil {
|
||||
os.Remove(ecxName + "." + cand.name)
|
||||
candErr = fmt.Errorf("volume %d: build .ecx from reconstructed .dat (%s layout): %w", volumeId, cand.name, err)
|
||||
continue
|
||||
}
|
||||
// The wrong layout de-stripes to garbage past the first block
|
||||
// boundary, so the layout that indexes more valid needles wins.
|
||||
// On a tie — garbage bytes do sometimes parse as plausible sizes — the
|
||||
// layout whose needle chain reached further into the .dat wins, which is
|
||||
// the distance the scan actually validated.
|
||||
if bestIdx < 0 || needles > bestNeedles || (needles == bestNeedles && size > realDatSize) {
|
||||
bestIdx = i
|
||||
bestNeedles = needles
|
||||
realDatSize = size
|
||||
version = ver
|
||||
}
|
||||
}
|
||||
if bestIdx < 0 {
|
||||
fail(candErr)
|
||||
return
|
||||
}
|
||||
defer os.Remove(tmpDat)
|
||||
|
||||
realDatSize, version, err := writeEcxFromDat(tmpDat, ecxName)
|
||||
if err != nil {
|
||||
os.Remove(ecxName)
|
||||
fail(fmt.Errorf("volume %d: build .ecx from reconstructed .dat: %w", volumeId, err))
|
||||
for i := range candidates {
|
||||
if i != bestIdx {
|
||||
os.Remove(ecxName + "." + candidates[i].name)
|
||||
}
|
||||
}
|
||||
if err := os.Rename(ecxName+"."+candidates[bestIdx].name, ecxName); err != nil {
|
||||
fail(fmt.Errorf("volume %d: publish %s: %w", volumeId, ecxName, err))
|
||||
return
|
||||
}
|
||||
if blockSize < 0 {
|
||||
if candidates[bestIdx].name == "uniform" {
|
||||
blockSize = shardSize
|
||||
} else {
|
||||
blockSize = 0
|
||||
}
|
||||
glog.Infof("volume %d: %s layout indexes %d needles over %d bytes; keeping it", volumeId, candidates[bestIdx].name, bestNeedles, realDatSize)
|
||||
}
|
||||
glog.Infof("volume %d: wrote %s from %d data shards", volumeId, ecxName, dataShards)
|
||||
|
||||
// Regenerate the .vif when missing so the volume can mount and future
|
||||
// rebuilds know the EC ratio and original .dat size.
|
||||
if !vifExists {
|
||||
// Regenerate the .vif when it is missing OR unusable (an empty stub, or one
|
||||
// with no EC config): the layout just recovered by the dual scan is the only
|
||||
// record of it, and leaving the stub in place would mount the volume as
|
||||
// legacy on the next start and serve the wrong offsets.
|
||||
if !vifUsable {
|
||||
size := datFileSize
|
||||
if size <= 0 {
|
||||
size = realDatSize
|
||||
}
|
||||
// Never publish a layout the mount path will refuse: NewEcVolume runs
|
||||
// the same check and fails closed, so an unvalidated write here would
|
||||
// trade a recoverable volume for one that can no longer be opened.
|
||||
if bsErr := erasure_coding.ValidateBlockSize(blockSize); bsErr != nil {
|
||||
fail(fmt.Errorf("volume %d: refusing to write %s: %w", volumeId, vifName, bsErr))
|
||||
return
|
||||
}
|
||||
volumeInfo := &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(version),
|
||||
DatFileSize: size,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(dataShards),
|
||||
ParityShards: uint32(parityShards),
|
||||
BlockSize: blockSize,
|
||||
},
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(vifName, volumeInfo); err != nil {
|
||||
@@ -451,25 +590,26 @@ func doFixEcxFromShards(basePath, baseFileName, collection string, volumeId int6
|
||||
// writeEcxFromDat scans a (reconstructed) .dat and writes an ascending-sorted
|
||||
// .ecx containing only live needles — the same on-disk shape
|
||||
// WriteSortedFileFromIdx produces when an EC volume is first encoded. It returns
|
||||
// the physical .dat size (the offset where the EC zero padding begins) and the
|
||||
// volume version read from the superblock.
|
||||
func writeEcxFromDat(datPath, ecxPath string) (datFileSize int64, version needle.Version, err error) {
|
||||
// the physical .dat size (the offset where the EC zero padding begins), the
|
||||
// number of live needles indexed, and the volume version read from the
|
||||
// superblock.
|
||||
func writeEcxFromDat(datPath, ecxPath string) (datFileSize int64, liveNeedles int64, version needle.Version, err error) {
|
||||
f, err := os.OpenFile(datPath, os.O_RDONLY, 0644)
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("open %s: %w", datPath, err)
|
||||
return 0, 0, 0, fmt.Errorf("open %s: %w", datPath, err)
|
||||
}
|
||||
datBackend := backend.NewDiskFile(f)
|
||||
defer datBackend.Close()
|
||||
|
||||
superBlock, err := super_block.ReadSuperBlock(datBackend)
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("read superblock: %w", err)
|
||||
return 0, 0, 0, fmt.Errorf("read superblock: %w", err)
|
||||
}
|
||||
version = superBlock.Version
|
||||
|
||||
fileSize, _, err := datBackend.GetStat()
|
||||
if err != nil {
|
||||
return 0, version, fmt.Errorf("stat %s: %w", datPath, err)
|
||||
return 0, 0, version, fmt.Errorf("stat %s: %w", datPath, err)
|
||||
}
|
||||
|
||||
nm := needle_map.NewMemDb()
|
||||
@@ -482,7 +622,7 @@ func writeEcxFromDat(datPath, ecxPath string) (datFileSize int64, version needle
|
||||
if readErr == io.EOF {
|
||||
break
|
||||
}
|
||||
return 0, version, fmt.Errorf("read needle header at offset %d: %w", offset, readErr)
|
||||
return 0, 0, version, fmt.Errorf("read needle header at offset %d: %w", offset, readErr)
|
||||
}
|
||||
// EC encoding zero-pads the tail of the last block row. An all-zero
|
||||
// header marks the start of that padding, i.e. the end of real needles.
|
||||
@@ -491,13 +631,13 @@ func writeEcxFromDat(datPath, ecxPath string) (datFileSize int64, version needle
|
||||
}
|
||||
if n.Size.IsValid() {
|
||||
if pe := nm.Set(n.Id, types.ToOffset(offset), n.Size); pe != nil {
|
||||
return 0, version, fmt.Errorf("set needle %d: %w", n.Id, pe)
|
||||
return 0, 0, version, fmt.Errorf("set needle %d: %w", n.Id, pe)
|
||||
}
|
||||
} else {
|
||||
// Deleted/invalid: drop it so the .ecx carries only live entries,
|
||||
// matching the encode-time WriteSortedFileFromIdx behavior.
|
||||
if pe := nm.Delete(n.Id); pe != nil {
|
||||
return 0, version, fmt.Errorf("delete needle %d: %w", n.Id, pe)
|
||||
return 0, 0, version, fmt.Errorf("delete needle %d: %w", n.Id, pe)
|
||||
}
|
||||
}
|
||||
offset += types.NeedleHeaderSize + rest
|
||||
@@ -506,16 +646,17 @@ func writeEcxFromDat(datPath, ecxPath string) (datFileSize int64, version needle
|
||||
|
||||
ecxFile, err := os.OpenFile(ecxPath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0644)
|
||||
if err != nil {
|
||||
return 0, version, fmt.Errorf("open %s: %w", ecxPath, err)
|
||||
return 0, 0, version, fmt.Errorf("open %s: %w", ecxPath, err)
|
||||
}
|
||||
defer ecxFile.Close()
|
||||
|
||||
if err := nm.AscendingVisit(func(value needle_map.NeedleValue) error {
|
||||
liveNeedles++
|
||||
_, writeErr := ecxFile.Write(value.ToBytes())
|
||||
return writeErr
|
||||
}); err != nil {
|
||||
return 0, version, fmt.Errorf("write %s: %w", ecxPath, err)
|
||||
return 0, 0, version, fmt.Errorf("write %s: %w", ecxPath, err)
|
||||
}
|
||||
|
||||
return datFileSize, version, nil
|
||||
return datFileSize, liveNeedles, version, nil
|
||||
}
|
||||
|
||||
@@ -17,11 +17,12 @@ import (
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// buildAndEncodeTestEcVolume writes a small volume (.dat + .idx), EC-encodes it
|
||||
// into .ec00..ec13, and produces the canonical sorted .ecx. It returns the base
|
||||
// path, the canonical .ecx bytes, and the original .dat size. A couple of
|
||||
// needles are deleted so the .ecx must exclude them (live entries only).
|
||||
func buildAndEncodeTestEcVolume(t *testing.T, dir, baseName string) (base string, canonicalEcx []byte, origDatSize int64) {
|
||||
// buildAndEncodeTestEcVolume writes a volume of needleCount needles of about
|
||||
// needleSize bytes (.dat + .idx), EC-encodes it into .ec00..ec13, and produces
|
||||
// the canonical sorted .ecx. It returns the base path, the canonical .ecx
|
||||
// bytes, the original .dat size, and the encode's bitrot protection. A couple
|
||||
// of needles are deleted so the .ecx must exclude them (live entries only).
|
||||
func buildAndEncodeTestEcVolume(t *testing.T, dir, baseName string, needleCount, needleSize int) (base string, canonicalEcx []byte, origDatSize int64, prot *volume_server_pb.EcBitrotProtection) {
|
||||
t.Helper()
|
||||
base = filepath.Join(dir, baseName)
|
||||
version := needle.GetCurrentVersion()
|
||||
@@ -41,10 +42,10 @@ func buildAndEncodeTestEcVolume(t *testing.T, dir, baseName string) (base string
|
||||
}
|
||||
|
||||
nm := needle_map.NewMemDb()
|
||||
for i := uint64(1); i <= 12; i++ {
|
||||
for i := uint64(1); i <= uint64(needleCount); i++ {
|
||||
n := new(needle.Needle)
|
||||
n.Id = types.Uint64ToNeedleId(i)
|
||||
n.Data = make([]byte, 200+int(i))
|
||||
n.Data = make([]byte, needleSize+int(i))
|
||||
rand.Read(n.Data)
|
||||
n.Checksum = needle.NewCRC(n.Data)
|
||||
offset, _, _, err := n.Append(datBackend, version)
|
||||
@@ -92,7 +93,8 @@ func buildAndEncodeTestEcVolume(t *testing.T, dir, baseName string) (base string
|
||||
idxFile.Close()
|
||||
nm.Close()
|
||||
|
||||
if _, err := erasure_coding.WriteEcFiles(base, erasure_coding.BackgroundECContext()); err != nil {
|
||||
prot, err = erasure_coding.WriteEcFiles(base, erasure_coding.BackgroundECContext())
|
||||
if err != nil {
|
||||
t.Fatalf("WriteEcFiles: %v", err)
|
||||
}
|
||||
if err := erasure_coding.WriteSortedFileFromIdx(base, ".ecx"); err != nil {
|
||||
@@ -105,7 +107,7 @@ func buildAndEncodeTestEcVolume(t *testing.T, dir, baseName string) (base string
|
||||
if len(canonicalEcx) == 0 {
|
||||
t.Fatal("canonical .ecx is empty")
|
||||
}
|
||||
return base, canonicalEcx, origDatSize
|
||||
return base, canonicalEcx, origDatSize, prot
|
||||
}
|
||||
|
||||
// TestFixEcxFromShards verifies the .ecx and .vif are rebuilt purely from the
|
||||
@@ -121,7 +123,7 @@ func TestFixEcxFromShards(t *testing.T) {
|
||||
|
||||
dir := t.TempDir()
|
||||
const volumeId = 7
|
||||
base, canonical, origDatSize := buildAndEncodeTestEcVolume(t, dir, "7")
|
||||
base, canonical, origDatSize, _ := buildAndEncodeTestEcVolume(t, dir, "7", 12, 200)
|
||||
|
||||
// Disaster: keep only the shards.
|
||||
for _, ext := range []string{".ecx", ".ecj", ".idx", ".dat", ".vif"} {
|
||||
@@ -175,7 +177,7 @@ func TestFixEcxFromShardsWithVif(t *testing.T) {
|
||||
|
||||
dir := t.TempDir()
|
||||
const volumeId = 9
|
||||
base, canonical, origDatSize := buildAndEncodeTestEcVolume(t, dir, "9")
|
||||
base, canonical, origDatSize, _ := buildAndEncodeTestEcVolume(t, dir, "9", 12, 200)
|
||||
|
||||
// Write a .vif as the volume server would after encoding.
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
@@ -234,7 +236,7 @@ func TestFixEcxFromShardsMissingShards(t *testing.T) {
|
||||
|
||||
dir := t.TempDir()
|
||||
const volumeId = 11
|
||||
base, canonical, origDatSize := buildAndEncodeTestEcVolume(t, dir, "11")
|
||||
base, canonical, origDatSize, _ := buildAndEncodeTestEcVolume(t, dir, "11", 12, 200)
|
||||
|
||||
// Lose every index/metadata file plus three shards (two data: .ec02, .ec05;
|
||||
// one parity: .ec11), keeping 11 of 14 — enough to reconstruct. The highest
|
||||
@@ -271,3 +273,60 @@ func TestFixEcxFromShardsMissingShards(t *testing.T) {
|
||||
t.Fatalf("dat size = %d, want %d", vi.GetDatFileSize(), origDatSize)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFixEcxFromShardsUniformLayout loses the .vif of a volume big enough that
|
||||
// the uniform and legacy layouts place bytes differently, and verifies the
|
||||
// layout is recovered — from the bitrot sidecar when it survives, and by
|
||||
// de-striping under both candidate layouts when it does not.
|
||||
func TestFixEcxFromShardsUniformLayout(t *testing.T) {
|
||||
oldData, oldParity := *fixEcDataShards, *fixEcParityShards
|
||||
*fixEcDataShards, *fixEcParityShards = 0, 0
|
||||
*fixIgnoreError = true
|
||||
t.Cleanup(func() {
|
||||
*fixIgnoreError = false
|
||||
*fixEcDataShards, *fixEcParityShards = oldData, oldParity
|
||||
})
|
||||
|
||||
for _, withSidecar := range []bool{true, false} {
|
||||
name := "from-sidecar"
|
||||
if !withSidecar {
|
||||
name = "dual-scan"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
const volumeId = 13
|
||||
base, canonical, _, prot := buildAndEncodeTestEcVolume(t, dir, "13", 13, 2*1024*1024)
|
||||
if got := prot.GetEcShardConfig().GetBlockSize(); got <= erasure_coding.ErasureCodingSmallBlockSize {
|
||||
t.Fatalf("fixture block size %d does not diverge from the legacy layout", got)
|
||||
}
|
||||
if withSidecar {
|
||||
if err := erasure_coding.SaveBitrotSidecar(erasure_coding.BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, ext := range []string{".ecx", ".ecj", ".idx", ".dat", ".vif"} {
|
||||
if err := os.Remove(base + ext); err != nil && !os.IsNotExist(err) {
|
||||
t.Fatalf("remove %s: %v", base+ext, err)
|
||||
}
|
||||
}
|
||||
|
||||
doFixEcxFromShards(dir, "13", "", volumeId)
|
||||
|
||||
recovered, err := os.ReadFile(base + ".ecx")
|
||||
if err != nil {
|
||||
t.Fatalf("recovered .ecx not written: %v", err)
|
||||
}
|
||||
if !bytes.Equal(canonical, recovered) {
|
||||
t.Fatalf(".ecx mismatch: canonical %d bytes, recovered %d bytes", len(canonical), len(recovered))
|
||||
}
|
||||
vi, _, found, err := volume_info.MaybeLoadVolumeInfo(base + ".vif")
|
||||
if err != nil || !found {
|
||||
t.Fatalf(".vif not regenerated: found=%v err=%v", found, err)
|
||||
}
|
||||
if got, want := vi.GetEcShardConfig().GetBlockSize(), prot.GetEcShardConfig().GetBlockSize(); got != want {
|
||||
t.Fatalf("regenerated .vif block size = %d, want %d", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -541,6 +541,7 @@ message VolumeEcShardsInfoResponse {
|
||||
uint64 volume_size = 2;
|
||||
uint64 file_count = 3;
|
||||
uint64 file_deleted_count = 4;
|
||||
EcShardConfig ec_shard_config = 10; // the layout this holder serves reads through; a binary predating a field reports it as unset
|
||||
}
|
||||
|
||||
message EcShardInfo {
|
||||
@@ -615,6 +616,7 @@ message EcShardConfig {
|
||||
uint32 data_shards = 1; // Number of data shards (e.g., 10)
|
||||
uint32 parity_shards = 2; // Number of parity shards (e.g., 4)
|
||||
int64 encode_ts_ns = 3; // encode time (unix nanos); a read served from a shard of a different encode run is rejected
|
||||
int64 block_size = 4; // uniform block layout: each shard is a single contiguous block of this many bytes; 0 = legacy 1GiB/1MiB two-tier layout
|
||||
}
|
||||
|
||||
// EcBitrotProtection is the entire content of a bitrot checksum sidecar
|
||||
|
||||
@@ -4448,6 +4448,7 @@ type VolumeEcShardsInfoResponse struct {
|
||||
VolumeSize uint64 `protobuf:"varint,2,opt,name=volume_size,json=volumeSize,proto3" json:"volume_size,omitempty"`
|
||||
FileCount uint64 `protobuf:"varint,3,opt,name=file_count,json=fileCount,proto3" json:"file_count,omitempty"`
|
||||
FileDeletedCount uint64 `protobuf:"varint,4,opt,name=file_deleted_count,json=fileDeletedCount,proto3" json:"file_deleted_count,omitempty"`
|
||||
EcShardConfig *EcShardConfig `protobuf:"bytes,10,opt,name=ec_shard_config,json=ecShardConfig,proto3" json:"ec_shard_config,omitempty"` // the layout this holder serves reads through; a binary predating a field reports it as unset
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -4510,6 +4511,13 @@ func (x *VolumeEcShardsInfoResponse) GetFileDeletedCount() uint64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *VolumeEcShardsInfoResponse) GetEcShardConfig() *EcShardConfig {
|
||||
if x != nil {
|
||||
return x.EcShardConfig
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type EcShardInfo struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
ShardId uint32 `protobuf:"varint,1,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"`
|
||||
@@ -5145,6 +5153,7 @@ type EcShardConfig struct {
|
||||
DataShards uint32 `protobuf:"varint,1,opt,name=data_shards,json=dataShards,proto3" json:"data_shards,omitempty"` // Number of data shards (e.g., 10)
|
||||
ParityShards uint32 `protobuf:"varint,2,opt,name=parity_shards,json=parityShards,proto3" json:"parity_shards,omitempty"` // Number of parity shards (e.g., 4)
|
||||
EncodeTsNs int64 `protobuf:"varint,3,opt,name=encode_ts_ns,json=encodeTsNs,proto3" json:"encode_ts_ns,omitempty"` // encode time (unix nanos); a read served from a shard of a different encode run is rejected
|
||||
BlockSize int64 `protobuf:"varint,4,opt,name=block_size,json=blockSize,proto3" json:"block_size,omitempty"` // uniform block layout: each shard is a single contiguous block of this many bytes; 0 = legacy 1GiB/1MiB two-tier layout
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -5200,6 +5209,13 @@ func (x *EcShardConfig) GetEncodeTsNs() int64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *EcShardConfig) GetBlockSize() int64 {
|
||||
if x != nil {
|
||||
return x.BlockSize
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// EcBitrotProtection is the entire content of a bitrot checksum sidecar
|
||||
// (<base>.ecsum for the legacy generation, <base>.ecsum.v<N> for vacuum
|
||||
// generation N). On disk it is wrapped in a fixed header carrying a CRC32C
|
||||
@@ -7509,14 +7525,16 @@ const file_volume_server_proto_rawDesc = "" +
|
||||
"\tdisk_type\x18\x04 \x01(\tR\bdiskType\" \n" +
|
||||
"\x1eVolumeEcShardsToVolumeResponse\"8\n" +
|
||||
"\x19VolumeEcShardsInfoRequest\x12\x1b\n" +
|
||||
"\tvolume_id\x18\x01 \x01(\rR\bvolumeId\"\xcf\x01\n" +
|
||||
"\tvolume_id\x18\x01 \x01(\rR\bvolumeId\"\x98\x02\n" +
|
||||
"\x1aVolumeEcShardsInfoResponse\x12C\n" +
|
||||
"\x0eec_shard_infos\x18\x01 \x03(\v2\x1d.volume_server_pb.EcShardInfoR\fecShardInfos\x12\x1f\n" +
|
||||
"\vvolume_size\x18\x02 \x01(\x04R\n" +
|
||||
"volumeSize\x12\x1d\n" +
|
||||
"\n" +
|
||||
"file_count\x18\x03 \x01(\x04R\tfileCount\x12,\n" +
|
||||
"\x12file_deleted_count\x18\x04 \x01(\x04R\x10fileDeletedCount\"y\n" +
|
||||
"\x12file_deleted_count\x18\x04 \x01(\x04R\x10fileDeletedCount\x12G\n" +
|
||||
"\x0fec_shard_config\x18\n" +
|
||||
" \x01(\v2\x1f.volume_server_pb.EcShardConfigR\recShardConfig\"y\n" +
|
||||
"\vEcShardInfo\x12\x19\n" +
|
||||
"\bshard_id\x18\x01 \x01(\rR\ashardId\x12\x12\n" +
|
||||
"\x04size\x18\x02 \x01(\x03R\x04size\x12\x1e\n" +
|
||||
@@ -7583,13 +7601,15 @@ const file_volume_server_proto_rawDesc = "" +
|
||||
"\rexpire_at_sec\x18\x06 \x01(\x04R\vexpireAtSec\x12\x1b\n" +
|
||||
"\tread_only\x18\a \x01(\bR\breadOnly\x12G\n" +
|
||||
"\x0fec_shard_config\x18\b \x01(\v2\x1f.volume_server_pb.EcShardConfigR\recShardConfig\x12/\n" +
|
||||
"\x14read_only_can_delete\x18\t \x01(\bR\x11readOnlyCanDelete\"w\n" +
|
||||
"\x14read_only_can_delete\x18\t \x01(\bR\x11readOnlyCanDelete\"\x96\x01\n" +
|
||||
"\rEcShardConfig\x12\x1f\n" +
|
||||
"\vdata_shards\x18\x01 \x01(\rR\n" +
|
||||
"dataShards\x12#\n" +
|
||||
"\rparity_shards\x18\x02 \x01(\rR\fparityShards\x12 \n" +
|
||||
"\fencode_ts_ns\x18\x03 \x01(\x03R\n" +
|
||||
"encodeTsNs\"\xbc\x02\n" +
|
||||
"encodeTsNs\x12\x1d\n" +
|
||||
"\n" +
|
||||
"block_size\x18\x04 \x01(\x03R\tblockSize\"\xbc\x02\n" +
|
||||
"\x12EcBitrotProtection\x12A\n" +
|
||||
"\talgorithm\x18\x01 \x01(\x0e2#.volume_server_pb.ChecksumAlgorithmR\talgorithm\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -7958,133 +7978,134 @@ var file_volume_server_proto_depIdxs = []int32{
|
||||
2, // 3: volume_server_pb.SetStateResponse.state:type_name -> volume_server_pb.VolumeServerState
|
||||
48, // 4: volume_server_pb.ReceiveFileRequest.info:type_name -> volume_server_pb.ReceiveFileInfo
|
||||
82, // 5: volume_server_pb.VolumeEcShardsInfoResponse.ec_shard_infos:type_name -> volume_server_pb.EcShardInfo
|
||||
88, // 6: volume_server_pb.ReadVolumeFileStatusResponse.volume_info:type_name -> volume_server_pb.VolumeInfo
|
||||
87, // 7: volume_server_pb.VolumeInfo.files:type_name -> volume_server_pb.RemoteFile
|
||||
89, // 8: volume_server_pb.VolumeInfo.ec_shard_config:type_name -> volume_server_pb.EcShardConfig
|
||||
0, // 9: volume_server_pb.EcBitrotProtection.algorithm:type_name -> volume_server_pb.ChecksumAlgorithm
|
||||
89, // 10: volume_server_pb.EcBitrotProtection.ec_shard_config:type_name -> volume_server_pb.EcShardConfig
|
||||
91, // 11: volume_server_pb.EcBitrotProtection.shards:type_name -> volume_server_pb.EcShardChecksums
|
||||
87, // 12: volume_server_pb.OldVersionVolumeInfo.files:type_name -> volume_server_pb.RemoteFile
|
||||
85, // 13: volume_server_pb.VolumeServerStatusResponse.disk_statuses:type_name -> volume_server_pb.DiskStatus
|
||||
86, // 14: volume_server_pb.VolumeServerStatusResponse.memory_status:type_name -> volume_server_pb.MemStatus
|
||||
2, // 15: volume_server_pb.VolumeServerStatusResponse.state:type_name -> volume_server_pb.VolumeServerState
|
||||
113, // 16: volume_server_pb.FetchAndWriteNeedleRequest.replicas:type_name -> volume_server_pb.FetchAndWriteNeedleRequest.Replica
|
||||
122, // 17: volume_server_pb.FetchAndWriteNeedleRequest.remote_conf:type_name -> remote_pb.RemoteConf
|
||||
123, // 18: volume_server_pb.FetchAndWriteNeedleRequest.remote_location:type_name -> remote_pb.RemoteStorageLocation
|
||||
1, // 19: volume_server_pb.ScrubVolumeRequest.mode:type_name -> volume_server_pb.VolumeScrubMode
|
||||
1, // 20: volume_server_pb.ScrubEcVolumeRequest.mode:type_name -> volume_server_pb.VolumeScrubMode
|
||||
82, // 21: volume_server_pb.ScrubEcVolumeResponse.broken_shard_infos:type_name -> volume_server_pb.EcShardInfo
|
||||
114, // 22: volume_server_pb.QueryRequest.filter:type_name -> volume_server_pb.QueryRequest.Filter
|
||||
115, // 23: volume_server_pb.QueryRequest.input_serialization:type_name -> volume_server_pb.QueryRequest.InputSerialization
|
||||
116, // 24: volume_server_pb.QueryRequest.output_serialization:type_name -> volume_server_pb.QueryRequest.OutputSerialization
|
||||
117, // 25: volume_server_pb.QueryRequest.InputSerialization.csv_input:type_name -> volume_server_pb.QueryRequest.InputSerialization.CSVInput
|
||||
118, // 26: volume_server_pb.QueryRequest.InputSerialization.json_input:type_name -> volume_server_pb.QueryRequest.InputSerialization.JSONInput
|
||||
119, // 27: volume_server_pb.QueryRequest.InputSerialization.parquet_input:type_name -> volume_server_pb.QueryRequest.InputSerialization.ParquetInput
|
||||
120, // 28: volume_server_pb.QueryRequest.OutputSerialization.csv_output:type_name -> volume_server_pb.QueryRequest.OutputSerialization.CSVOutput
|
||||
121, // 29: volume_server_pb.QueryRequest.OutputSerialization.json_output:type_name -> volume_server_pb.QueryRequest.OutputSerialization.JSONOutput
|
||||
3, // 30: volume_server_pb.VolumeServer.BatchDelete:input_type -> volume_server_pb.BatchDeleteRequest
|
||||
7, // 31: volume_server_pb.VolumeServer.VacuumVolumeCheck:input_type -> volume_server_pb.VacuumVolumeCheckRequest
|
||||
9, // 32: volume_server_pb.VolumeServer.VacuumVolumeCompact:input_type -> volume_server_pb.VacuumVolumeCompactRequest
|
||||
11, // 33: volume_server_pb.VolumeServer.VacuumVolumeCommit:input_type -> volume_server_pb.VacuumVolumeCommitRequest
|
||||
13, // 34: volume_server_pb.VolumeServer.VacuumVolumeCleanup:input_type -> volume_server_pb.VacuumVolumeCleanupRequest
|
||||
15, // 35: volume_server_pb.VolumeServer.DeleteCollection:input_type -> volume_server_pb.DeleteCollectionRequest
|
||||
17, // 36: volume_server_pb.VolumeServer.AllocateVolume:input_type -> volume_server_pb.AllocateVolumeRequest
|
||||
19, // 37: volume_server_pb.VolumeServer.VolumeSyncStatus:input_type -> volume_server_pb.VolumeSyncStatusRequest
|
||||
21, // 38: volume_server_pb.VolumeServer.VolumeIncrementalCopy:input_type -> volume_server_pb.VolumeIncrementalCopyRequest
|
||||
23, // 39: volume_server_pb.VolumeServer.VolumeMount:input_type -> volume_server_pb.VolumeMountRequest
|
||||
25, // 40: volume_server_pb.VolumeServer.VolumeUnmount:input_type -> volume_server_pb.VolumeUnmountRequest
|
||||
27, // 41: volume_server_pb.VolumeServer.VolumeConsolidateIndex:input_type -> volume_server_pb.VolumeConsolidateIndexRequest
|
||||
29, // 42: volume_server_pb.VolumeServer.VolumeDelete:input_type -> volume_server_pb.VolumeDeleteRequest
|
||||
31, // 43: volume_server_pb.VolumeServer.VolumeMarkReadonly:input_type -> volume_server_pb.VolumeMarkReadonlyRequest
|
||||
33, // 44: volume_server_pb.VolumeServer.VolumeMarkWritable:input_type -> volume_server_pb.VolumeMarkWritableRequest
|
||||
35, // 45: volume_server_pb.VolumeServer.VolumeConfigure:input_type -> volume_server_pb.VolumeConfigureRequest
|
||||
37, // 46: volume_server_pb.VolumeServer.VolumeStatus:input_type -> volume_server_pb.VolumeStatusRequest
|
||||
39, // 47: volume_server_pb.VolumeServer.GetState:input_type -> volume_server_pb.GetStateRequest
|
||||
41, // 48: volume_server_pb.VolumeServer.SetState:input_type -> volume_server_pb.SetStateRequest
|
||||
43, // 49: volume_server_pb.VolumeServer.VolumeCopy:input_type -> volume_server_pb.VolumeCopyRequest
|
||||
83, // 50: volume_server_pb.VolumeServer.ReadVolumeFileStatus:input_type -> volume_server_pb.ReadVolumeFileStatusRequest
|
||||
45, // 51: volume_server_pb.VolumeServer.CopyFile:input_type -> volume_server_pb.CopyFileRequest
|
||||
47, // 52: volume_server_pb.VolumeServer.ReceiveFile:input_type -> volume_server_pb.ReceiveFileRequest
|
||||
50, // 53: volume_server_pb.VolumeServer.ReadNeedleBlob:input_type -> volume_server_pb.ReadNeedleBlobRequest
|
||||
52, // 54: volume_server_pb.VolumeServer.ReadNeedleMeta:input_type -> volume_server_pb.ReadNeedleMetaRequest
|
||||
54, // 55: volume_server_pb.VolumeServer.WriteNeedleBlob:input_type -> volume_server_pb.WriteNeedleBlobRequest
|
||||
56, // 56: volume_server_pb.VolumeServer.ReadAllNeedles:input_type -> volume_server_pb.ReadAllNeedlesRequest
|
||||
58, // 57: volume_server_pb.VolumeServer.VolumeTailSender:input_type -> volume_server_pb.VolumeTailSenderRequest
|
||||
60, // 58: volume_server_pb.VolumeServer.VolumeTailReceiver:input_type -> volume_server_pb.VolumeTailReceiverRequest
|
||||
62, // 59: volume_server_pb.VolumeServer.VolumeEcShardsGenerate:input_type -> volume_server_pb.VolumeEcShardsGenerateRequest
|
||||
64, // 60: volume_server_pb.VolumeServer.VolumeEcShardsRebuild:input_type -> volume_server_pb.VolumeEcShardsRebuildRequest
|
||||
66, // 61: volume_server_pb.VolumeServer.VolumeEcShardsCopy:input_type -> volume_server_pb.VolumeEcShardsCopyRequest
|
||||
68, // 62: volume_server_pb.VolumeServer.VolumeEcShardsDelete:input_type -> volume_server_pb.VolumeEcShardsDeleteRequest
|
||||
70, // 63: volume_server_pb.VolumeServer.VolumeEcShardsMount:input_type -> volume_server_pb.VolumeEcShardsMountRequest
|
||||
72, // 64: volume_server_pb.VolumeServer.VolumeEcShardsUnmount:input_type -> volume_server_pb.VolumeEcShardsUnmountRequest
|
||||
74, // 65: volume_server_pb.VolumeServer.VolumeEcShardRead:input_type -> volume_server_pb.VolumeEcShardReadRequest
|
||||
76, // 66: volume_server_pb.VolumeServer.VolumeEcBlobDelete:input_type -> volume_server_pb.VolumeEcBlobDeleteRequest
|
||||
78, // 67: volume_server_pb.VolumeServer.VolumeEcShardsToVolume:input_type -> volume_server_pb.VolumeEcShardsToVolumeRequest
|
||||
80, // 68: volume_server_pb.VolumeServer.VolumeEcShardsInfo:input_type -> volume_server_pb.VolumeEcShardsInfoRequest
|
||||
93, // 69: volume_server_pb.VolumeServer.VolumeTierMoveDatToRemote:input_type -> volume_server_pb.VolumeTierMoveDatToRemoteRequest
|
||||
95, // 70: volume_server_pb.VolumeServer.VolumeTierMoveDatFromRemote:input_type -> volume_server_pb.VolumeTierMoveDatFromRemoteRequest
|
||||
97, // 71: volume_server_pb.VolumeServer.VolumeServerStatus:input_type -> volume_server_pb.VolumeServerStatusRequest
|
||||
99, // 72: volume_server_pb.VolumeServer.VolumeServerLeave:input_type -> volume_server_pb.VolumeServerLeaveRequest
|
||||
101, // 73: volume_server_pb.VolumeServer.FetchAndWriteNeedle:input_type -> volume_server_pb.FetchAndWriteNeedleRequest
|
||||
103, // 74: volume_server_pb.VolumeServer.ScrubVolume:input_type -> volume_server_pb.ScrubVolumeRequest
|
||||
105, // 75: volume_server_pb.VolumeServer.ScrubEcVolume:input_type -> volume_server_pb.ScrubEcVolumeRequest
|
||||
107, // 76: volume_server_pb.VolumeServer.Query:input_type -> volume_server_pb.QueryRequest
|
||||
109, // 77: volume_server_pb.VolumeServer.VolumeNeedleStatus:input_type -> volume_server_pb.VolumeNeedleStatusRequest
|
||||
111, // 78: volume_server_pb.VolumeServer.Ping:input_type -> volume_server_pb.PingRequest
|
||||
4, // 79: volume_server_pb.VolumeServer.BatchDelete:output_type -> volume_server_pb.BatchDeleteResponse
|
||||
8, // 80: volume_server_pb.VolumeServer.VacuumVolumeCheck:output_type -> volume_server_pb.VacuumVolumeCheckResponse
|
||||
10, // 81: volume_server_pb.VolumeServer.VacuumVolumeCompact:output_type -> volume_server_pb.VacuumVolumeCompactResponse
|
||||
12, // 82: volume_server_pb.VolumeServer.VacuumVolumeCommit:output_type -> volume_server_pb.VacuumVolumeCommitResponse
|
||||
14, // 83: volume_server_pb.VolumeServer.VacuumVolumeCleanup:output_type -> volume_server_pb.VacuumVolumeCleanupResponse
|
||||
16, // 84: volume_server_pb.VolumeServer.DeleteCollection:output_type -> volume_server_pb.DeleteCollectionResponse
|
||||
18, // 85: volume_server_pb.VolumeServer.AllocateVolume:output_type -> volume_server_pb.AllocateVolumeResponse
|
||||
20, // 86: volume_server_pb.VolumeServer.VolumeSyncStatus:output_type -> volume_server_pb.VolumeSyncStatusResponse
|
||||
22, // 87: volume_server_pb.VolumeServer.VolumeIncrementalCopy:output_type -> volume_server_pb.VolumeIncrementalCopyResponse
|
||||
24, // 88: volume_server_pb.VolumeServer.VolumeMount:output_type -> volume_server_pb.VolumeMountResponse
|
||||
26, // 89: volume_server_pb.VolumeServer.VolumeUnmount:output_type -> volume_server_pb.VolumeUnmountResponse
|
||||
28, // 90: volume_server_pb.VolumeServer.VolumeConsolidateIndex:output_type -> volume_server_pb.VolumeConsolidateIndexResponse
|
||||
30, // 91: volume_server_pb.VolumeServer.VolumeDelete:output_type -> volume_server_pb.VolumeDeleteResponse
|
||||
32, // 92: volume_server_pb.VolumeServer.VolumeMarkReadonly:output_type -> volume_server_pb.VolumeMarkReadonlyResponse
|
||||
34, // 93: volume_server_pb.VolumeServer.VolumeMarkWritable:output_type -> volume_server_pb.VolumeMarkWritableResponse
|
||||
36, // 94: volume_server_pb.VolumeServer.VolumeConfigure:output_type -> volume_server_pb.VolumeConfigureResponse
|
||||
38, // 95: volume_server_pb.VolumeServer.VolumeStatus:output_type -> volume_server_pb.VolumeStatusResponse
|
||||
40, // 96: volume_server_pb.VolumeServer.GetState:output_type -> volume_server_pb.GetStateResponse
|
||||
42, // 97: volume_server_pb.VolumeServer.SetState:output_type -> volume_server_pb.SetStateResponse
|
||||
44, // 98: volume_server_pb.VolumeServer.VolumeCopy:output_type -> volume_server_pb.VolumeCopyResponse
|
||||
84, // 99: volume_server_pb.VolumeServer.ReadVolumeFileStatus:output_type -> volume_server_pb.ReadVolumeFileStatusResponse
|
||||
46, // 100: volume_server_pb.VolumeServer.CopyFile:output_type -> volume_server_pb.CopyFileResponse
|
||||
49, // 101: volume_server_pb.VolumeServer.ReceiveFile:output_type -> volume_server_pb.ReceiveFileResponse
|
||||
51, // 102: volume_server_pb.VolumeServer.ReadNeedleBlob:output_type -> volume_server_pb.ReadNeedleBlobResponse
|
||||
53, // 103: volume_server_pb.VolumeServer.ReadNeedleMeta:output_type -> volume_server_pb.ReadNeedleMetaResponse
|
||||
55, // 104: volume_server_pb.VolumeServer.WriteNeedleBlob:output_type -> volume_server_pb.WriteNeedleBlobResponse
|
||||
57, // 105: volume_server_pb.VolumeServer.ReadAllNeedles:output_type -> volume_server_pb.ReadAllNeedlesResponse
|
||||
59, // 106: volume_server_pb.VolumeServer.VolumeTailSender:output_type -> volume_server_pb.VolumeTailSenderResponse
|
||||
61, // 107: volume_server_pb.VolumeServer.VolumeTailReceiver:output_type -> volume_server_pb.VolumeTailReceiverResponse
|
||||
63, // 108: volume_server_pb.VolumeServer.VolumeEcShardsGenerate:output_type -> volume_server_pb.VolumeEcShardsGenerateResponse
|
||||
65, // 109: volume_server_pb.VolumeServer.VolumeEcShardsRebuild:output_type -> volume_server_pb.VolumeEcShardsRebuildResponse
|
||||
67, // 110: volume_server_pb.VolumeServer.VolumeEcShardsCopy:output_type -> volume_server_pb.VolumeEcShardsCopyResponse
|
||||
69, // 111: volume_server_pb.VolumeServer.VolumeEcShardsDelete:output_type -> volume_server_pb.VolumeEcShardsDeleteResponse
|
||||
71, // 112: volume_server_pb.VolumeServer.VolumeEcShardsMount:output_type -> volume_server_pb.VolumeEcShardsMountResponse
|
||||
73, // 113: volume_server_pb.VolumeServer.VolumeEcShardsUnmount:output_type -> volume_server_pb.VolumeEcShardsUnmountResponse
|
||||
75, // 114: volume_server_pb.VolumeServer.VolumeEcShardRead:output_type -> volume_server_pb.VolumeEcShardReadResponse
|
||||
77, // 115: volume_server_pb.VolumeServer.VolumeEcBlobDelete:output_type -> volume_server_pb.VolumeEcBlobDeleteResponse
|
||||
79, // 116: volume_server_pb.VolumeServer.VolumeEcShardsToVolume:output_type -> volume_server_pb.VolumeEcShardsToVolumeResponse
|
||||
81, // 117: volume_server_pb.VolumeServer.VolumeEcShardsInfo:output_type -> volume_server_pb.VolumeEcShardsInfoResponse
|
||||
94, // 118: volume_server_pb.VolumeServer.VolumeTierMoveDatToRemote:output_type -> volume_server_pb.VolumeTierMoveDatToRemoteResponse
|
||||
96, // 119: volume_server_pb.VolumeServer.VolumeTierMoveDatFromRemote:output_type -> volume_server_pb.VolumeTierMoveDatFromRemoteResponse
|
||||
98, // 120: volume_server_pb.VolumeServer.VolumeServerStatus:output_type -> volume_server_pb.VolumeServerStatusResponse
|
||||
100, // 121: volume_server_pb.VolumeServer.VolumeServerLeave:output_type -> volume_server_pb.VolumeServerLeaveResponse
|
||||
102, // 122: volume_server_pb.VolumeServer.FetchAndWriteNeedle:output_type -> volume_server_pb.FetchAndWriteNeedleResponse
|
||||
104, // 123: volume_server_pb.VolumeServer.ScrubVolume:output_type -> volume_server_pb.ScrubVolumeResponse
|
||||
106, // 124: volume_server_pb.VolumeServer.ScrubEcVolume:output_type -> volume_server_pb.ScrubEcVolumeResponse
|
||||
108, // 125: volume_server_pb.VolumeServer.Query:output_type -> volume_server_pb.QueriedStripe
|
||||
110, // 126: volume_server_pb.VolumeServer.VolumeNeedleStatus:output_type -> volume_server_pb.VolumeNeedleStatusResponse
|
||||
112, // 127: volume_server_pb.VolumeServer.Ping:output_type -> volume_server_pb.PingResponse
|
||||
79, // [79:128] is the sub-list for method output_type
|
||||
30, // [30:79] is the sub-list for method input_type
|
||||
30, // [30:30] is the sub-list for extension type_name
|
||||
30, // [30:30] is the sub-list for extension extendee
|
||||
0, // [0:30] is the sub-list for field type_name
|
||||
89, // 6: volume_server_pb.VolumeEcShardsInfoResponse.ec_shard_config:type_name -> volume_server_pb.EcShardConfig
|
||||
88, // 7: volume_server_pb.ReadVolumeFileStatusResponse.volume_info:type_name -> volume_server_pb.VolumeInfo
|
||||
87, // 8: volume_server_pb.VolumeInfo.files:type_name -> volume_server_pb.RemoteFile
|
||||
89, // 9: volume_server_pb.VolumeInfo.ec_shard_config:type_name -> volume_server_pb.EcShardConfig
|
||||
0, // 10: volume_server_pb.EcBitrotProtection.algorithm:type_name -> volume_server_pb.ChecksumAlgorithm
|
||||
89, // 11: volume_server_pb.EcBitrotProtection.ec_shard_config:type_name -> volume_server_pb.EcShardConfig
|
||||
91, // 12: volume_server_pb.EcBitrotProtection.shards:type_name -> volume_server_pb.EcShardChecksums
|
||||
87, // 13: volume_server_pb.OldVersionVolumeInfo.files:type_name -> volume_server_pb.RemoteFile
|
||||
85, // 14: volume_server_pb.VolumeServerStatusResponse.disk_statuses:type_name -> volume_server_pb.DiskStatus
|
||||
86, // 15: volume_server_pb.VolumeServerStatusResponse.memory_status:type_name -> volume_server_pb.MemStatus
|
||||
2, // 16: volume_server_pb.VolumeServerStatusResponse.state:type_name -> volume_server_pb.VolumeServerState
|
||||
113, // 17: volume_server_pb.FetchAndWriteNeedleRequest.replicas:type_name -> volume_server_pb.FetchAndWriteNeedleRequest.Replica
|
||||
122, // 18: volume_server_pb.FetchAndWriteNeedleRequest.remote_conf:type_name -> remote_pb.RemoteConf
|
||||
123, // 19: volume_server_pb.FetchAndWriteNeedleRequest.remote_location:type_name -> remote_pb.RemoteStorageLocation
|
||||
1, // 20: volume_server_pb.ScrubVolumeRequest.mode:type_name -> volume_server_pb.VolumeScrubMode
|
||||
1, // 21: volume_server_pb.ScrubEcVolumeRequest.mode:type_name -> volume_server_pb.VolumeScrubMode
|
||||
82, // 22: volume_server_pb.ScrubEcVolumeResponse.broken_shard_infos:type_name -> volume_server_pb.EcShardInfo
|
||||
114, // 23: volume_server_pb.QueryRequest.filter:type_name -> volume_server_pb.QueryRequest.Filter
|
||||
115, // 24: volume_server_pb.QueryRequest.input_serialization:type_name -> volume_server_pb.QueryRequest.InputSerialization
|
||||
116, // 25: volume_server_pb.QueryRequest.output_serialization:type_name -> volume_server_pb.QueryRequest.OutputSerialization
|
||||
117, // 26: volume_server_pb.QueryRequest.InputSerialization.csv_input:type_name -> volume_server_pb.QueryRequest.InputSerialization.CSVInput
|
||||
118, // 27: volume_server_pb.QueryRequest.InputSerialization.json_input:type_name -> volume_server_pb.QueryRequest.InputSerialization.JSONInput
|
||||
119, // 28: volume_server_pb.QueryRequest.InputSerialization.parquet_input:type_name -> volume_server_pb.QueryRequest.InputSerialization.ParquetInput
|
||||
120, // 29: volume_server_pb.QueryRequest.OutputSerialization.csv_output:type_name -> volume_server_pb.QueryRequest.OutputSerialization.CSVOutput
|
||||
121, // 30: volume_server_pb.QueryRequest.OutputSerialization.json_output:type_name -> volume_server_pb.QueryRequest.OutputSerialization.JSONOutput
|
||||
3, // 31: volume_server_pb.VolumeServer.BatchDelete:input_type -> volume_server_pb.BatchDeleteRequest
|
||||
7, // 32: volume_server_pb.VolumeServer.VacuumVolumeCheck:input_type -> volume_server_pb.VacuumVolumeCheckRequest
|
||||
9, // 33: volume_server_pb.VolumeServer.VacuumVolumeCompact:input_type -> volume_server_pb.VacuumVolumeCompactRequest
|
||||
11, // 34: volume_server_pb.VolumeServer.VacuumVolumeCommit:input_type -> volume_server_pb.VacuumVolumeCommitRequest
|
||||
13, // 35: volume_server_pb.VolumeServer.VacuumVolumeCleanup:input_type -> volume_server_pb.VacuumVolumeCleanupRequest
|
||||
15, // 36: volume_server_pb.VolumeServer.DeleteCollection:input_type -> volume_server_pb.DeleteCollectionRequest
|
||||
17, // 37: volume_server_pb.VolumeServer.AllocateVolume:input_type -> volume_server_pb.AllocateVolumeRequest
|
||||
19, // 38: volume_server_pb.VolumeServer.VolumeSyncStatus:input_type -> volume_server_pb.VolumeSyncStatusRequest
|
||||
21, // 39: volume_server_pb.VolumeServer.VolumeIncrementalCopy:input_type -> volume_server_pb.VolumeIncrementalCopyRequest
|
||||
23, // 40: volume_server_pb.VolumeServer.VolumeMount:input_type -> volume_server_pb.VolumeMountRequest
|
||||
25, // 41: volume_server_pb.VolumeServer.VolumeUnmount:input_type -> volume_server_pb.VolumeUnmountRequest
|
||||
27, // 42: volume_server_pb.VolumeServer.VolumeConsolidateIndex:input_type -> volume_server_pb.VolumeConsolidateIndexRequest
|
||||
29, // 43: volume_server_pb.VolumeServer.VolumeDelete:input_type -> volume_server_pb.VolumeDeleteRequest
|
||||
31, // 44: volume_server_pb.VolumeServer.VolumeMarkReadonly:input_type -> volume_server_pb.VolumeMarkReadonlyRequest
|
||||
33, // 45: volume_server_pb.VolumeServer.VolumeMarkWritable:input_type -> volume_server_pb.VolumeMarkWritableRequest
|
||||
35, // 46: volume_server_pb.VolumeServer.VolumeConfigure:input_type -> volume_server_pb.VolumeConfigureRequest
|
||||
37, // 47: volume_server_pb.VolumeServer.VolumeStatus:input_type -> volume_server_pb.VolumeStatusRequest
|
||||
39, // 48: volume_server_pb.VolumeServer.GetState:input_type -> volume_server_pb.GetStateRequest
|
||||
41, // 49: volume_server_pb.VolumeServer.SetState:input_type -> volume_server_pb.SetStateRequest
|
||||
43, // 50: volume_server_pb.VolumeServer.VolumeCopy:input_type -> volume_server_pb.VolumeCopyRequest
|
||||
83, // 51: volume_server_pb.VolumeServer.ReadVolumeFileStatus:input_type -> volume_server_pb.ReadVolumeFileStatusRequest
|
||||
45, // 52: volume_server_pb.VolumeServer.CopyFile:input_type -> volume_server_pb.CopyFileRequest
|
||||
47, // 53: volume_server_pb.VolumeServer.ReceiveFile:input_type -> volume_server_pb.ReceiveFileRequest
|
||||
50, // 54: volume_server_pb.VolumeServer.ReadNeedleBlob:input_type -> volume_server_pb.ReadNeedleBlobRequest
|
||||
52, // 55: volume_server_pb.VolumeServer.ReadNeedleMeta:input_type -> volume_server_pb.ReadNeedleMetaRequest
|
||||
54, // 56: volume_server_pb.VolumeServer.WriteNeedleBlob:input_type -> volume_server_pb.WriteNeedleBlobRequest
|
||||
56, // 57: volume_server_pb.VolumeServer.ReadAllNeedles:input_type -> volume_server_pb.ReadAllNeedlesRequest
|
||||
58, // 58: volume_server_pb.VolumeServer.VolumeTailSender:input_type -> volume_server_pb.VolumeTailSenderRequest
|
||||
60, // 59: volume_server_pb.VolumeServer.VolumeTailReceiver:input_type -> volume_server_pb.VolumeTailReceiverRequest
|
||||
62, // 60: volume_server_pb.VolumeServer.VolumeEcShardsGenerate:input_type -> volume_server_pb.VolumeEcShardsGenerateRequest
|
||||
64, // 61: volume_server_pb.VolumeServer.VolumeEcShardsRebuild:input_type -> volume_server_pb.VolumeEcShardsRebuildRequest
|
||||
66, // 62: volume_server_pb.VolumeServer.VolumeEcShardsCopy:input_type -> volume_server_pb.VolumeEcShardsCopyRequest
|
||||
68, // 63: volume_server_pb.VolumeServer.VolumeEcShardsDelete:input_type -> volume_server_pb.VolumeEcShardsDeleteRequest
|
||||
70, // 64: volume_server_pb.VolumeServer.VolumeEcShardsMount:input_type -> volume_server_pb.VolumeEcShardsMountRequest
|
||||
72, // 65: volume_server_pb.VolumeServer.VolumeEcShardsUnmount:input_type -> volume_server_pb.VolumeEcShardsUnmountRequest
|
||||
74, // 66: volume_server_pb.VolumeServer.VolumeEcShardRead:input_type -> volume_server_pb.VolumeEcShardReadRequest
|
||||
76, // 67: volume_server_pb.VolumeServer.VolumeEcBlobDelete:input_type -> volume_server_pb.VolumeEcBlobDeleteRequest
|
||||
78, // 68: volume_server_pb.VolumeServer.VolumeEcShardsToVolume:input_type -> volume_server_pb.VolumeEcShardsToVolumeRequest
|
||||
80, // 69: volume_server_pb.VolumeServer.VolumeEcShardsInfo:input_type -> volume_server_pb.VolumeEcShardsInfoRequest
|
||||
93, // 70: volume_server_pb.VolumeServer.VolumeTierMoveDatToRemote:input_type -> volume_server_pb.VolumeTierMoveDatToRemoteRequest
|
||||
95, // 71: volume_server_pb.VolumeServer.VolumeTierMoveDatFromRemote:input_type -> volume_server_pb.VolumeTierMoveDatFromRemoteRequest
|
||||
97, // 72: volume_server_pb.VolumeServer.VolumeServerStatus:input_type -> volume_server_pb.VolumeServerStatusRequest
|
||||
99, // 73: volume_server_pb.VolumeServer.VolumeServerLeave:input_type -> volume_server_pb.VolumeServerLeaveRequest
|
||||
101, // 74: volume_server_pb.VolumeServer.FetchAndWriteNeedle:input_type -> volume_server_pb.FetchAndWriteNeedleRequest
|
||||
103, // 75: volume_server_pb.VolumeServer.ScrubVolume:input_type -> volume_server_pb.ScrubVolumeRequest
|
||||
105, // 76: volume_server_pb.VolumeServer.ScrubEcVolume:input_type -> volume_server_pb.ScrubEcVolumeRequest
|
||||
107, // 77: volume_server_pb.VolumeServer.Query:input_type -> volume_server_pb.QueryRequest
|
||||
109, // 78: volume_server_pb.VolumeServer.VolumeNeedleStatus:input_type -> volume_server_pb.VolumeNeedleStatusRequest
|
||||
111, // 79: volume_server_pb.VolumeServer.Ping:input_type -> volume_server_pb.PingRequest
|
||||
4, // 80: volume_server_pb.VolumeServer.BatchDelete:output_type -> volume_server_pb.BatchDeleteResponse
|
||||
8, // 81: volume_server_pb.VolumeServer.VacuumVolumeCheck:output_type -> volume_server_pb.VacuumVolumeCheckResponse
|
||||
10, // 82: volume_server_pb.VolumeServer.VacuumVolumeCompact:output_type -> volume_server_pb.VacuumVolumeCompactResponse
|
||||
12, // 83: volume_server_pb.VolumeServer.VacuumVolumeCommit:output_type -> volume_server_pb.VacuumVolumeCommitResponse
|
||||
14, // 84: volume_server_pb.VolumeServer.VacuumVolumeCleanup:output_type -> volume_server_pb.VacuumVolumeCleanupResponse
|
||||
16, // 85: volume_server_pb.VolumeServer.DeleteCollection:output_type -> volume_server_pb.DeleteCollectionResponse
|
||||
18, // 86: volume_server_pb.VolumeServer.AllocateVolume:output_type -> volume_server_pb.AllocateVolumeResponse
|
||||
20, // 87: volume_server_pb.VolumeServer.VolumeSyncStatus:output_type -> volume_server_pb.VolumeSyncStatusResponse
|
||||
22, // 88: volume_server_pb.VolumeServer.VolumeIncrementalCopy:output_type -> volume_server_pb.VolumeIncrementalCopyResponse
|
||||
24, // 89: volume_server_pb.VolumeServer.VolumeMount:output_type -> volume_server_pb.VolumeMountResponse
|
||||
26, // 90: volume_server_pb.VolumeServer.VolumeUnmount:output_type -> volume_server_pb.VolumeUnmountResponse
|
||||
28, // 91: volume_server_pb.VolumeServer.VolumeConsolidateIndex:output_type -> volume_server_pb.VolumeConsolidateIndexResponse
|
||||
30, // 92: volume_server_pb.VolumeServer.VolumeDelete:output_type -> volume_server_pb.VolumeDeleteResponse
|
||||
32, // 93: volume_server_pb.VolumeServer.VolumeMarkReadonly:output_type -> volume_server_pb.VolumeMarkReadonlyResponse
|
||||
34, // 94: volume_server_pb.VolumeServer.VolumeMarkWritable:output_type -> volume_server_pb.VolumeMarkWritableResponse
|
||||
36, // 95: volume_server_pb.VolumeServer.VolumeConfigure:output_type -> volume_server_pb.VolumeConfigureResponse
|
||||
38, // 96: volume_server_pb.VolumeServer.VolumeStatus:output_type -> volume_server_pb.VolumeStatusResponse
|
||||
40, // 97: volume_server_pb.VolumeServer.GetState:output_type -> volume_server_pb.GetStateResponse
|
||||
42, // 98: volume_server_pb.VolumeServer.SetState:output_type -> volume_server_pb.SetStateResponse
|
||||
44, // 99: volume_server_pb.VolumeServer.VolumeCopy:output_type -> volume_server_pb.VolumeCopyResponse
|
||||
84, // 100: volume_server_pb.VolumeServer.ReadVolumeFileStatus:output_type -> volume_server_pb.ReadVolumeFileStatusResponse
|
||||
46, // 101: volume_server_pb.VolumeServer.CopyFile:output_type -> volume_server_pb.CopyFileResponse
|
||||
49, // 102: volume_server_pb.VolumeServer.ReceiveFile:output_type -> volume_server_pb.ReceiveFileResponse
|
||||
51, // 103: volume_server_pb.VolumeServer.ReadNeedleBlob:output_type -> volume_server_pb.ReadNeedleBlobResponse
|
||||
53, // 104: volume_server_pb.VolumeServer.ReadNeedleMeta:output_type -> volume_server_pb.ReadNeedleMetaResponse
|
||||
55, // 105: volume_server_pb.VolumeServer.WriteNeedleBlob:output_type -> volume_server_pb.WriteNeedleBlobResponse
|
||||
57, // 106: volume_server_pb.VolumeServer.ReadAllNeedles:output_type -> volume_server_pb.ReadAllNeedlesResponse
|
||||
59, // 107: volume_server_pb.VolumeServer.VolumeTailSender:output_type -> volume_server_pb.VolumeTailSenderResponse
|
||||
61, // 108: volume_server_pb.VolumeServer.VolumeTailReceiver:output_type -> volume_server_pb.VolumeTailReceiverResponse
|
||||
63, // 109: volume_server_pb.VolumeServer.VolumeEcShardsGenerate:output_type -> volume_server_pb.VolumeEcShardsGenerateResponse
|
||||
65, // 110: volume_server_pb.VolumeServer.VolumeEcShardsRebuild:output_type -> volume_server_pb.VolumeEcShardsRebuildResponse
|
||||
67, // 111: volume_server_pb.VolumeServer.VolumeEcShardsCopy:output_type -> volume_server_pb.VolumeEcShardsCopyResponse
|
||||
69, // 112: volume_server_pb.VolumeServer.VolumeEcShardsDelete:output_type -> volume_server_pb.VolumeEcShardsDeleteResponse
|
||||
71, // 113: volume_server_pb.VolumeServer.VolumeEcShardsMount:output_type -> volume_server_pb.VolumeEcShardsMountResponse
|
||||
73, // 114: volume_server_pb.VolumeServer.VolumeEcShardsUnmount:output_type -> volume_server_pb.VolumeEcShardsUnmountResponse
|
||||
75, // 115: volume_server_pb.VolumeServer.VolumeEcShardRead:output_type -> volume_server_pb.VolumeEcShardReadResponse
|
||||
77, // 116: volume_server_pb.VolumeServer.VolumeEcBlobDelete:output_type -> volume_server_pb.VolumeEcBlobDeleteResponse
|
||||
79, // 117: volume_server_pb.VolumeServer.VolumeEcShardsToVolume:output_type -> volume_server_pb.VolumeEcShardsToVolumeResponse
|
||||
81, // 118: volume_server_pb.VolumeServer.VolumeEcShardsInfo:output_type -> volume_server_pb.VolumeEcShardsInfoResponse
|
||||
94, // 119: volume_server_pb.VolumeServer.VolumeTierMoveDatToRemote:output_type -> volume_server_pb.VolumeTierMoveDatToRemoteResponse
|
||||
96, // 120: volume_server_pb.VolumeServer.VolumeTierMoveDatFromRemote:output_type -> volume_server_pb.VolumeTierMoveDatFromRemoteResponse
|
||||
98, // 121: volume_server_pb.VolumeServer.VolumeServerStatus:output_type -> volume_server_pb.VolumeServerStatusResponse
|
||||
100, // 122: volume_server_pb.VolumeServer.VolumeServerLeave:output_type -> volume_server_pb.VolumeServerLeaveResponse
|
||||
102, // 123: volume_server_pb.VolumeServer.FetchAndWriteNeedle:output_type -> volume_server_pb.FetchAndWriteNeedleResponse
|
||||
104, // 124: volume_server_pb.VolumeServer.ScrubVolume:output_type -> volume_server_pb.ScrubVolumeResponse
|
||||
106, // 125: volume_server_pb.VolumeServer.ScrubEcVolume:output_type -> volume_server_pb.ScrubEcVolumeResponse
|
||||
108, // 126: volume_server_pb.VolumeServer.Query:output_type -> volume_server_pb.QueriedStripe
|
||||
110, // 127: volume_server_pb.VolumeServer.VolumeNeedleStatus:output_type -> volume_server_pb.VolumeNeedleStatusResponse
|
||||
112, // 128: volume_server_pb.VolumeServer.Ping:output_type -> volume_server_pb.PingResponse
|
||||
80, // [80:129] is the sub-list for method output_type
|
||||
31, // [31:80] is the sub-list for method input_type
|
||||
31, // [31:31] is the sub-list for extension type_name
|
||||
31, // [31:31] is the sub-list for extension extendee
|
||||
0, // [0:31] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_volume_server_proto_init() }
|
||||
|
||||
@@ -122,9 +122,6 @@ func (vs *VolumeServer) VolumeEcShardsGenerate(ctx context.Context, req *volume_
|
||||
return nil, fmt.Errorf("WriteSortedFileFromIdx %s: %v", v.IndexFileName(), err)
|
||||
}
|
||||
|
||||
// snapshot .dat file size before encoding — must match what .ecx references
|
||||
datSize, _, _ := v.FileStat()
|
||||
|
||||
// write .ec00 ~ .ec[TotalShards-1] files using context
|
||||
ecBitrot, err := erasure_coding.WriteEcFiles(baseFileName, ecCtx)
|
||||
if err != nil {
|
||||
@@ -151,7 +148,10 @@ func (vs *VolumeServer) VolumeEcShardsGenerate(ctx context.Context, req *volume_
|
||||
}
|
||||
volumeInfo := &volume_server_pb.VolumeInfo{Version: uint32(v.Version())}
|
||||
volumeInfo.ExpireAtSec = expireAtSec
|
||||
volumeInfo.DatFileSize = int64(datSize)
|
||||
// The size the encode actually read, not a separate stat: a replica-sync
|
||||
// write can land between two stats of a live .dat, and the .vif would then
|
||||
// record a DatFileSize and a BlockSize describing different files.
|
||||
volumeInfo.DatFileSize = ecCtx.DatFileSize
|
||||
|
||||
// Validate EC configuration before saving to .vif
|
||||
if ecCtx.DataShards <= 0 || ecCtx.ParityShards <= 0 || ecCtx.Total() > erasure_coding.MaxShardCount {
|
||||
@@ -165,6 +165,7 @@ func (vs *VolumeServer) VolumeEcShardsGenerate(ctx context.Context, req *volume_
|
||||
DataShards: uint32(ecCtx.DataShards),
|
||||
ParityShards: uint32(ecCtx.ParityShards),
|
||||
EncodeTsNs: time.Now().UnixNano(),
|
||||
BlockSize: ecCtx.BlockSize,
|
||||
}
|
||||
glog.V(1).Infof("Saving EC config to .vif for volume %d: %d+%d (total: %d)",
|
||||
req.VolumeId, ecCtx.DataShards, ecCtx.ParityShards, ecCtx.Total())
|
||||
@@ -239,17 +240,22 @@ func (vs *VolumeServer) VolumeEcShardsRebuild(ctx context.Context, req *volume_s
|
||||
// On multi-disk servers, existing local shards may be on a different disk
|
||||
// than where copied shards were placed during ec.rebuild.
|
||||
rebuildDataDir := rebuildLocation.Directory
|
||||
var additionalDirs []string
|
||||
for _, otherLocation := range otherLocationsWithShards {
|
||||
additionalDirs = append(additionalDirs, otherLocation.Directory)
|
||||
}
|
||||
additionalDirs := rebuildSearchDirs(rebuildLocation, otherLocationsWithShards)
|
||||
|
||||
// Rebuild missing EC files, searching all disk locations for input shards.
|
||||
// Present input shards are verified against the bitrot sidecar (when present)
|
||||
// and corrupt ones are regenerated; unsafe_ignore_sidecar bypasses the guard.
|
||||
start := time.Now()
|
||||
dataBaseFileName := path.Join(rebuildDataDir, baseFileName)
|
||||
generatedShardIds, err := erasure_coding.RebuildEcFiles(dataBaseFileName, erasure_coding.BackgroundECContext(), req.UnsafeIgnoreSidecar, additionalDirs...)
|
||||
// Resolve the layout ONCE and use that same answer for the rebuild and for
|
||||
// the backfill below: a manifest describing these shards has to record the
|
||||
// geometry they were actually reconstructed with.
|
||||
rebuildCtx, resolveErr := erasure_coding.ResolveRebuildECContext(dataBaseFileName, erasure_coding.BackgroundECContext(), additionalDirs)
|
||||
if resolveErr != nil {
|
||||
recordEcRebuild("failure", time.Since(start))
|
||||
return nil, fmt.Errorf("resolve rebuild layout for %s: %v", dataBaseFileName, resolveErr)
|
||||
}
|
||||
generatedShardIds, err := erasure_coding.RebuildEcFiles(dataBaseFileName, rebuildCtx, req.UnsafeIgnoreSidecar, additionalDirs...)
|
||||
if err != nil {
|
||||
recordEcRebuild("failure", time.Since(start))
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: %v", dataBaseFileName, err)
|
||||
@@ -273,14 +279,19 @@ func (vs *VolumeServer) VolumeEcShardsRebuild(ctx context.Context, req *volume_s
|
||||
// blesses current bytes; ComputeProtectionFromShards refuses a partial
|
||||
// manifest, so a multi-server rebuild that cannot reach all shards just skips.
|
||||
if erasure_coding.BitrotProtectionEnabled {
|
||||
sidecarPath := erasure_coding.BitrotSidecarPath(dataBaseFileName, 0)
|
||||
if _, statErr := os.Stat(sidecarPath); os.IsNotExist(statErr) {
|
||||
ctx := erasure_coding.NewDefaultECContext("", 0)
|
||||
if vi, _, found, _ := volume_info.MaybeLoadVolumeInfo(dataBaseFileName + ".vif"); found && vi.EcShardConfig != nil {
|
||||
if ds, ps := int(vi.EcShardConfig.DataShards), int(vi.EcShardConfig.ParityShards); ds > 0 && ps > 0 && ds+ps <= erasure_coding.MaxShardCount {
|
||||
ctx = &erasure_coding.ECContext{DataShards: ds, ParityShards: ps}
|
||||
}
|
||||
}
|
||||
// "No sidecar yet" has to be asked of every place one could be, not
|
||||
// just this directory. A split -dir/-dir.idx layout keeps it with the
|
||||
// index and a sibling disk may hold it, and answering from the data
|
||||
// base alone would write a fresh TOFU baseline over a volume that
|
||||
// already has a manifest — blessing whatever the shards currently say
|
||||
// and shadowing the real record, since the data base is searched first.
|
||||
if erasure_coding.FindBitrotSidecar(0, dataBaseFileName, indexBaseFileName, additionalDirs...) == "" {
|
||||
sidecarPath := erasure_coding.BitrotSidecarPath(dataBaseFileName, 0)
|
||||
// The manifest must describe the shards as rebuilt, so it takes the
|
||||
// context the rebuild resolved — not a narrower re-derivation that
|
||||
// reads only this directory's .vif and drops the block size, which
|
||||
// records the legacy layout for shards written with a uniform one.
|
||||
ctx := rebuildCtx
|
||||
if prot, berr := erasure_coding.ComputeProtectionFromShards(dataBaseFileName, ctx, 0, additionalDirs); berr != nil {
|
||||
glog.V(2).Infof("bitrot backfill skipped for %s: %v", dataBaseFileName, berr)
|
||||
} else if werr := erasure_coding.SaveBitrotSidecar(sidecarPath, prot); werr != nil {
|
||||
@@ -710,6 +721,36 @@ func removeBitrotSidecars(baseFilename string) error {
|
||||
return firstErr
|
||||
}
|
||||
|
||||
// rebuildSearchDirs lists every directory besides the rebuild's own data
|
||||
// directory that a rebuild may have to read from. Shards are only half of it:
|
||||
// a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif with the index, and on a
|
||||
// multi-disk server the chosen disk may hold nothing but shards while the
|
||||
// volume's .vif or generation-0 .ecsum sits on a sibling. Miss those and the
|
||||
// layout resolution falls back to the default ratio and the legacy block
|
||||
// size, reconstructing through the wrong matrix. The rebuild's own INDEX
|
||||
// directory belongs here too — callers pass the data-directory base name, so
|
||||
// it is not otherwise searched. Empty and duplicate entries are dropped.
|
||||
func rebuildSearchDirs(rebuildLocation *storage.DiskLocation, otherLocations []*storage.DiskLocation) []string {
|
||||
var dirs []string
|
||||
appendDir := func(dir string) {
|
||||
if dir == "" || dir == rebuildLocation.Directory {
|
||||
return
|
||||
}
|
||||
for _, existing := range dirs {
|
||||
if existing == dir {
|
||||
return
|
||||
}
|
||||
}
|
||||
dirs = append(dirs, dir)
|
||||
}
|
||||
appendDir(rebuildLocation.IdxDirectory)
|
||||
for _, otherLocation := range otherLocations {
|
||||
appendDir(otherLocation.Directory)
|
||||
appendDir(otherLocation.IdxDirectory)
|
||||
}
|
||||
return dirs
|
||||
}
|
||||
|
||||
func checkEcVolumeStatus(bName string, location *storage.DiskLocation) (hasEcxFile bool, hasIdxFile bool, existingShardCount int, err error) {
|
||||
// check whether to delete the .ecx and .ecj file also
|
||||
fileInfos, err := os.ReadDir(location.Directory)
|
||||
@@ -782,6 +823,27 @@ func (vs *VolumeServer) VolumeEcShardsMount(ctx context.Context, req *volume_ser
|
||||
}
|
||||
}
|
||||
|
||||
// A shard delivery can bring the checksum manifest with it, but the receive
|
||||
// path only writes the file. When this server already had the volume
|
||||
// mounted, the EcVolume in memory keeps whatever protection state it
|
||||
// resolved at mount — off, for a volume whose sidecar arrives now — until a
|
||||
// remount. Re-resolve it here, where the shards it describes were added.
|
||||
// Every per-disk runtime, not just the first: a vid mounts as one EcVolume
|
||||
// per disk, the delivery lands the .ecsum on one of them, and the
|
||||
// first-match FindEcVolume would leave the siblings reporting no protection
|
||||
// until a remount. Each re-resolves against its own data and index base, so
|
||||
// a shared -dir.idx reaches all of them.
|
||||
//
|
||||
// Resolving across every EC metadata directory is what makes that reload
|
||||
// mean something. Startup mirroring gives each shard-bearing disk its own
|
||||
// .ecx/.ecj/.vif but deliberately not the sidecar, so a runtime restricted
|
||||
// to its own two directories would find nothing however often it reloaded.
|
||||
// One delivered copy, reachable from all of them.
|
||||
ecMetadataDirs := vs.store.EcMetadataDirs()
|
||||
for _, v := range vs.store.FindAllEcVolumes(needle.VolumeId(req.VolumeId)) {
|
||||
v.ReloadBitrotSidecar(ecMetadataDirs...)
|
||||
}
|
||||
|
||||
return &volume_server_pb.VolumeEcShardsMountResponse{}, nil
|
||||
}
|
||||
|
||||
@@ -1003,7 +1065,7 @@ func (vs *VolumeServer) VolumeEcShardsToVolume(ctx context.Context, req *volume_
|
||||
// boundary, so the layout must not be derived from datFileSize. WriteDatFile
|
||||
// infers the layout from the shard size when .vif does not record it.
|
||||
// write .dat file from .ec00 ~ .ec09 files
|
||||
if err := erasure_coding.WriteDatFile(dataBaseFileName, datFileSize, v.DatFileSize(), shardFileNames); err != nil {
|
||||
if err := erasure_coding.WriteDatFile(dataBaseFileName, datFileSize, v.DatFileSize(), shardFileNames, v.ECContext.LargeBlockSize(), v.ECContext.SmallBlockSize()); err != nil {
|
||||
return nil, fmt.Errorf("WriteDatFile %s: %v", dataBaseFileName, err)
|
||||
}
|
||||
|
||||
@@ -1179,11 +1241,25 @@ func (vs *VolumeServer) VolumeEcShardsInfo(ctx context.Context, req *volume_serv
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Report the layout this holder will actually serve reads through. It is
|
||||
// the only way a coordinator can tell a holder that understands the
|
||||
// uniform block layout from one that dropped the unknown .vif field on the
|
||||
// floor and mounted the volume as legacy.
|
||||
var ecShardConfig *volume_server_pb.EcShardConfig
|
||||
if primary.ECContext != nil {
|
||||
ecShardConfig = &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(primary.ECContext.DataShards),
|
||||
ParityShards: uint32(primary.ECContext.ParityShards),
|
||||
BlockSize: primary.ECContext.BlockSize,
|
||||
}
|
||||
}
|
||||
|
||||
res := &volume_server_pb.VolumeEcShardsInfoResponse{
|
||||
EcShardInfos: shardInfos,
|
||||
FileCount: files,
|
||||
FileDeletedCount: filesDeleted,
|
||||
VolumeSize: totalSize,
|
||||
EcShardConfig: ecShardConfig,
|
||||
}
|
||||
|
||||
return res, nil
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package weed_server
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage"
|
||||
)
|
||||
|
||||
func loc(dir, idxDir string) *storage.DiskLocation {
|
||||
return &storage.DiskLocation{Directory: dir, IdxDirectory: idxDir}
|
||||
}
|
||||
|
||||
// The rebuild reads its shards from one directory but resolves the volume's
|
||||
// layout — ratio and uniform block size — from the .vif or the generation-0
|
||||
// .ecsum, which on a multi-disk server may sit anywhere. Every directory that
|
||||
// could hold one has to be in the search list, or the resolution silently
|
||||
// falls back to 10+4 with the legacy striping and reconstructs through the
|
||||
// wrong matrix.
|
||||
func TestRebuildSearchDirs(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
rebuild *storage.DiskLocation
|
||||
others []*storage.DiskLocation
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
name: "the rebuild's own index directory is searched",
|
||||
rebuild: loc("/data1", "/idx1"),
|
||||
want: []string{"/idx1"},
|
||||
},
|
||||
{
|
||||
name: "an unsplit rebuild location contributes nothing",
|
||||
rebuild: loc("/data1", "/data1"),
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
// The case two reviewers flagged: a sibling holding only shards
|
||||
// while its index directory holds this volume's .vif.
|
||||
name: "a sibling's index directory is searched, not just its data directory",
|
||||
rebuild: loc("/data1", "/data1"),
|
||||
others: []*storage.DiskLocation{loc("/data2", "/idx2")},
|
||||
want: []string{"/data2", "/idx2"},
|
||||
},
|
||||
{
|
||||
name: "shared index directories are listed once",
|
||||
rebuild: loc("/data1", "/shared-idx"),
|
||||
others: []*storage.DiskLocation{loc("/data2", "/shared-idx"), loc("/data3", "/shared-idx")},
|
||||
want: []string{"/shared-idx", "/data2", "/data3"},
|
||||
},
|
||||
{
|
||||
name: "the rebuild's data directory is never repeated",
|
||||
rebuild: loc("/data1", "/idx1"),
|
||||
others: []*storage.DiskLocation{loc("/data1", "/idx1"), loc("/data2", "/data1")},
|
||||
want: []string{"/idx1", "/data2"},
|
||||
},
|
||||
{
|
||||
name: "empty directories are dropped",
|
||||
rebuild: loc("/data1", ""),
|
||||
others: []*storage.DiskLocation{loc("/data2", "")},
|
||||
want: []string{"/data2"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := rebuildSearchDirs(tc.rebuild, tc.others)
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("rebuildSearchDirs() = %v, want %v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,7 @@ import (
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
"github.com/seaweedfs/seaweedfs/weed/util"
|
||||
)
|
||||
|
||||
@@ -111,7 +112,12 @@ func TestFetchEcIndexFromPeers_CopiesIndexOverGrpc(t *testing.T) {
|
||||
if err := os.WriteFile(srcBase+".ecj", []byte("journal"), 0o644); err != nil {
|
||||
t.Fatalf("write source .ecj: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(srcBase+".vif", []byte("volinfo"), 0o644); err != nil {
|
||||
// A real .vif: the receiver MOUNTS the volume from the copied files, and a
|
||||
// mount refuses a .vif it cannot parse rather than guessing the layout.
|
||||
if err := volume_info.SaveVolumeInfo(srcBase+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 10, ParityShards: 4},
|
||||
}); err != nil {
|
||||
t.Fatalf("write source .vif: %v", err)
|
||||
}
|
||||
|
||||
@@ -223,7 +229,12 @@ func TestVolumeEcShardsMount_RecoverMissingIndex(t *testing.T) {
|
||||
if err := os.WriteFile(srcBase+".ecj", nil, 0o644); err != nil {
|
||||
t.Fatalf("write source .ecj: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(srcBase+".vif", []byte("volinfo"), 0o644); err != nil {
|
||||
// A real .vif: the receiver MOUNTS the volume from the copied files, and a
|
||||
// mount refuses a .vif it cannot parse rather than guessing the layout.
|
||||
if err := volume_info.SaveVolumeInfo(srcBase+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 10, ParityShards: 4},
|
||||
}); err != nil {
|
||||
t.Fatalf("write source .vif: %v", err)
|
||||
}
|
||||
srcGrpcPort := serveGrpc(t, func(s *grpc.Server) {
|
||||
|
||||
@@ -484,26 +484,17 @@ func (l *DiskLocation) checkOrphanedShards(shards []string, collection string, v
|
||||
// erasure_coding.DataShardsCount so that tests writing a custom layout
|
||||
// to .vif compute the matching shard size, and so custom-ratio builds
|
||||
// (e.g. enterprise) can swap the default without touching this helper.
|
||||
// The padded shard length is the same number under both layouts —
|
||||
// TestUniformBlockSizeMatchesLegacyShardSize asserts the equivalence for every
|
||||
// input — so defer to the encoder's own helper instead of keeping a second copy
|
||||
// of the padding rule that a future change would have to be made in twice. An
|
||||
// empty .dat keeps its historic answer: the legacy encoder emits no block for
|
||||
// it, where UniformBlockSize floors at one.
|
||||
func calculateExpectedShardSize(datFileSize int64, dataShardCount int) int64 {
|
||||
if dataShardCount <= 0 {
|
||||
if dataShardCount <= 0 || datFileSize <= 0 {
|
||||
return 0
|
||||
}
|
||||
var shardSize int64
|
||||
|
||||
// Process large blocks (1GB * dataShardCount per batch)
|
||||
largeBatchSize := int64(erasure_coding.ErasureCodingLargeBlockSize) * int64(dataShardCount)
|
||||
numLargeBatches := datFileSize / largeBatchSize
|
||||
shardSize = numLargeBatches * int64(erasure_coding.ErasureCodingLargeBlockSize)
|
||||
remainingSize := datFileSize - (numLargeBatches * largeBatchSize)
|
||||
|
||||
// Process remaining data in small blocks (1MB * dataShardCount per batch)
|
||||
if remainingSize > 0 {
|
||||
smallBatchSize := int64(erasure_coding.ErasureCodingSmallBlockSize) * int64(dataShardCount)
|
||||
numSmallBatches := (remainingSize + smallBatchSize - 1) / smallBatchSize // Ceiling division
|
||||
shardSize += numSmallBatches * int64(erasure_coding.ErasureCodingSmallBlockSize)
|
||||
}
|
||||
|
||||
return shardSize
|
||||
return erasure_coding.UniformBlockSize(datFileSize, dataShardCount)
|
||||
}
|
||||
|
||||
// validateEcVolume reports whether the EC files for (collection, vid) on this
|
||||
|
||||
@@ -195,6 +195,7 @@ func buildProtectionFromBuilders(ctx *ECContext, builders []*shardChecksumBuilde
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ctx.DataShards),
|
||||
ParityShards: uint32(ctx.ParityShards),
|
||||
BlockSize: ctx.BlockSize,
|
||||
},
|
||||
Shards: shards,
|
||||
EncodeUuid: NewEncodeUUID(),
|
||||
@@ -431,6 +432,7 @@ func ComputeProtectionFromShards(baseFileName string, ctx *ECContext, generation
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ctx.DataShards),
|
||||
ParityShards: uint32(ctx.ParityShards),
|
||||
BlockSize: ctx.BlockSize,
|
||||
},
|
||||
Shards: shards,
|
||||
EncodeUuid: NewEncodeUUID(),
|
||||
@@ -474,53 +476,141 @@ func (ev *EcVolume) BitrotProtection() (*volume_server_pb.EcBitrotProtection, Bi
|
||||
return ev.bitrot, ev.bitrotStatus
|
||||
}
|
||||
|
||||
// ReloadBitrotSidecar re-resolves the checksum sidecar for a volume that is
|
||||
// already mounted — a shard delivery can bring the manifest with it, and the
|
||||
// receive path only writes the file, so without this the in-memory volume
|
||||
// keeps the protection state it resolved at mount (off) until a remount.
|
||||
func (ev *EcVolume) ReloadBitrotSidecar(additionalDirs ...string) {
|
||||
if err := ev.loadActiveBitrotSidecar(additionalDirs...); err != nil {
|
||||
glog.Warningf("ec volume %d: reload bitrot sidecar: %v", ev.VolumeId, err)
|
||||
}
|
||||
}
|
||||
|
||||
// loadActiveBitrotSidecar loads the generation-0 checksum sidecar into the
|
||||
// volume. Best-effort: any failure leaves protection off/invalid without
|
||||
// failing the mount. OSS only produces generation-0 (fresh-encode) sidecars.
|
||||
func (ev *EcVolume) loadActiveBitrotSidecar() {
|
||||
ev.loadBitrotForGeneration(0)
|
||||
func (ev *EcVolume) loadActiveBitrotSidecar(additionalDirs ...string) error {
|
||||
return ev.loadBitrotForGeneration(0, additionalDirs...)
|
||||
}
|
||||
|
||||
// loadBitrotForGeneration loads and validates the sidecar describing generation
|
||||
// `generation`, setting ev.bitrot/ev.bitrotStatus. Called at mount. Absent or
|
||||
// generation/config-mismatched => BitrotOff; self-integrity/manifest failure =>
|
||||
// generation-mismatched => BitrotOff; self-integrity/manifest failure =>
|
||||
// BitrotInvalid; usable => BitrotOn.
|
||||
func (ev *EcVolume) loadBitrotForGeneration(generation uint32) {
|
||||
//
|
||||
// A sidecar written FOR THIS generation that disagrees with the volume's shard
|
||||
// geometry is different in kind from those: both files record the layout the
|
||||
// generation was encoded with, so a disagreement means one of them is wrong and
|
||||
// reads through the other would land at the wrong shard offsets. That returns
|
||||
// an error and fails the mount instead of quietly dropping to unprotected
|
||||
// reads.
|
||||
func (ev *EcVolume) loadBitrotForGeneration(generation uint32, additionalDirs ...string) error {
|
||||
ev.bitrotLock.Lock()
|
||||
defer ev.bitrotLock.Unlock()
|
||||
ev.bitrot = nil
|
||||
ev.bitrotStatus = BitrotOff
|
||||
|
||||
if ev.ECContext == nil {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
path := findBitrotSidecar(generation, ev.DataBaseFileName(), ev.IndexBaseFileName())
|
||||
path := findBitrotSidecar(generation, ev.DataBaseFileName(), ev.IndexBaseFileName(), additionalDirs...)
|
||||
if path == "" {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
prot, err := LoadBitrotSidecar(path)
|
||||
if err != nil {
|
||||
glog.Warningf("ec volume %d: bitrot sidecar %s self-integrity failed: %v", ev.VolumeId, path, err)
|
||||
ev.bitrotStatus = BitrotInvalid
|
||||
return
|
||||
return nil
|
||||
}
|
||||
if prot.Generation != generation {
|
||||
return // not for this generation -> off, not corruption
|
||||
return nil // not for this generation -> off, not corruption
|
||||
}
|
||||
if prot.EcShardConfig == nil ||
|
||||
int(prot.EcShardConfig.DataShards) != ev.ECContext.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ev.ECContext.ParityShards {
|
||||
return
|
||||
if prot.EcShardConfig == nil {
|
||||
return nil // records no geometry -> nothing to contradict
|
||||
}
|
||||
if int(prot.EcShardConfig.DataShards) != ev.ECContext.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ev.ECContext.ParityShards ||
|
||||
prot.EcShardConfig.BlockSize != ev.ECContext.BlockSize {
|
||||
return fmt.Errorf("ec volume %d generation %d: %s records layout %d+%d block %d but the volume is mounted as %d+%d block %d; refusing to serve one of the two layouts",
|
||||
ev.VolumeId, generation, path,
|
||||
prot.EcShardConfig.DataShards, prot.EcShardConfig.ParityShards, prot.EcShardConfig.BlockSize,
|
||||
ev.ECContext.DataShards, ev.ECContext.ParityShards, ev.ECContext.BlockSize)
|
||||
}
|
||||
if err := ValidateBitrotManifest(prot, ev.ECContext.DataShards, ev.ECContext.ParityShards); err != nil {
|
||||
glog.Warningf("ec volume %d: bitrot sidecar %s manifest invalid: %v", ev.VolumeId, path, err)
|
||||
ev.bitrotStatus = BitrotInvalid
|
||||
return
|
||||
return nil
|
||||
}
|
||||
ev.bitrot = prot
|
||||
ev.bitrotStatus = BitrotOn
|
||||
glog.V(1).Infof("ec volume %d: loaded bitrot protection generation %d (%d shards, block_size %d)",
|
||||
ev.VolumeId, generation, len(prot.Shards), prot.BlockSize)
|
||||
return nil
|
||||
}
|
||||
|
||||
// layoutFromSidecar resolves a volume's EC layout from its generation-0
|
||||
// checksum sidecar, searching the data base and then the index base. A split
|
||||
// -dir/-dir.idx layout keeps the sidecar with the index, so probing the data
|
||||
// base alone reports "absent" for a volume that has the record right there —
|
||||
// and absent is the one answer that selects the legacy 10+4 layout.
|
||||
//
|
||||
// The three answers stay distinct: (nil, false, nil) means genuinely absent
|
||||
// and the caller may fall back to the defaults; a non-nil error means the
|
||||
// record exists but cannot establish the layout, which must fail rather than
|
||||
// default; otherwise the config is usable.
|
||||
func layoutFromSidecar(dataBaseFileName, indexBaseFileName string) (cfg *volume_server_pb.EcShardConfig, found bool, err error) {
|
||||
path := findBitrotSidecar(0, dataBaseFileName, indexBaseFileName)
|
||||
if path == "" {
|
||||
return nil, false, nil
|
||||
}
|
||||
cfg, err = EcShardConfigFromSidecarPath(path)
|
||||
return cfg, true, err
|
||||
}
|
||||
|
||||
// EcShardConfigFromSidecar reads the generation-0 bitrot sidecar's record of a
|
||||
// volume's EC config. The three answers are distinct on purpose: absent means
|
||||
// the volume may genuinely predate the sidecar, which is the only case a caller
|
||||
// may answer with the legacy layout; present-but-unusable means the one
|
||||
// surviving record of the geometry is corrupt, and guessing from there returns
|
||||
// wrong bytes rather than no bytes.
|
||||
func EcShardConfigFromSidecar(baseFileName string) (cfg *volume_server_pb.EcShardConfig, found bool, err error) {
|
||||
path := BitrotSidecarPath(baseFileName, 0)
|
||||
if _, statErr := os.Stat(path); statErr != nil {
|
||||
if os.IsNotExist(statErr) {
|
||||
return nil, false, nil
|
||||
}
|
||||
return nil, false, fmt.Errorf("stat %s: %w", path, statErr)
|
||||
}
|
||||
cfg, err = EcShardConfigFromSidecarPath(path)
|
||||
return cfg, true, err
|
||||
}
|
||||
|
||||
// EcShardConfigFromSidecarPath is EcShardConfigFromSidecar for a sidecar whose
|
||||
// path the caller already resolved — the rebuild finds it across a multi-disk
|
||||
// server's directories, not only next to the base name.
|
||||
func EcShardConfigFromSidecarPath(path string) (*volume_server_pb.EcShardConfig, error) {
|
||||
prot, loadErr := LoadBitrotSidecar(path)
|
||||
if loadErr != nil {
|
||||
return nil, fmt.Errorf("read %s: %w", path, loadErr)
|
||||
}
|
||||
// The un-suffixed sidecar describes generation 0 and nothing else; one
|
||||
// stamped for another generation is not a record of these shards.
|
||||
if prot.GetGeneration() != 0 {
|
||||
return nil, fmt.Errorf("%s records generation %d, not generation 0", path, prot.GetGeneration())
|
||||
}
|
||||
cfg := prot.GetEcShardConfig()
|
||||
if cfg == nil {
|
||||
return nil, fmt.Errorf("%s records no EC config", path)
|
||||
}
|
||||
if !ValidEcShardCounts(cfg.GetDataShards(), cfg.GetParityShards()) {
|
||||
return nil, fmt.Errorf("%s records invalid shard counts %d+%d",
|
||||
path, cfg.GetDataShards(), cfg.GetParityShards())
|
||||
}
|
||||
if bsErr := ValidateBlockSize(cfg.GetBlockSize()); bsErr != nil {
|
||||
return nil, fmt.Errorf("%s: %w", path, bsErr)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// RemoveBitrotSidecars removes the legacy <base>.ecsum and any versioned
|
||||
@@ -534,6 +624,15 @@ func RemoveBitrotSidecars(base string) {
|
||||
}
|
||||
}
|
||||
|
||||
// FindBitrotSidecar is findBitrotSidecar for callers outside this package. It
|
||||
// answers "does this volume already have a manifest, and where" — a question
|
||||
// that cannot be settled from one base name, because a split -dir/-dir.idx
|
||||
// layout keeps the sidecar with the index and a multi-disk server may keep it
|
||||
// on a sibling. Returns "" when no candidate exists.
|
||||
func FindBitrotSidecar(generation uint32, dataBase, indexBase string, additionalDirs ...string) string {
|
||||
return findBitrotSidecar(generation, dataBase, indexBase, additionalDirs...)
|
||||
}
|
||||
|
||||
// findBitrotSidecar resolves the sidecar path for a generation, searching the
|
||||
// data base, the index base, and any additional directories — mirroring how
|
||||
// shard/.vif lookups handle split data/idx layouts and per-disk mirrors.
|
||||
|
||||
@@ -13,6 +13,16 @@ type ECContext struct {
|
||||
ParityShards int
|
||||
Collection string
|
||||
VolumeId needle.VolumeId
|
||||
// BlockSize > 0 selects the uniform block layout: every shard is a single
|
||||
// contiguous block of this many bytes, so consecutive .dat ranges stay on
|
||||
// one shard instead of striping across all of them at 1MiB granularity.
|
||||
// 0 is the legacy two-tier 1GiB/1MiB layout.
|
||||
BlockSize int64
|
||||
// DatFileSize is the length of the .dat the encode actually read, set by
|
||||
// WriteEcFiles from the same measurement it derives BlockSize from. The
|
||||
// .vif must record the two together: they describe one file, and a live
|
||||
// volume can grow between two separate stats.
|
||||
DatFileSize int64
|
||||
}
|
||||
|
||||
// Total returns the total number of shards (data + parity)
|
||||
@@ -20,6 +30,33 @@ func (ctx *ECContext) Total() int {
|
||||
return ctx.DataShards + ctx.ParityShards
|
||||
}
|
||||
|
||||
// ValidEcShardCounts reports whether a recorded (data, parity) pair could
|
||||
// describe a real EC volume. The sum is taken in uint64 deliberately: on a
|
||||
// 32-bit build `int` is 32 bits, so converting each count first and adding
|
||||
// them wraps for values near the uint32 ceiling — 0x7fffffff + 0x7fffffff
|
||||
// lands at -2, which slips under the MaxShardCount bound.
|
||||
func ValidEcShardCounts(dataShards, parityShards uint32) bool {
|
||||
return dataShards > 0 && parityShards > 0 &&
|
||||
uint64(dataShards)+uint64(parityShards) <= uint64(MaxShardCount)
|
||||
}
|
||||
|
||||
// LargeBlockSize returns the large-block length of this context's shard
|
||||
// layout; nil-safe so callers can pass an unset context for the legacy layout.
|
||||
func (ctx *ECContext) LargeBlockSize() int64 {
|
||||
if ctx != nil && ctx.BlockSize > 0 {
|
||||
return ctx.BlockSize
|
||||
}
|
||||
return ErasureCodingLargeBlockSize
|
||||
}
|
||||
|
||||
// SmallBlockSize returns the small-block length of this context's shard layout.
|
||||
func (ctx *ECContext) SmallBlockSize() int64 {
|
||||
if ctx != nil && ctx.BlockSize > 0 {
|
||||
return ctx.BlockSize
|
||||
}
|
||||
return ErasureCodingSmallBlockSize
|
||||
}
|
||||
|
||||
// NewDefaultECContext creates a context with default 10+4 shard configuration
|
||||
func NewDefaultECContext(collection string, volumeId needle.VolumeId) *ECContext {
|
||||
return &ECContext{
|
||||
|
||||
@@ -233,8 +233,10 @@ func iterateEcjFile(baseFileName string, processNeedleFn func(key types.NeedleId
|
||||
// large-block row boundary, and deriving the layout from the shrunk extent
|
||||
// would read the shards in the wrong block order. Pass zero when the .vif does
|
||||
// not record the encode-time size to infer the layout from the shard size.
|
||||
func WriteDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int64, shardFileNames []string) error {
|
||||
return writeDatFile(baseFileName, datFileSize, encodedDatFileSize, shardFileNames, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize)
|
||||
// largeBlockSize/smallBlockSize are the volume's shard block layout, e.g.
|
||||
// ctx.LargeBlockSize()/ctx.SmallBlockSize() from its .vif EC config.
|
||||
func WriteDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int64, shardFileNames []string, largeBlockSize int64, smallBlockSize int64) error {
|
||||
return writeDatFile(baseFileName, datFileSize, encodedDatFileSize, shardFileNames, largeBlockSize, smallBlockSize)
|
||||
}
|
||||
|
||||
func writeDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int64, shardFileNames []string, largeBlockSize int64, smallBlockSize int64) error {
|
||||
@@ -288,7 +290,11 @@ func writeDatFile(baseFileName string, datFileSize int64, encodedDatFileSize int
|
||||
// A shard size that is an exact multiple of the large block size is
|
||||
// ambiguous: N large rows, or N-1 large rows plus a full small-block
|
||||
// region. The two layouts only agree below the last large row.
|
||||
if shardSize%largeBlockSize == 0 && datFileSize > (shardSize/largeBlockSize-1)*largeBlockSize*int64(dataShards) {
|
||||
// A uniform layout has no such ambiguity — its large and small blocks
|
||||
// are the same size, so every reading of the shard is the same one, and
|
||||
// without this precondition the check fires on every volume.
|
||||
if largeBlockSize != smallBlockSize &&
|
||||
shardSize%largeBlockSize == 0 && datFileSize > (shardSize/largeBlockSize-1)*largeBlockSize*int64(dataShards) {
|
||||
return fmt.Errorf("shard size %d of %s does not identify the block layout; re-encode to record the dat size in .vif", shardSize, baseFileName)
|
||||
}
|
||||
encodedDatFileSize = int64(dataShards) * shardSize
|
||||
|
||||
@@ -217,7 +217,7 @@ func TestDecodeAtomicPublish(t *testing.T) {
|
||||
// final .dat nor a partial .dat.tmp behind.
|
||||
datBase := filepath.Join(dir, "bar_2")
|
||||
missingShards := []string{filepath.Join(dir, "does_not_exist.ec00")}
|
||||
if err := erasure_coding.WriteDatFile(datBase, 100, 100, missingShards); err == nil {
|
||||
if err := erasure_coding.WriteDatFile(datBase, 100, 100, missingShards, erasure_coding.ErasureCodingLargeBlockSize, erasure_coding.ErasureCodingSmallBlockSize); err == nil {
|
||||
t.Fatalf("expected WriteDatFile to fail on missing shard")
|
||||
}
|
||||
if _, err := os.Stat(datBase + ".dat"); !os.IsNotExist(err) {
|
||||
|
||||
@@ -62,12 +62,126 @@ func WriteSortedFileFromIdx(baseFileName string, ext string) (e error) {
|
||||
// BackgroundECContext for the default ratio, or an explicit ctx for a configured
|
||||
// (e.g. custom-ratio) layout. It returns the bitrot protection (per-shard block
|
||||
// CRC32C) computed during the single encode pass; the caller persists it as a
|
||||
// <base>.ecsum sidecar.
|
||||
// <base>.ecsum sidecar, and persists ctx.BlockSize and ctx.DatFileSize (both
|
||||
// set here, from one measurement of the .dat) to the .vif so readers resolve
|
||||
// the shard block layout.
|
||||
func WriteEcFiles(baseFileName string, ctx *ECContext) (*volume_server_pb.EcBitrotProtection, error) {
|
||||
if ctx == nil || ctx.Total() == 0 {
|
||||
if ctx == nil {
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
} else if ctx.Total() == 0 {
|
||||
// Fill the placeholder in place rather than swapping the pointer: the
|
||||
// caller reads BlockSize and DatFileSize back off the context it
|
||||
// passed, and a replacement leaves it holding the zero values.
|
||||
ctx.DataShards, ctx.ParityShards = DataShardsCount, ParityShardsCount
|
||||
}
|
||||
return generateEcFiles(baseFileName, 256*1024, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize, ctx)
|
||||
// Always encode with the uniform block layout, sized for this .dat. Both
|
||||
// the block size and the .dat length it was derived from are left on ctx,
|
||||
// so the caller persists a .vif whose two fields describe one measurement
|
||||
// — a second stat could see a different size on a volume still taking
|
||||
// writes.
|
||||
fi, err := os.Stat(baseFileName + ".dat")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to stat dat file: %w", err)
|
||||
}
|
||||
ctx.DatFileSize = fi.Size()
|
||||
ctx.BlockSize = UniformBlockSize(fi.Size(), ctx.DataShards)
|
||||
return generateEcFiles(baseFileName, 256*1024, ctx.BlockSize, ctx.BlockSize, ctx)
|
||||
}
|
||||
|
||||
// ValidateBlockSize reports whether a `.vif`-recorded shard block size is one
|
||||
// an encoder could have produced. 0 means the legacy two-tier layout, which is
|
||||
// always valid; anything positive must be a whole number of small blocks,
|
||||
// because that is what UniformBlockSize rounds to. A negative or unaligned
|
||||
// value is corruption, and using it would map every read to the wrong shard
|
||||
// offset.
|
||||
func ValidateBlockSize(blockSize int64) error {
|
||||
if blockSize == 0 {
|
||||
return nil
|
||||
}
|
||||
if blockSize < 0 || blockSize%ErasureCodingSmallBlockSize != 0 {
|
||||
return fmt.Errorf("invalid shard block size %d: expected 0 (legacy) or a multiple of %d",
|
||||
blockSize, ErasureCodingSmallBlockSize)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// UniformBlockSize returns the per-shard block size of the uniform layout for
|
||||
// a .dat of the given size: ceil(datFileSize/dataShards) rounded up to a whole
|
||||
// small block. For every input this equals the legacy layout's padded shard
|
||||
// size, so only the byte placement differs between the two layouts, never the
|
||||
// shard length.
|
||||
func UniformBlockSize(datFileSize int64, dataShards int) int64 {
|
||||
perShard := (datFileSize + int64(dataShards) - 1) / int64(dataShards)
|
||||
blocks := (perShard + ErasureCodingSmallBlockSize - 1) / ErasureCodingSmallBlockSize
|
||||
if blocks < 1 {
|
||||
blocks = 1
|
||||
}
|
||||
return blocks * ErasureCodingSmallBlockSize
|
||||
}
|
||||
|
||||
// ResolveRebuildECContext answers which shard layout a rebuild of baseFileName
|
||||
// will use: the caller's context when it already states one, else the volume's
|
||||
// own metadata — its `.vif` in any of the directories the rebuild can read,
|
||||
// then the generation-0 bitrot sidecar, and only then the build defaults.
|
||||
// Exported so a caller that must agree with the rebuild (the post-rebuild
|
||||
// bitrot backfill writes a manifest describing these very shards) resolves
|
||||
// once and uses the same answer, instead of re-deriving it from a narrower
|
||||
// search and recording a layout the rebuild did not use.
|
||||
func ResolveRebuildECContext(baseFileName string, ctx *ECContext, additionalDirs []string) (*ECContext, error) {
|
||||
if ctx == nil || ctx.Total() == 0 {
|
||||
// Resolve the layout from the .vif to preserve the original configuration.
|
||||
vifPath := findVifPath(baseFileName, additionalDirs)
|
||||
volumeInfo, _, foundVif, vifErr := volume_info.MaybeLoadVolumeInfo(vifPath)
|
||||
if vifErr != nil {
|
||||
// The .vif exists but cannot be read or parsed. Fail closed rather
|
||||
// than silently falling back to the default ratio, which would
|
||||
// rebuild a custom-ratio volume with the wrong layout. Pass an
|
||||
// explicit ctx to override.
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: cannot load .vif: %w", baseFileName, vifErr)
|
||||
}
|
||||
switch {
|
||||
case foundVif && volumeInfo.EcShardConfig != nil &&
|
||||
ValidEcShardCounts(volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards):
|
||||
if bsErr := ValidateBlockSize(volumeInfo.EcShardConfig.GetBlockSize()); bsErr != nil {
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: %s: %w", baseFileName, vifPath, bsErr)
|
||||
}
|
||||
ctx = &ECContext{
|
||||
DataShards: int(volumeInfo.EcShardConfig.DataShards),
|
||||
ParityShards: int(volumeInfo.EcShardConfig.ParityShards),
|
||||
BlockSize: volumeInfo.EcShardConfig.GetBlockSize(),
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with config from .vif: %s", baseFileName, ctx.String())
|
||||
case foundVif && volumeInfo.EcShardConfig != nil:
|
||||
// A recorded-but-impossible ratio is corruption, not a reason to
|
||||
// substitute the default one: a 12+4 volume rebuilt as 10+4
|
||||
// reconstructs from the wrong matrix and never regenerates shards
|
||||
// 14-15. Pass an explicit ctx to override.
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: %s records invalid shard counts %d+%d",
|
||||
baseFileName, vifPath, volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards)
|
||||
default:
|
||||
// No usable .vif: the bitrot sidecar records the same config at
|
||||
// encode time and is then the surviving authority. Reading the
|
||||
// default ratio and the legacy block size instead would rebuild
|
||||
// from the wrong geometry AND make the sidecar look like it
|
||||
// disagrees, which silently skips every checksum check below.
|
||||
if sidecarPath := findBitrotSidecar(0, baseFileName, baseFileName, additionalDirs...); sidecarPath != "" {
|
||||
cfg, cfgErr := EcShardConfigFromSidecarPath(sidecarPath)
|
||||
if cfgErr != nil {
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: no usable .vif and %w", baseFileName, cfgErr)
|
||||
}
|
||||
ctx = &ECContext{
|
||||
DataShards: int(cfg.GetDataShards()),
|
||||
ParityShards: int(cfg.GetParityShards()),
|
||||
BlockSize: cfg.GetBlockSize(),
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with config from the bitrot sidecar: %s", baseFileName, ctx.String())
|
||||
break
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with default config", baseFileName)
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
}
|
||||
}
|
||||
return ctx, nil
|
||||
}
|
||||
|
||||
// RebuildEcFiles rebuilds missing EC shard files. Pass BackgroundECContext to
|
||||
@@ -79,38 +193,11 @@ func WriteEcFiles(baseFileName string, ctx *ECContext) (*volume_server_pb.EcBitr
|
||||
// present input shards are verified against it and corrupt ones are excluded
|
||||
// from Reed-Solomon and regenerated; unsafeIgnoreSidecar bypasses that guard.
|
||||
func RebuildEcFiles(baseFileName string, ctx *ECContext, unsafeIgnoreSidecar bool, additionalDirs ...string) ([]uint32, error) {
|
||||
if ctx == nil || ctx.Total() == 0 {
|
||||
// Resolve the layout from the .vif to preserve the original configuration.
|
||||
volumeInfo, _, foundVif, vifErr := volume_info.MaybeLoadVolumeInfo(baseFileName + ".vif")
|
||||
if vifErr != nil {
|
||||
// The .vif exists but cannot be read or parsed. Fail closed rather
|
||||
// than silently falling back to the default ratio, which would
|
||||
// rebuild a custom-ratio volume with the wrong layout. Pass an
|
||||
// explicit ctx to override.
|
||||
return nil, fmt.Errorf("RebuildEcFiles %s: cannot load .vif: %w", baseFileName, vifErr)
|
||||
}
|
||||
if foundVif && volumeInfo.EcShardConfig != nil {
|
||||
ds := int(volumeInfo.EcShardConfig.DataShards)
|
||||
ps := int(volumeInfo.EcShardConfig.ParityShards)
|
||||
|
||||
// Validate EC config before using it
|
||||
if ds > 0 && ps > 0 && ds+ps <= MaxShardCount {
|
||||
ctx = &ECContext{
|
||||
DataShards: ds,
|
||||
ParityShards: ps,
|
||||
}
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with config from .vif: %s", baseFileName, ctx.String())
|
||||
} else {
|
||||
glog.Warningf("Invalid EC config in .vif for %s (data=%d, parity=%d), using default", baseFileName, ds, ps)
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
}
|
||||
} else {
|
||||
glog.V(0).Infof("Rebuilding EC files for %s with default config", baseFileName)
|
||||
ctx = NewDefaultECContext("", 0)
|
||||
}
|
||||
ctx, err := ResolveRebuildECContext(baseFileName, ctx, additionalDirs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return generateMissingEcFiles(baseFileName, 256*1024, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize, ctx, unsafeIgnoreSidecar, additionalDirs)
|
||||
return generateMissingEcFiles(baseFileName, 256*1024, ctx, unsafeIgnoreSidecar, additionalDirs)
|
||||
}
|
||||
|
||||
func ToExt(ecIndex int) string {
|
||||
@@ -159,7 +246,10 @@ func findShardFile(baseFileName string, ext string, additionalDirs []string) str
|
||||
return ""
|
||||
}
|
||||
|
||||
func generateMissingEcFiles(baseFileName string, bufferSize int, largeBlockSize int64, smallBlockSize int64, ctx *ECContext, unsafeIgnoreSidecar bool, additionalDirs []string) (generatedShardIds []uint32, err error) {
|
||||
// generateMissingEcFiles takes no block sizes: Reed-Solomon reconstruction is
|
||||
// layout-agnostic — it rebuilds a missing shard from the same offsets of the
|
||||
// survivors — so the shard layout only ever reaches it through ctx.
|
||||
func generateMissingEcFiles(baseFileName string, bufferSize int, ctx *ECContext, unsafeIgnoreSidecar bool, additionalDirs []string) (generatedShardIds []uint32, err error) {
|
||||
|
||||
// Pass 1: discover which shards exist and which are missing,
|
||||
// opening input files but NOT creating output files yet.
|
||||
@@ -363,6 +453,30 @@ func cleanupRebuildOutputs(outputFiles []*os.File, writePaths []string) {
|
||||
}
|
||||
}
|
||||
|
||||
// findVifPath locates the volume's `.vif` for a rebuild: next to the shards
|
||||
// first, then in every directory the caller also handed us — the index
|
||||
// directory of a split `-dir`/`-dir.idx` layout, and the sibling disks of a
|
||||
// multi-disk server, where `.ecx`/`.ecj`/`.vif` may live while this disk holds
|
||||
// only shards. Returns the data-base path when nothing exists, so the caller's
|
||||
// "not found" handling stays on the canonical name.
|
||||
func findVifPath(baseFileName string, additionalDirs []string) string {
|
||||
dataPath := baseFileName + ".vif"
|
||||
candidates := []string{dataPath}
|
||||
base := filepath.Base(baseFileName)
|
||||
for _, dir := range additionalDirs {
|
||||
if dir == "" {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, filepath.Join(dir, base)+".vif")
|
||||
}
|
||||
for _, c := range candidates {
|
||||
if _, err := os.Stat(c); err == nil {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return dataPath
|
||||
}
|
||||
|
||||
// loadRebuildSidecar loads and validates the generation-0 checksum sidecar for a
|
||||
// rebuild. RebuildEcFiles operates on the un-suffixed (generation 0) shard
|
||||
// names, so only the legacy sidecar is relevant here. Returns BitrotOff when
|
||||
@@ -381,10 +495,20 @@ func loadRebuildSidecar(baseFileName string, ctx *ECContext, additionalDirs []st
|
||||
if prot.Generation != 0 {
|
||||
return nil, BitrotOff
|
||||
}
|
||||
if prot.EcShardConfig == nil ||
|
||||
int(prot.EcShardConfig.DataShards) != ctx.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ctx.ParityShards {
|
||||
return nil, BitrotOff
|
||||
if prot.EcShardConfig == nil {
|
||||
return nil, BitrotOff // records no geometry -> nothing to contradict
|
||||
}
|
||||
if int(prot.EcShardConfig.DataShards) != ctx.DataShards ||
|
||||
int(prot.EcShardConfig.ParityShards) != ctx.ParityShards ||
|
||||
prot.EcShardConfig.BlockSize != ctx.BlockSize {
|
||||
// Both records describe the same encode, so a disagreement means the
|
||||
// rebuild is about to reconstruct under a geometry the checksums do not
|
||||
// cover. Treating that as "no protection" skipped every input and
|
||||
// output check exactly when they matter most.
|
||||
glog.Warningf("bitrot: sidecar %s records layout %d+%d block %d but the rebuild uses %d+%d block %d",
|
||||
path, prot.EcShardConfig.DataShards, prot.EcShardConfig.ParityShards, prot.EcShardConfig.BlockSize,
|
||||
ctx.DataShards, ctx.ParityShards, ctx.BlockSize)
|
||||
return nil, BitrotInvalid
|
||||
}
|
||||
if err := ValidateBitrotManifest(prot, ctx.DataShards, ctx.ParityShards); err != nil {
|
||||
glog.Warningf("bitrot: sidecar %s manifest invalid: %v", path, err)
|
||||
|
||||
@@ -5,7 +5,10 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// readEcxSizeField returns the size field of the .ecx entry for needleId, or
|
||||
@@ -295,3 +298,101 @@ func TestRebuildEcFiles_CustomRatioRebuildsByteIdentical(t *testing.T) {
|
||||
t.Errorf("unexpected shard .ec12 for a 9+3 volume")
|
||||
}
|
||||
}
|
||||
|
||||
// With no .vif, the bitrot sidecar is the surviving record of the volume's
|
||||
// geometry. Defaulting to 10+4 and the legacy block layout instead would
|
||||
// reconstruct a custom-ratio volume from the wrong matrix — and would make the
|
||||
// sidecar look like it disagrees, silently skipping every checksum check.
|
||||
func TestRebuildEcFiles_ResolvesGeometryFromSidecarWithoutVif(t *testing.T) {
|
||||
if !BitrotProtectionEnabled {
|
||||
t.Skip("bitrot protection is compiled out")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "vol")
|
||||
ctx := &ECContext{DataShards: 12, ParityShards: 4}
|
||||
writeRandomDat(t, base, 7000)
|
||||
|
||||
prot, err := WriteEcFiles(base, ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("WriteEcFiles: %v", err)
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("save sidecar: %v", err)
|
||||
}
|
||||
// No .vif at all: the sidecar has to answer.
|
||||
const dropped = 11
|
||||
want, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read shard: %v", err)
|
||||
}
|
||||
if err := os.Remove(base + ToExt(dropped)); err != nil {
|
||||
t.Fatalf("remove shard: %v", err)
|
||||
}
|
||||
|
||||
// nil ctx: the production RPC's BackgroundECContext path.
|
||||
if _, err := RebuildEcFiles(base, nil, false); err != nil {
|
||||
t.Fatalf("RebuildEcFiles: %v", err)
|
||||
}
|
||||
got, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read rebuilt shard: %v", err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("rebuilt shard %d differs; the rebuild used the wrong geometry", dropped)
|
||||
}
|
||||
// The 12+4 shards past the default ratio must still be there.
|
||||
for _, id := range []int{14, 15} {
|
||||
if _, err := os.Stat(base + ToExt(id)); err != nil {
|
||||
t.Errorf("shard %d missing after rebuild: %v", id, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A split -dir/-dir.idx layout keeps the .vif with the index, and a multi-disk
|
||||
// server may leave this disk holding only shards. Probing just the data base
|
||||
// resolves such a volume to the default ratio and the legacy block layout.
|
||||
func TestRebuildEcFiles_FindsVifInAnAdditionalDir(t *testing.T) {
|
||||
dataDir := t.TempDir()
|
||||
idxDir := t.TempDir()
|
||||
base := filepath.Join(dataDir, "vol")
|
||||
ctx := &ECContext{DataShards: 12, ParityShards: 4}
|
||||
writeRandomDat(t, base, 7000)
|
||||
|
||||
if _, err := WriteEcFiles(base, ctx); err != nil {
|
||||
t.Fatalf("WriteEcFiles: %v", err)
|
||||
}
|
||||
// The volume's .vif lives with the index, not the shards.
|
||||
if err := volume_info.SaveVolumeInfo(filepath.Join(idxDir, "vol")+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.GetCurrentVersion()),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 12, ParityShards: 4, BlockSize: ctx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
}
|
||||
|
||||
const dropped = 11
|
||||
want, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read shard: %v", err)
|
||||
}
|
||||
if err := os.Remove(base + ToExt(dropped)); err != nil {
|
||||
t.Fatalf("remove shard: %v", err)
|
||||
}
|
||||
|
||||
if _, err := RebuildEcFiles(base, nil, true, idxDir); err != nil {
|
||||
t.Fatalf("RebuildEcFiles: %v", err)
|
||||
}
|
||||
got, err := os.ReadFile(base + ToExt(dropped))
|
||||
if err != nil {
|
||||
t.Fatalf("read rebuilt shard: %v", err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("rebuilt shard %d differs; the rebuild used the wrong geometry", dropped)
|
||||
}
|
||||
for _, id := range []int{14, 15} {
|
||||
if _, err := os.Stat(base + ToExt(id)); err != nil {
|
||||
t.Errorf("shard %d missing after rebuild: %v", id, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -326,7 +326,7 @@ func testDecodeDat(t *testing.T, datSize int64) {
|
||||
shardFileNames[i] = fmt.Sprintf("%s%s", baseFileName, ctx.ToExt(i))
|
||||
}
|
||||
|
||||
err = WriteDatFile(decodedBase, datSize, datSize, shardFileNames)
|
||||
err = WriteDatFile(decodedBase, datSize, datSize, shardFileNames, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize)
|
||||
require.NoError(t, err, "WriteDatFile")
|
||||
|
||||
// The atomic publish must rename the temp file away, never leaving it behind.
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
package erasure_coding
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// legacyShardSize is the padded shard length the two-tier layout produces:
|
||||
// whole 1GiB rows, then the remainder in 1MiB rows.
|
||||
func legacyShardSize(datFileSize int64, dataShards int) int64 {
|
||||
largeRow := int64(ErasureCodingLargeBlockSize) * int64(dataShards)
|
||||
nLarge := datFileSize / largeRow
|
||||
size := nLarge * ErasureCodingLargeBlockSize
|
||||
if rem := datFileSize - nLarge*largeRow; rem > 0 {
|
||||
smallRow := int64(ErasureCodingSmallBlockSize) * int64(dataShards)
|
||||
size += (rem + smallRow - 1) / smallRow * ErasureCodingSmallBlockSize
|
||||
}
|
||||
return size
|
||||
}
|
||||
|
||||
// The uniform layout must never change a shard's length, only the byte
|
||||
// placement — capacity math and shard-size credibility checks depend on it.
|
||||
func TestUniformBlockSizeMatchesLegacyShardSize(t *testing.T) {
|
||||
sizes := []int64{
|
||||
8, 1000, ErasureCodingSmallBlockSize,
|
||||
10 * ErasureCodingSmallBlockSize,
|
||||
10*ErasureCodingSmallBlockSize + 1,
|
||||
25 * 1024 * 1024,
|
||||
10 * ErasureCodingLargeBlockSize,
|
||||
10*ErasureCodingLargeBlockSize + 1,
|
||||
10*ErasureCodingLargeBlockSize - ErasureCodingSmallBlockSize,
|
||||
30 * 1024 * 1024 * 1024,
|
||||
30*1024*1024*1024 + 12345,
|
||||
}
|
||||
for _, datSize := range sizes {
|
||||
for _, ds := range []int{10, 9, 5, 1} {
|
||||
if got, want := UniformBlockSize(datSize, ds), legacyShardSize(datSize, ds); got != want {
|
||||
t.Errorf("UniformBlockSize(%d, %d) = %d, legacy shard size %d", datSize, ds, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
if got := UniformBlockSize(0, 10); got != ErasureCodingSmallBlockSize {
|
||||
t.Errorf("UniformBlockSize(0, 10) = %d, want one small block", got)
|
||||
}
|
||||
}
|
||||
|
||||
// encodeLayoutFixture writes a random .dat and encodes it with the given block
|
||||
// sizes, returning the .dat content.
|
||||
func encodeLayoutFixture(t *testing.T, baseFileName string, datSize int64, large, small int64, ctx *ECContext) []byte {
|
||||
t.Helper()
|
||||
data := make([]byte, datSize)
|
||||
rand.New(rand.NewSource(datSize)).Read(data)
|
||||
if err := os.WriteFile(baseFileName+".dat", data, 0o644); err != nil {
|
||||
t.Fatalf("write .dat: %v", err)
|
||||
}
|
||||
if _, err := generateEcFiles(baseFileName, 256*1024, large, small, ctx); err != nil {
|
||||
t.Fatalf("generateEcFiles: %v", err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
// readIntervals assembles a byte range from the shard files the way the read
|
||||
// path does.
|
||||
func readIntervals(t *testing.T, baseFileName string, ctx *ECContext, intervals []Interval) []byte {
|
||||
t.Helper()
|
||||
var assembled []byte
|
||||
for _, iv := range intervals {
|
||||
shardId, shardOffset := iv.ToShardIdAndOffset(ctx.LargeBlockSize(), ctx.SmallBlockSize())
|
||||
f, err := os.Open(baseFileName + ctx.ToExt(int(shardId)))
|
||||
if err != nil {
|
||||
t.Fatalf("open shard %d: %v", shardId, err)
|
||||
}
|
||||
buf := make([]byte, iv.Size)
|
||||
if _, err := f.ReadAt(buf, shardOffset); err != nil {
|
||||
t.Fatalf("read shard %d at %d: %v", shardId, shardOffset, err)
|
||||
}
|
||||
f.Close()
|
||||
assembled = append(assembled, buf...)
|
||||
}
|
||||
return assembled
|
||||
}
|
||||
|
||||
// Encode 25MB — large enough that the uniform and legacy layouts place bytes
|
||||
// differently — under both layouts and verify LocateData maps every probed
|
||||
// range back to the original bytes.
|
||||
func TestLocateDataMatchesEncoderForBothLayouts(t *testing.T) {
|
||||
const datSize = 25*1024*1024 + 12345
|
||||
dir := t.TempDir()
|
||||
|
||||
uniformCtx := NewDefaultECContext("", 1)
|
||||
uniformCtx.BlockSize = UniformBlockSize(datSize, uniformCtx.DataShards)
|
||||
legacyCtx := NewDefaultECContext("", 2)
|
||||
|
||||
for name, ctx := range map[string]*ECContext{"uniform": uniformCtx, "legacy": legacyCtx} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
base := fmt.Sprintf("%s/%s", dir, name)
|
||||
data := encodeLayoutFixture(t, base, datSize, ctx.LargeBlockSize(), ctx.SmallBlockSize(), ctx)
|
||||
|
||||
shardSize := datSize / int64(ctx.DataShards)
|
||||
for _, probe := range []struct{ offset, size int64 }{
|
||||
{0, 1000},
|
||||
{ctx.SmallBlockSize() - 100, 200}, // straddles a block boundary
|
||||
{5 * 1024 * 1024, 4 * 1024 * 1024},
|
||||
{datSize - 2000, 2000},
|
||||
} {
|
||||
intervals := LocateData(ctx.LargeBlockSize(), ctx.SmallBlockSize(), shardSize, probe.offset, types.Size(probe.size))
|
||||
got := readIntervals(t, base, ctx, intervals)
|
||||
if !bytes.Equal(got, data[probe.offset:probe.offset+probe.size]) {
|
||||
t.Fatalf("%s layout: bytes at [%d,+%d) do not round-trip", name, probe.offset, probe.size)
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the uniform layout: a 2MB range inside one 3MB
|
||||
// block is a single interval instead of three 1MB stripes.
|
||||
intervals := LocateData(ctx.LargeBlockSize(), ctx.SmallBlockSize(), shardSize, 6*1024*1024+512*1024, types.Size(2*1024*1024))
|
||||
if name == "uniform" && len(intervals) != 1 {
|
||||
t.Errorf("uniform layout reads 2MB in %d intervals, want 1", len(intervals))
|
||||
}
|
||||
if name == "legacy" && len(intervals) != 3 {
|
||||
t.Errorf("legacy layout reads 2MB in %d intervals, want 3", len(intervals))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Encode under both layouts and verify WriteDatFile reconstructs the original
|
||||
// .dat byte for byte — and that decoding a uniform volume with the legacy
|
||||
// geometry does NOT, i.e. the recorded block size is load-bearing.
|
||||
func TestUniformDecodeRoundTrip(t *testing.T) {
|
||||
const datSize = 25 * 1024 * 1024
|
||||
dir := t.TempDir()
|
||||
base := dir + "/1"
|
||||
|
||||
ctx := NewDefaultECContext("", 1)
|
||||
ctx.BlockSize = UniformBlockSize(datSize, ctx.DataShards)
|
||||
data := encodeLayoutFixture(t, base, datSize, ctx.BlockSize, ctx.BlockSize, ctx)
|
||||
|
||||
shardFileNames := make([]string, ctx.DataShards)
|
||||
for i := range shardFileNames {
|
||||
shardFileNames[i] = base + ctx.ToExt(i)
|
||||
}
|
||||
|
||||
if err := WriteDatFile(dir+"/decoded", datSize, datSize, shardFileNames, ctx.LargeBlockSize(), ctx.SmallBlockSize()); err != nil {
|
||||
t.Fatalf("WriteDatFile: %v", err)
|
||||
}
|
||||
decoded, err := os.ReadFile(dir + "/decoded.dat")
|
||||
if err != nil {
|
||||
t.Fatalf("read decoded: %v", err)
|
||||
}
|
||||
if !bytes.Equal(decoded, data) {
|
||||
t.Fatal("uniform decode with recorded geometry does not round-trip")
|
||||
}
|
||||
|
||||
if err := WriteDatFile(dir+"/wrong", datSize, datSize, shardFileNames, ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize); err != nil {
|
||||
t.Fatalf("WriteDatFile legacy geometry: %v", err)
|
||||
}
|
||||
wrong, err := os.ReadFile(dir + "/wrong.dat")
|
||||
if err != nil {
|
||||
t.Fatalf("read wrong: %v", err)
|
||||
}
|
||||
if bytes.Equal(wrong, data) {
|
||||
t.Fatal("decoding a uniform volume with legacy geometry should scramble it; the layouts did not diverge")
|
||||
}
|
||||
}
|
||||
|
||||
// Mount fixtures through NewEcVolume and read a large extent back through
|
||||
// LocateEcShardNeedleInterval, proving the .vif block size steers the read
|
||||
// path: a legacy .vif (no block size) keeps the legacy interpretation, a
|
||||
// uniform .vif reads uniform shards.
|
||||
func TestEcVolumeGeometryFromVif(t *testing.T) {
|
||||
const datSize = 25 * 1024 * 1024
|
||||
|
||||
for _, layout := range []string{"legacy", "uniform"} {
|
||||
t.Run(layout, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
vid := needle.VolumeId(7)
|
||||
base := EcShardFileName("", dir, int(vid))
|
||||
|
||||
ctx := NewDefaultECContext("", vid)
|
||||
if layout == "uniform" {
|
||||
ctx.BlockSize = UniformBlockSize(datSize, ctx.DataShards)
|
||||
}
|
||||
data := encodeLayoutFixture(t, base, datSize, ctx.LargeBlockSize(), ctx.SmallBlockSize(), ctx)
|
||||
|
||||
if err := os.WriteFile(base+".ecx", nil, 0o644); err != nil {
|
||||
t.Fatalf("write .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.GetCurrentVersion()),
|
||||
DatFileSize: datSize,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ctx.DataShards),
|
||||
ParityShards: uint32(ctx.ParityShards),
|
||||
BlockSize: ctx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
}
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", vid)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEcVolume: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if ev.ECContext.BlockSize != ctx.BlockSize {
|
||||
t.Fatalf("loaded BlockSize = %d, want %d", ev.ECContext.BlockSize, ctx.BlockSize)
|
||||
}
|
||||
for i := 0; i < ctx.DataShards; i++ {
|
||||
shard, err := NewEcVolumeShard(types.HardDriveType, dir, "", vid, ShardId(i))
|
||||
if err != nil {
|
||||
t.Fatalf("NewEcVolumeShard %d: %v", i, err)
|
||||
}
|
||||
ev.AddEcVolumeShard(shard)
|
||||
}
|
||||
|
||||
// Sweep a large extent through the volume's own interval mapping.
|
||||
// LocateEcShardNeedleInterval expands the size by the needle
|
||||
// overhead, so compare only the probed prefix.
|
||||
const probeSize = datSize - 64*1024
|
||||
intervals := ev.LocateEcShardNeedleInterval(needle.GetCurrentVersion(), 0, types.Size(probeSize))
|
||||
var assembled []byte
|
||||
for _, iv := range intervals {
|
||||
shardId, shardOffset := ev.IntervalToShardIdAndOffset(iv)
|
||||
shard, found := ev.FindEcVolumeShard(shardId)
|
||||
if !found {
|
||||
t.Fatalf("shard %d not mounted", shardId)
|
||||
}
|
||||
buf := make([]byte, iv.Size)
|
||||
if _, err := shard.ReadAt(buf, shardOffset); err != nil {
|
||||
t.Fatalf("read shard %d at %d: %v", shardId, shardOffset, err)
|
||||
}
|
||||
assembled = append(assembled, buf...)
|
||||
}
|
||||
if len(assembled) < probeSize {
|
||||
t.Fatalf("assembled %d bytes, want at least %d", len(assembled), probeSize)
|
||||
}
|
||||
if !bytes.Equal(assembled[:probeSize], data[:probeSize]) {
|
||||
t.Fatalf("%s layout: full-extent read through the %s .vif does not match the .dat", layout, layout)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -173,7 +173,16 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
|
||||
}
|
||||
}
|
||||
ev.Version = needle.Version3
|
||||
if volumeInfo, _, found, _ := volume_info.MaybeLoadVolumeInfo(vifFileName); found {
|
||||
// A present-but-unreadable or malformed .vif FAILS the mount: every new
|
||||
// encode records a positive uniform block size there, and defaulting to
|
||||
// the legacy layout would serve those shards with the wrong offset math.
|
||||
// Absent stays legal — legacy volumes predate the sidecar.
|
||||
volumeInfo, _, found, vifErr := volume_info.MaybeLoadVolumeInfo(vifFileName)
|
||||
if vifErr != nil {
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: load %s: %w", vid, vifFileName, vifErr)
|
||||
}
|
||||
if found {
|
||||
ev.Version = needle.Version(volumeInfo.Version)
|
||||
ev.datFileSize = volumeInfo.DatFileSize
|
||||
ev.ExpireAtSec = volumeInfo.ExpireAtSec
|
||||
@@ -184,21 +193,56 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
|
||||
ps := int(volumeInfo.EcShardConfig.ParityShards)
|
||||
ev.EncodeTsNs = volumeInfo.EcShardConfig.GetEncodeTsNs()
|
||||
|
||||
// Validate shard counts to prevent zero or invalid values
|
||||
if ds <= 0 || ps <= 0 || ds+ps > MaxShardCount {
|
||||
glog.Warningf("Invalid EC config in VolumeInfo for volume %d (data=%d, parity=%d), using defaults", vid, ds, ps)
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
// A config that is PRESENT but records an impossible ratio is not
|
||||
// a volume to fall back on: substituting the default 10+4 with the
|
||||
// legacy layout would read uniform shards with the wrong offset
|
||||
// math and answer with the wrong bytes. Only an ENTIRELY absent
|
||||
// config means "this predates the record", which the else-branch
|
||||
// below serves with the legacy defaults.
|
||||
if !ValidEcShardCounts(volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards) {
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: %s records invalid shard counts %d+%d",
|
||||
vid, vifFileName, volumeInfo.EcShardConfig.DataShards, volumeInfo.EcShardConfig.ParityShards)
|
||||
} else if blockErr := ValidateBlockSize(volumeInfo.EcShardConfig.GetBlockSize()); blockErr != nil {
|
||||
// A recorded block size that no encoder could have produced maps
|
||||
// every read to the wrong shard offset. Refuse the mount rather
|
||||
// than serve those bytes or silently pick a layout.
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: %s: %w", vid, vifFileName, blockErr)
|
||||
} else {
|
||||
ev.ECContext = &ECContext{
|
||||
Collection: collection,
|
||||
VolumeId: vid,
|
||||
DataShards: ds,
|
||||
ParityShards: ps,
|
||||
BlockSize: volumeInfo.EcShardConfig.GetBlockSize(),
|
||||
}
|
||||
glog.V(1).Infof("Loaded EC config from VolumeInfo for volume %d: %s", vid, ev.ECContext.String())
|
||||
}
|
||||
} else {
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
// A vif that carries no ecShardConfig answers nothing about the
|
||||
// layout — it is no more informative than an absent one, so it
|
||||
// must not skip the sidecar. Going straight to the defaults here
|
||||
// read a uniform volume with the legacy offset math.
|
||||
cfg, sidecarFound, sidecarErr := layoutFromSidecar(dataBaseFileName, indexBaseFileName)
|
||||
if sidecarErr != nil {
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: %s records no EC config and the bitrot sidecar cannot establish the layout: %w", vid, vifFileName, sidecarErr)
|
||||
}
|
||||
if sidecarFound {
|
||||
ev.ECContext = &ECContext{
|
||||
Collection: collection,
|
||||
VolumeId: vid,
|
||||
DataShards: int(cfg.GetDataShards()),
|
||||
ParityShards: int(cfg.GetParityShards()),
|
||||
BlockSize: cfg.GetBlockSize(),
|
||||
}
|
||||
ev.EncodeTsNs = cfg.GetEncodeTsNs()
|
||||
glog.V(0).Infof("ec volume %d: .vif records no EC config; took it from the bitrot sidecar: %s",
|
||||
vid, ev.ECContext.String())
|
||||
} else {
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Don't fabricate a stub .vif here: a version-only stub implies the
|
||||
@@ -207,14 +251,48 @@ func NewEcVolume(diskType types.DiskType, dir string, dirIdx string, collection
|
||||
// mistake for an authoritative config. Mount with in-memory defaults and
|
||||
// leave the real .vif to the encoder or a recovery tool (the Rust volume
|
||||
// server already behaves this way).
|
||||
glog.Warningf("vif file not found, using defaults, volumeId:%d, filename:%s", vid, vifFileName)
|
||||
//
|
||||
// The bitrot sidecar records the same EC config at encode time, so when
|
||||
// it is present it answers the layout question the missing .vif cannot:
|
||||
// defaulting a uniform-layout volume to the legacy block sizes maps
|
||||
// every read to the wrong shard offset. `weed fix -ecx` reads the
|
||||
// sidecar for the same reason.
|
||||
ev.ECContext = NewDefaultECContext(collection, vid)
|
||||
cfg, sidecarFound, sidecarErr := layoutFromSidecar(dataBaseFileName, indexBaseFileName)
|
||||
if sidecarErr != nil {
|
||||
// With no .vif the sidecar is the ONLY record of this volume's
|
||||
// layout. Present but unusable is not "assume legacy" — that
|
||||
// answers reads with the wrong shard offsets, which is worse than
|
||||
// not answering at all.
|
||||
ev.Close()
|
||||
return nil, fmt.Errorf("ec volume %d: no .vif and the bitrot sidecar cannot establish the layout: %w", vid, sidecarErr)
|
||||
}
|
||||
if sidecarFound {
|
||||
ev.ECContext = &ECContext{
|
||||
Collection: collection,
|
||||
VolumeId: vid,
|
||||
DataShards: int(cfg.GetDataShards()),
|
||||
ParityShards: int(cfg.GetParityShards()),
|
||||
BlockSize: cfg.GetBlockSize(),
|
||||
}
|
||||
ev.EncodeTsNs = cfg.GetEncodeTsNs()
|
||||
glog.V(0).Infof("ec volume %d: .vif missing; took EC config from the bitrot sidecar: %s",
|
||||
vid, ev.ECContext.String())
|
||||
} else {
|
||||
// Only now are the defaults what the volume actually mounted on;
|
||||
// logging this after the sidecar answered would send an operator
|
||||
// triaging wrong bytes after the legacy layout instead.
|
||||
glog.Warningf("vif file not found, using defaults, volumeId:%d, filename:%s", vid, vifFileName)
|
||||
}
|
||||
}
|
||||
|
||||
ev.ShardLocations = make(map[ShardId][]pb.ServerAddress)
|
||||
|
||||
// Load the active-generation bitrot checksum sidecar (optional).
|
||||
ev.loadActiveBitrotSidecar()
|
||||
if err := ev.loadActiveBitrotSidecar(); err != nil {
|
||||
ev.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
@@ -528,11 +606,17 @@ func (ev *EcVolume) LocateEcShardNeedleInterval(version needle.Version, offset i
|
||||
shardSize = shard.ecdFileSize - 1
|
||||
}
|
||||
// calculate the locations in the ec shards
|
||||
intervals = LocateData(ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize, shardSize, offset, types.Size(needle.GetActualSize(size, version)))
|
||||
intervals = LocateData(ev.ECContext.LargeBlockSize(), ev.ECContext.SmallBlockSize(), shardSize, offset, types.Size(needle.GetActualSize(size, version)))
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// IntervalToShardIdAndOffset resolves an interval against this volume's shard
|
||||
// block layout.
|
||||
func (ev *EcVolume) IntervalToShardIdAndOffset(interval Interval) (ShardId, int64) {
|
||||
return interval.ToShardIdAndOffset(ev.ECContext.LargeBlockSize(), ev.ECContext.SmallBlockSize())
|
||||
}
|
||||
|
||||
func (ev *EcVolume) FindNeedleFromEcx(needleId types.NeedleId) (offset types.Offset, size types.Size, err error) {
|
||||
offset, size, err = SearchNeedleFromSortedIndex(ev.ecxFile, ev.ecxFileSize, needleId, nil)
|
||||
if err != nil {
|
||||
|
||||
@@ -238,7 +238,7 @@ func (ecv *EcVolume) ScrubLocal() (int64, []*volume_server_pb.EcShardInfo, []err
|
||||
localShardIds := []ShardId{}
|
||||
|
||||
for i, iv := range locations {
|
||||
sid, soffset := iv.ToShardIdAndOffset(ErasureCodingLargeBlockSize, ErasureCodingSmallBlockSize)
|
||||
sid, soffset := ecv.IntervalToShardIdAndOffset(iv)
|
||||
ssize := int64(iv.Size.Raw())
|
||||
shard, found := ecv.FindEcVolumeShard(sid)
|
||||
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
package erasure_coding
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
func uniformSidecar(t *testing.T, base string, ds, ps uint32, blockSize int64) {
|
||||
t.Helper()
|
||||
prot := &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: ds, ParityShards: ps, BlockSize: blockSize,
|
||||
},
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("seed .ecsum: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The sidecar records the layout at encode time, so it answers what a .vif
|
||||
// cannot. A .vif that merely omits ecShardConfig is no more informative than an
|
||||
// absent one — going straight to the defaults reads a uniform volume with the
|
||||
// legacy offset math and returns the wrong bytes.
|
||||
func TestNewEcVolumeTakesLayoutFromSidecarWhenVifHasNoConfig(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
// A config-free .vif: version and dat size only, as a legacy encode left it.
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
uniformSidecar(t, base, 12, 4, 3*1024*1024)
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.ECContext; got.DataShards != 12 || got.ParityShards != 4 || got.BlockSize != 3*1024*1024 {
|
||||
t.Errorf("layout = %d+%d block %d, want 12+4 block 3MiB", got.DataShards, got.ParityShards, got.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// A split -dir/-dir.idx layout keeps the sidecar with the INDEX. Probing only
|
||||
// the data base reports "absent", and absent is the one answer that selects the
|
||||
// legacy layout.
|
||||
func TestNewEcVolumeFindsSidecarInTheIndexDirectory(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
writeVif bool
|
||||
}{
|
||||
{name: "no vif at all"},
|
||||
{name: "vif without ec config", writeVif: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
dataDir, idxDir := t.TempDir(), t.TempDir()
|
||||
idxBase := filepath.Join(idxDir, "2")
|
||||
if err := os.WriteFile(idxBase+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if tc.writeVif {
|
||||
if err := volume_info.SaveVolumeInfo(idxBase+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
}
|
||||
uniformSidecar(t, idxBase, 12, 4, 3*1024*1024)
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, dataDir, idxDir, "", needle.VolumeId(2))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.ECContext; got.DataShards != 12 || got.ParityShards != 4 || got.BlockSize != 3*1024*1024 {
|
||||
t.Errorf("layout = %d+%d block %d, want 12+4 block 3MiB", got.DataShards, got.ParityShards, got.BlockSize)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Absence stays legal — a volume encoded before either record exists is a
|
||||
// genuine legacy volume, and only that case may select the legacy layout.
|
||||
func TestNewEcVolumeConfigFreeVifWithNoSidecarUsesDefaults(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "3")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(3))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.ECContext; got.DataShards != DataShardsCount || got.ParityShards != ParityShardsCount || got.BlockSize != 0 {
|
||||
t.Errorf("layout = %d+%d block %d, want the legacy defaults", got.DataShards, got.ParityShards, got.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// Present but unusable must fail the mount rather than default, in the
|
||||
// config-free-vif branch exactly as in the absent-vif one.
|
||||
func TestNewEcVolumeConfigFreeVifWithUnusableSidecarFails(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "4")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
// Stamped for another generation: not a record of these shards.
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 5,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 12, ParityShards: 4},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .ecsum: %v", err)
|
||||
}
|
||||
if _, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(4)); err == nil {
|
||||
t.Fatal("an unusable sole layout record must fail the mount")
|
||||
}
|
||||
}
|
||||
|
||||
// Startup mirroring gives each shard-bearing disk its own .ecx/.ecj/.vif but
|
||||
// deliberately not the .ecsum, and a repair delivers exactly one copy. A
|
||||
// runtime restricted to its own two directories therefore reports no
|
||||
// protection no matter how often it reloads — so the resolution has to reach
|
||||
// the sibling disk that actually received the manifest.
|
||||
func TestReloadBitrotSidecarFindsItOnASiblingDisk(t *testing.T) {
|
||||
mine, sibling := t.TempDir(), t.TempDir()
|
||||
myBase := filepath.Join(mine, "7")
|
||||
if err := os.WriteFile(myBase+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(myBase+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 0,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
|
||||
ev, err := NewEcVolume(types.HardDriveType, mine, mine, "", needle.VolumeId(7))
|
||||
if err != nil {
|
||||
t.Fatalf("mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if got := ev.bitrotStatus; got != BitrotOff {
|
||||
t.Fatalf("precondition: status = %v, want BitrotOff before the delivery", got)
|
||||
}
|
||||
|
||||
// The delivery lands the manifest on the sibling disk only.
|
||||
prot := &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 0,
|
||||
},
|
||||
}
|
||||
for i := 0; i < 14; i++ {
|
||||
prot.Shards = append(prot.Shards, &volume_server_pb.EcShardChecksums{
|
||||
ShardId: uint32(i), CoveredSize: 1, BlockCrc32C: make([]byte, 4),
|
||||
})
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(filepath.Join(sibling, "7"), 0), prot); err != nil {
|
||||
t.Fatalf("seed sibling .ecsum: %v", err)
|
||||
}
|
||||
|
||||
// Reloading against only its own directories still finds nothing.
|
||||
ev.ReloadBitrotSidecar()
|
||||
if got := ev.bitrotStatus; got != BitrotOff {
|
||||
t.Errorf("own-directories reload: status = %v, want BitrotOff", got)
|
||||
}
|
||||
|
||||
ev.ReloadBitrotSidecar(mine, sibling)
|
||||
if got := ev.bitrotStatus; got != BitrotOn {
|
||||
t.Errorf("cross-disk reload: status = %v, want BitrotOn", got)
|
||||
}
|
||||
}
|
||||
|
||||
// "Does this volume already have a manifest" cannot be answered from one base
|
||||
// name. The rebuild's opportunistic backfill asks it before writing a TOFU
|
||||
// baseline, and a false "no" there writes a second sidecar at the data base
|
||||
// that then shadows the real one — blessing whatever the shards currently say.
|
||||
func TestFindBitrotSidecarSearchesEveryCandidate(t *testing.T) {
|
||||
dataDir, idxDir, sibling := t.TempDir(), t.TempDir(), t.TempDir()
|
||||
dataBase := filepath.Join(dataDir, "8")
|
||||
idxBase := filepath.Join(idxDir, "8")
|
||||
siblingBase := filepath.Join(sibling, "8")
|
||||
|
||||
if got := FindBitrotSidecar(0, dataBase, idxBase, sibling); got != "" {
|
||||
t.Errorf("no sidecar anywhere should answer empty, got %q", got)
|
||||
}
|
||||
|
||||
// Each location on its own: any one of them is enough to answer "yes".
|
||||
for _, tc := range []struct{ name, base string }{
|
||||
{"data directory", dataBase},
|
||||
{"index directory", idxBase},
|
||||
{"sibling disk", siblingBase},
|
||||
} {
|
||||
want := BitrotSidecarPath(tc.base, 0)
|
||||
if err := os.WriteFile(want, []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("%s: seed: %v", tc.name, err)
|
||||
}
|
||||
if got := FindBitrotSidecar(0, dataBase, idxBase, sibling); got != want {
|
||||
t.Errorf("%s: got %q, want %q", tc.name, got, want)
|
||||
}
|
||||
if err := os.Remove(want); err != nil {
|
||||
t.Fatalf("%s: cleanup: %v", tc.name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// With copies everywhere the data base wins, which is the order every
|
||||
// resolver uses — so a stray copy there shadows the others.
|
||||
for _, base := range []string{siblingBase, idxBase, dataBase} {
|
||||
if err := os.WriteFile(BitrotSidecarPath(base, 0), []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("seed %s: %v", base, err)
|
||||
}
|
||||
}
|
||||
if got, want := FindBitrotSidecar(0, dataBase, idxBase, sibling), BitrotSidecarPath(dataBase, 0); got != want {
|
||||
t.Errorf("precedence: got %q, want the data base %q", got, want)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
package erasure_coding
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// A present-but-malformed .vif must FAIL the mount: every new encode
|
||||
// records a positive uniform block size there, and silently defaulting
|
||||
// to the legacy layout would serve those shards with the wrong offset
|
||||
// math. Absence stays legal — legacy volumes predate the sidecar.
|
||||
func TestNewEcVolumeFailsOnMalformedVif(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(base+".vif", []byte("not json"), 0644); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
_, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err == nil {
|
||||
t.Fatal("mount over a malformed .vif must fail")
|
||||
}
|
||||
if !strings.Contains(err.Error(), ".vif") {
|
||||
t.Errorf("error should name the vif: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewEcVolumeAbsentVifMountsWithDefaults(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err != nil {
|
||||
t.Fatalf("legacy mount without .vif must succeed: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if ev.ECContext.BlockSize != 0 {
|
||||
t.Errorf("legacy mount must use the legacy layout: BlockSize=%d", ev.ECContext.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// A block size no encoder could have produced (negative, or not a whole
|
||||
// number of small blocks) maps every read to the wrong shard offset, so the
|
||||
// mount refuses it rather than serving those bytes.
|
||||
func TestNewEcVolumeRejectsImplausibleBlockSize(t *testing.T) {
|
||||
for _, blockSize := range []int64{1, -1, 3*1024*1024 + 1} {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: blockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
if _, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1)); err == nil {
|
||||
t.Errorf("block size %d must fail the mount", blockSize)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewEcVolumeAcceptsAlignedBlockSize(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 3 * 1024 * 1024,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
ev, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err != nil {
|
||||
t.Fatalf("aligned block size must mount: %v", err)
|
||||
}
|
||||
defer ev.Close()
|
||||
if ev.ECContext.BlockSize != 3*1024*1024 {
|
||||
t.Errorf("BlockSize = %d, want 3MiB", ev.ECContext.BlockSize)
|
||||
}
|
||||
}
|
||||
|
||||
// The .vif and the .ecsum both record the layout their generation was encoded
|
||||
// with. When a generation-matching sidecar disagrees, one of them is wrong and
|
||||
// reads through the other land at the wrong shard offsets — so the mount must
|
||||
// refuse rather than quietly drop to unprotected reads.
|
||||
func TestNewEcVolumeRejectsSidecarGeometryDisagreement(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("seed .ecx: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(base+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.Version3),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 0, // says legacy
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed .vif: %v", err)
|
||||
}
|
||||
// The sidecar for the same generation says uniform.
|
||||
prot := &volume_server_pb.EcBitrotProtection{
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 3 * 1024 * 1024,
|
||||
},
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("seed .ecsum: %v", err)
|
||||
}
|
||||
|
||||
_, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1))
|
||||
if err == nil {
|
||||
t.Fatal("a generation-matching layout disagreement must fail the mount")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "block") {
|
||||
t.Errorf("the error should name the disagreement: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// With no .vif the bitrot sidecar is the ONLY record of the volume's layout.
|
||||
// Present but unusable — unreadable, stamped for another generation, or
|
||||
// carrying an incomplete ratio — must fail the mount: answering reads from a
|
||||
// guessed layout returns wrong bytes rather than none. Genuine absence stays
|
||||
// legal, and is covered by TestNewEcVolumeAbsentVifMountsWithDefaults.
|
||||
func TestNewEcVolumeRejectsUnusableSoleSidecar(t *testing.T) {
|
||||
seed := map[string]*volume_server_pb.EcBitrotProtection{
|
||||
"another generation": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 5,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 10, ParityShards: 4},
|
||||
},
|
||||
"no ec config": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
},
|
||||
"incomplete ratio": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{DataShards: 10},
|
||||
},
|
||||
"invalid block size": {
|
||||
Algorithm: volume_server_pb.ChecksumAlgorithm_CHECKSUM_CRC32C,
|
||||
BlockSize: uint32(BitrotBlockSize),
|
||||
Generation: 0,
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: 10, ParityShards: 4, BlockSize: 3*1024*1024 + 1,
|
||||
},
|
||||
},
|
||||
}
|
||||
for name, prot := range seed {
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "1")
|
||||
if err := os.WriteFile(base+".ecx", []byte{}, 0644); err != nil {
|
||||
t.Fatalf("%s: seed .ecx: %v", name, err)
|
||||
}
|
||||
if err := SaveBitrotSidecar(BitrotSidecarPath(base, 0), prot); err != nil {
|
||||
t.Fatalf("%s: seed .ecsum: %v", name, err)
|
||||
}
|
||||
if _, err := NewEcVolume(types.HardDriveType, dir, dir, "", needle.VolumeId(1)); err == nil {
|
||||
t.Errorf("%s: an unusable sole layout record must fail the mount", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,11 @@ import (
|
||||
type ServerShardInventory struct {
|
||||
Bits ShardBits
|
||||
QueryError error
|
||||
// BlockSize is the shard block layout this holder reports for the volume —
|
||||
// the one it will serve reads through. A binary predating the uniform
|
||||
// layout drops the field off the wire and reports 0, which is how
|
||||
// RequireAgreedBlockLayout tells such a holder from one that agrees.
|
||||
BlockSize int64
|
||||
}
|
||||
|
||||
// Query errors are recorded per-server and treated as zero shards rather
|
||||
@@ -50,6 +55,7 @@ func VerifyShardsAcrossServers(ctx context.Context, volumeID uint32,
|
||||
}
|
||||
inv.Bits = inv.Bits.Set(ShardId(s.ShardId))
|
||||
}
|
||||
inv.BlockSize = resp.GetEcShardConfig().GetBlockSize()
|
||||
return nil
|
||||
})
|
||||
if callErr != nil {
|
||||
@@ -99,6 +105,39 @@ func RequireRecoverableShardSet(volumeID uint32, shardsPresent ShardBits, dataSh
|
||||
volumeID, totalShards-len(missing), totalShards, dataShards, missing)
|
||||
}
|
||||
|
||||
// RequireAgreedBlockLayout gates source-volume deletion on every holder that
|
||||
// answered agreeing with the layout the shards were encoded under.
|
||||
//
|
||||
// The uniform block layout lives in a `.vif` field that older volume servers
|
||||
// have never heard of: such a server parses the file, silently discards the
|
||||
// unknown field, and mounts the volume on the legacy 1GiB/1MiB striping. Its
|
||||
// reads then land at the wrong shard offsets and return wrong bytes, and
|
||||
// nothing else in the encode path notices — the shard files are the same
|
||||
// length under either layout. Asking each holder which layout it is serving is
|
||||
// the one question that separates the two, and asking it here means the answer
|
||||
// arrives while the source .dat is still on disk.
|
||||
//
|
||||
// Holders that could not be reached, or that report no shard of this volume,
|
||||
// are skipped: RequireRecoverableShardSet already covers a missing holder, and
|
||||
// one that answered nothing has promised nothing.
|
||||
func RequireAgreedBlockLayout(volumeID uint32, encodedBlockSize int64, perServer map[string]ServerShardInventory) error {
|
||||
disagreeing := make([]string, 0, len(perServer))
|
||||
for server, inv := range perServer {
|
||||
if inv.QueryError != nil || inv.Bits.Count() == 0 {
|
||||
continue
|
||||
}
|
||||
if inv.BlockSize != encodedBlockSize {
|
||||
disagreeing = append(disagreeing, fmt.Sprintf("%s serves block size %d", server, inv.BlockSize))
|
||||
}
|
||||
}
|
||||
if len(disagreeing) == 0 {
|
||||
return nil
|
||||
}
|
||||
sort.Strings(disagreeing)
|
||||
return fmt.Errorf("volume %d was encoded with shard block size %d but %v; upgrade every volume server to a build that understands the uniform shard block layout before encoding",
|
||||
volumeID, encodedBlockSize, disagreeing)
|
||||
}
|
||||
|
||||
func SummarizeShardInventory(perServer map[string]ServerShardInventory) string {
|
||||
servers := make([]string, 0, len(perServer))
|
||||
for s := range perServer {
|
||||
|
||||
@@ -345,6 +345,49 @@ func (s *Store) FindEcVolumeWithShard(vid needle.VolumeId, shardId erasure_codin
|
||||
return nil, nil, false
|
||||
}
|
||||
|
||||
// EcMetadataDirs lists every directory on this server that could hold an EC
|
||||
// volume's metadata — each disk's data and index directory. Startup mirroring
|
||||
// gives each shard-bearing disk its own .ecx/.ecj/.vif, but the checksum
|
||||
// sidecar is not mirrored and a repair delivers exactly one copy, so a runtime
|
||||
// looking only at its own two directories cannot see it. Handing this list to
|
||||
// the sidecar resolution keeps one authoritative copy reachable from every
|
||||
// runtime rather than duplicating a file that is rewritten as shards are
|
||||
// repaired and generations published.
|
||||
func (s *Store) EcMetadataDirs() []string {
|
||||
var dirs []string
|
||||
appendDir := func(dir string) {
|
||||
if dir == "" {
|
||||
return
|
||||
}
|
||||
for _, existing := range dirs {
|
||||
if existing == dir {
|
||||
return
|
||||
}
|
||||
}
|
||||
dirs = append(dirs, dir)
|
||||
}
|
||||
for _, location := range s.Locations {
|
||||
appendDir(location.Directory)
|
||||
appendDir(location.IdxDirectory)
|
||||
}
|
||||
return dirs
|
||||
}
|
||||
|
||||
// FindAllEcVolumes returns every per-disk *EcVolume the store maps for vid. A vid
|
||||
// can mount on N disks as N distinct runtimes, and the first-match FindEcVolume
|
||||
// hides the siblings — so anything that has to reach the whole volume, rather
|
||||
// than any one runtime of it, iterates this instead. Order mirrors the
|
||||
// deterministic s.Locations order; nil when no disk holds the vid.
|
||||
func (s *Store) FindAllEcVolumes(vid needle.VolumeId) []*erasure_coding.EcVolume {
|
||||
var evs []*erasure_coding.EcVolume
|
||||
for _, location := range s.Locations {
|
||||
if ev, found := location.FindEcVolume(vid); found {
|
||||
evs = append(evs, ev)
|
||||
}
|
||||
}
|
||||
return evs
|
||||
}
|
||||
|
||||
func (s *Store) FindEcVolume(vid needle.VolumeId) (*erasure_coding.EcVolume, bool) {
|
||||
for _, location := range s.Locations {
|
||||
if s, found := location.FindEcVolume(vid); found {
|
||||
@@ -436,10 +479,6 @@ func (s *Store) ReadEcShardNeedle(vid needle.VolumeId, n *needle.Needle, onReadS
|
||||
return 0, fmt.Errorf("ec shard %d not found", vid)
|
||||
}
|
||||
|
||||
func (s *Store) IntervalToShardIdAndOffset(iv erasure_coding.Interval) (erasure_coding.ShardId, int64) {
|
||||
return iv.ToShardIdAndOffset(erasure_coding.ErasureCodingLargeBlockSize, erasure_coding.ErasureCodingSmallBlockSize)
|
||||
}
|
||||
|
||||
var (
|
||||
// ecRecoverBudget bounds the interval-sized buffers EC recovery holds across
|
||||
// all concurrent reads: a peer that is slow to fail keeps a whole fan-out of
|
||||
@@ -507,7 +546,7 @@ func (s *Store) readEcShardIntervals(needleId types.NeedleId, ecVolume *erasure_
|
||||
|
||||
// readOneEcShardInterval fills data, which must be interval.Size long.
|
||||
func (s *Store) readOneEcShardInterval(needleId types.NeedleId, ecVolume *erasure_coding.EcVolume, interval erasure_coding.Interval, data []byte) (is_deleted bool, err error) {
|
||||
shardId, actualOffset := s.IntervalToShardIdAndOffset(interval)
|
||||
shardId, actualOffset := ecVolume.IntervalToShardIdAndOffset(interval)
|
||||
|
||||
// try local read
|
||||
err = s.readLocalEcShardInterval(ecVolume, shardId, data, actualOffset)
|
||||
|
||||
@@ -52,7 +52,7 @@ func (s *Store) doDeleteNeedleFromAtLeastOneRemoteEcShards(ecVolume *erasure_cod
|
||||
return erasure_coding.NotFoundError
|
||||
}
|
||||
|
||||
primaryShardId, _ := intervals[0].ToShardIdAndOffset(erasure_coding.ErasureCodingLargeBlockSize, erasure_coding.ErasureCodingSmallBlockSize)
|
||||
primaryShardId, _ := ecVolume.IntervalToShardIdAndOffset(intervals[0])
|
||||
|
||||
// Normal path: delete on exactly one node holding the primary data shard.
|
||||
err = s.doDeleteNeedleFromRemoteEcShardServers(primaryShardId, ecVolume, needleId)
|
||||
|
||||
@@ -2,17 +2,112 @@ package storage
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"math/rand"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/types"
|
||||
"github.com/seaweedfs/seaweedfs/weed/storage/volume_info"
|
||||
)
|
||||
|
||||
// encodeAndMountEcVolume writes needles into a fresh volume, EC-encodes it
|
||||
// (uniform layout), stamps the .vif the way VolumeEcShardsGenerate does, and
|
||||
// mounts every shard on the store.
|
||||
func encodeAndMountEcVolume(t *testing.T, store *Store, vid needle.VolumeId, needles []*needle.Needle) *erasure_coding.EcVolume {
|
||||
t.Helper()
|
||||
dir := store.Locations[0].Directory
|
||||
|
||||
v, err := NewVolume(dir, dir, "", vid, NeedleMapInMemory, &super_block.ReplicaPlacement{}, &needle.TTL{}, 0, needle.GetCurrentVersion(), 0, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("new volume: %v", err)
|
||||
}
|
||||
for _, n := range needles {
|
||||
if _, _, _, err := v.writeNeedle2(n, true, false, false); err != nil {
|
||||
t.Fatalf("write needle: %v", err)
|
||||
}
|
||||
}
|
||||
baseFileName := v.DataFileName()
|
||||
v.Close()
|
||||
|
||||
datSize, err := os.Stat(baseFileName + ".dat")
|
||||
if err != nil {
|
||||
t.Fatalf("stat .dat: %v", err)
|
||||
}
|
||||
ecCtx := erasure_coding.NewDefaultECContext("", vid)
|
||||
if _, err := erasure_coding.WriteEcFiles(baseFileName, ecCtx); err != nil {
|
||||
t.Fatalf("write ec files: %v", err)
|
||||
}
|
||||
if err := erasure_coding.WriteSortedFileFromIdx(baseFileName, ".ecx"); err != nil {
|
||||
t.Fatalf("write .ecx: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(baseFileName+".ecj", nil, 0o644); err != nil {
|
||||
t.Fatalf("write .ecj: %v", err)
|
||||
}
|
||||
if err := volume_info.SaveVolumeInfo(baseFileName+".vif", &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.GetCurrentVersion()),
|
||||
DatFileSize: datSize.Size(),
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: uint32(ecCtx.DataShards),
|
||||
ParityShards: uint32(ecCtx.ParityShards),
|
||||
BlockSize: ecCtx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
}
|
||||
for _, ext := range []string{".dat", ".idx"} {
|
||||
if err := os.Remove(baseFileName + ext); err != nil {
|
||||
t.Fatalf("remove %s: %v", ext, err)
|
||||
}
|
||||
}
|
||||
|
||||
for shardId := 0; shardId < erasure_coding.TotalShardsCount; shardId++ {
|
||||
if err := store.MountEcShards("", vid, erasure_coding.ShardId(shardId), ""); err != nil {
|
||||
t.Fatalf("mount shard %d: %v", shardId, err)
|
||||
}
|
||||
}
|
||||
ecVolume, found := store.Locations[0].FindEcVolume(vid)
|
||||
if !found {
|
||||
t.Fatal("ec volume not mounted")
|
||||
}
|
||||
if ecVolume.ECContext.BlockSize != ecCtx.BlockSize {
|
||||
t.Fatalf("mounted BlockSize = %d, want %d", ecVolume.ECContext.BlockSize, ecCtx.BlockSize)
|
||||
}
|
||||
|
||||
// Every shard is local, so seed the location cache to keep the read off the
|
||||
// master this test does not have.
|
||||
ecVolume.ShardLocationsLock.Lock()
|
||||
for shardId := 0; shardId < erasure_coding.TotalShardsCount; shardId++ {
|
||||
ecVolume.ShardLocations[erasure_coding.ShardId(shardId)] = []pb.ServerAddress{"localhost:8080"}
|
||||
}
|
||||
ecVolume.ShardLocationsRefreshTime = time.Now()
|
||||
ecVolume.ShardLocationsLock.Unlock()
|
||||
return ecVolume
|
||||
}
|
||||
|
||||
func randomNeedleOfSize(id uint64, size int) *needle.Needle {
|
||||
n := new(needle.Needle)
|
||||
n.Id = types.Uint64ToNeedleId(id)
|
||||
n.Data = make([]byte, size)
|
||||
rand.New(rand.NewSource(int64(id))).Read(n.Data)
|
||||
n.Checksum = needle.NewCRC(n.Data)
|
||||
return n
|
||||
}
|
||||
|
||||
// A needle larger than one EC block is split over consecutive blocks, which
|
||||
// live on different shards. The intervals are read concurrently, so check they
|
||||
// still come back in order.
|
||||
func TestReadEcShardNeedleSpanningBlocks(t *testing.T) {
|
||||
store := newTestStore(t, 1)
|
||||
const vid = needle.VolumeId(7)
|
||||
store, ecVolume, n := newLocalEcVolume(t, vid)
|
||||
|
||||
n := randomNeedleOfSize(42, 3*erasure_coding.ErasureCodingSmallBlockSize+1234)
|
||||
ecVolume := encodeAndMountEcVolume(t, store, vid, []*needle.Needle{n})
|
||||
|
||||
_, _, intervals, err := ecVolume.LocateEcShardNeedle(n.Id, ecVolume.Version)
|
||||
if err != nil {
|
||||
@@ -31,3 +126,52 @@ func TestReadEcShardNeedleSpanningBlocks(t *testing.T) {
|
||||
t.Fatalf("read back %d bytes, want the %d written", len(got.Data), len(n.Data))
|
||||
}
|
||||
}
|
||||
|
||||
// On a volume big enough for multi-MB blocks, the uniform layout keeps a
|
||||
// needle smaller than the block in one interval on one shard, and reads every
|
||||
// needle back intact.
|
||||
func TestReadEcShardNeedleUniformLayout(t *testing.T) {
|
||||
store := newTestStore(t, 1)
|
||||
const vid = needle.VolumeId(8)
|
||||
|
||||
// ~26MB of needles: block size becomes 3MB, so the layouts diverge and a
|
||||
// 1MB needle would stripe across shards under the legacy layout.
|
||||
var needles []*needle.Needle
|
||||
for i := uint64(1); i <= 5; i++ {
|
||||
needles = append(needles, randomNeedleOfSize(i, 4*1024*1024))
|
||||
}
|
||||
for i := uint64(6); i <= 11; i++ {
|
||||
needles = append(needles, randomNeedleOfSize(i, 1024*1024))
|
||||
}
|
||||
ecVolume := encodeAndMountEcVolume(t, store, vid, needles)
|
||||
if ecVolume.ECContext.BlockSize <= erasure_coding.ErasureCodingSmallBlockSize {
|
||||
t.Fatalf("block size %d does not diverge from the legacy layout", ecVolume.ECContext.BlockSize)
|
||||
}
|
||||
|
||||
// With 3MB blocks a 1MB needle sits inside one block (bar a boundary
|
||||
// straddle), where the legacy layout would stripe it at 1MB granularity.
|
||||
singleInterval := false
|
||||
for _, n := range needles[5:] {
|
||||
_, _, intervals, err := ecVolume.LocateEcShardNeedle(n.Id, ecVolume.Version)
|
||||
if err != nil {
|
||||
t.Fatalf("locate needle %v: %v", n.Id, err)
|
||||
}
|
||||
if len(intervals) == 1 {
|
||||
singleInterval = true
|
||||
}
|
||||
}
|
||||
if !singleInterval {
|
||||
t.Fatal("no 1MB needle mapped to a single interval; uniform layout not in effect")
|
||||
}
|
||||
|
||||
for _, n := range needles {
|
||||
got := new(needle.Needle)
|
||||
got.Id = n.Id
|
||||
if _, err := store.ReadEcShardNeedle(vid, got, nil); err != nil {
|
||||
t.Fatalf("read needle %v: %v", n.Id, err)
|
||||
}
|
||||
if !bytes.Equal(got.Data, n.Data) {
|
||||
t.Fatalf("needle %v: read back %d bytes that do not match the %d written", n.Id, len(got.Data), len(n.Data))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -95,7 +95,8 @@ func writeEcVolumeFiles(t *testing.T, dir string, vid needle.VolumeId) (baseFile
|
||||
if err != nil {
|
||||
t.Fatalf("stat .dat: %v", err)
|
||||
}
|
||||
if _, err := erasure_coding.WriteEcFiles(baseFileName, erasure_coding.BackgroundECContext()); err != nil {
|
||||
ecCtx := erasure_coding.BackgroundECContext()
|
||||
if _, err := erasure_coding.WriteEcFiles(baseFileName, ecCtx); err != nil {
|
||||
t.Fatalf("write ec files: %v", err)
|
||||
}
|
||||
if err := erasure_coding.WriteSortedFileFromIdx(baseFileName, ".ecx"); err != nil {
|
||||
@@ -110,6 +111,10 @@ func writeEcVolumeFiles(t *testing.T, dir string, vid needle.VolumeId) (baseFile
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: erasure_coding.DataShardsCount,
|
||||
ParityShards: erasure_coding.ParityShardsCount,
|
||||
// The encode writes the uniform layout; a .vif that omitted its
|
||||
// block size would mount these shards as legacy and read every
|
||||
// interval at the wrong offset.
|
||||
BlockSize: ecCtx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
@@ -162,7 +167,7 @@ func TestRecoverOneRemoteEcShardIntervalUsesLocalShards(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("locate needle: %v", err)
|
||||
}
|
||||
shardIdToRecover, actualOffset := store.IntervalToShardIdAndOffset(intervals[0])
|
||||
shardIdToRecover, actualOffset := ecVolume.IntervalToShardIdAndOffset(intervals[0])
|
||||
|
||||
got := make([]byte, intervals[0].Size)
|
||||
nRead, _, err := store.recoverOneRemoteEcShardInterval(n.Id, ecVolume, shardIdToRecover, got, actualOffset)
|
||||
|
||||
@@ -47,14 +47,14 @@ func (s *Store) ScrubEcVolume(vid needle.VolumeId, mode volume_server_pb.VolumeS
|
||||
|
||||
shardIds := make([]erasure_coding.ShardId, len(intervals))
|
||||
for i, iv := range intervals {
|
||||
sid, _ := s.IntervalToShardIdAndOffset(iv)
|
||||
sid, _ := ecv.IntervalToShardIdAndOffset(iv)
|
||||
shardIds[i] = sid
|
||||
}
|
||||
slices.Sort(shardIds)
|
||||
|
||||
for i, iv := range intervals {
|
||||
chunk := make([]byte, iv.Size)
|
||||
shardId, offset := s.IntervalToShardIdAndOffset(iv)
|
||||
shardId, offset := ecv.IntervalToShardIdAndOffset(iv)
|
||||
|
||||
// try a local shard read first...
|
||||
if err := s.readLocalEcShardInterval(ecv, shardId, chunk, offset); err == nil {
|
||||
|
||||
@@ -45,7 +45,8 @@ func newLocalEcVolume(t *testing.T, vid needle.VolumeId) (*Store, *erasure_codin
|
||||
if err != nil {
|
||||
t.Fatalf("stat .dat: %v", err)
|
||||
}
|
||||
if _, err := erasure_coding.WriteEcFiles(baseFileName, erasure_coding.BackgroundECContext()); err != nil {
|
||||
ecCtx := erasure_coding.BackgroundECContext()
|
||||
if _, err := erasure_coding.WriteEcFiles(baseFileName, ecCtx); err != nil {
|
||||
t.Fatalf("write ec files: %v", err)
|
||||
}
|
||||
if err := erasure_coding.WriteSortedFileFromIdx(baseFileName, ".ecx"); err != nil {
|
||||
@@ -60,6 +61,10 @@ func newLocalEcVolume(t *testing.T, vid needle.VolumeId) (*Store, *erasure_codin
|
||||
EcShardConfig: &volume_server_pb.EcShardConfig{
|
||||
DataShards: erasure_coding.DataShardsCount,
|
||||
ParityShards: erasure_coding.ParityShardsCount,
|
||||
// The encode writes the uniform layout; a .vif that omitted its
|
||||
// block size would mount these shards as legacy and read every
|
||||
// interval at the wrong offset.
|
||||
BlockSize: ecCtx.BlockSize,
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("save .vif: %v", err)
|
||||
|
||||
@@ -51,6 +51,11 @@ type ErasureCodingTask struct {
|
||||
// pre-distribute sweep. deleteOriginalVolume skips these so it does not
|
||||
// re-delete and remove the now-EC .vif those servers share.
|
||||
emptyReplicasDeleted map[string]bool
|
||||
|
||||
// encodedBlockSize is the shard block layout WriteEcFiles actually encoded
|
||||
// with, read back off the EC context. Every holder must report serving the
|
||||
// same one before the source volume may be deleted.
|
||||
encodedBlockSize int64
|
||||
}
|
||||
|
||||
// NewErasureCodingTask creates a new unified EC task instance
|
||||
@@ -583,16 +588,23 @@ func (t *ErasureCodingTask) generateEcShardsLocally(localFiles map[string]string
|
||||
}
|
||||
|
||||
// Generate EC shard files (.ec00 ~ .ec13)
|
||||
ecBitrot, err := erasure_coding.WriteEcFiles(baseName, erasure_coding.BackgroundECContext())
|
||||
ecCtx := erasure_coding.BackgroundECContext()
|
||||
ecBitrot, err := erasure_coding.WriteEcFiles(baseName, ecCtx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to generate EC shard files: %w", err)
|
||||
}
|
||||
// The layout the shards were actually written in, for the holder agreement
|
||||
// check before the source is deleted.
|
||||
t.encodedBlockSize = ecCtx.BlockSize
|
||||
// Persist the bitrot checksum sidecar (generation 0) alongside the shards so
|
||||
// it travels with them during distribution. Best-effort: a failed sidecar
|
||||
// write leaves the generation unprotected rather than failing the encode.
|
||||
// it travels with them during distribution. Protection was asked for, and
|
||||
// this write is orders of magnitude smaller than the shards that just
|
||||
// landed: if it fails the disk is in trouble, and continuing would delete
|
||||
// the source replicas in exchange for a generation that is both unprotected
|
||||
// and missing the geometry record the .vif fallback reads.
|
||||
if erasure_coding.BitrotProtectionEnabled && ecBitrot != nil {
|
||||
if serr := erasure_coding.SaveBitrotSidecar(erasure_coding.BitrotSidecarPath(baseName, 0), ecBitrot); serr != nil {
|
||||
glog.Warningf("failed to write EC bitrot sidecar for %s: %v", baseName, serr)
|
||||
return nil, fmt.Errorf("write EC bitrot sidecar for %s: %w", baseName, serr)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -662,6 +674,7 @@ func (t *ErasureCodingTask) generateEcShardsLocally(localFiles map[string]string
|
||||
if ecBitrot != nil && ecBitrot.EcShardConfig != nil {
|
||||
ecShardConfig.DataShards = ecBitrot.EcShardConfig.DataShards
|
||||
ecShardConfig.ParityShards = ecBitrot.EcShardConfig.ParityShards
|
||||
ecShardConfig.BlockSize = ecBitrot.EcShardConfig.BlockSize
|
||||
}
|
||||
volumeInfo := &volume_server_pb.VolumeInfo{
|
||||
Version: uint32(needle.GetCurrentVersion()),
|
||||
@@ -675,30 +688,40 @@ func (t *ErasureCodingTask) generateEcShardsLocally(localFiles map[string]string
|
||||
} else {
|
||||
glog.Warningf("stat %s for .vif dat file size: %v", datFile, err)
|
||||
}
|
||||
// The .vif carries the shard block layout the holders will read through, so
|
||||
// neither the write nor the inclusion below is optional: an encode that
|
||||
// distributed shards without it would leave every reader falling back to
|
||||
// the legacy layout, and the task deletes the source replicas afterwards.
|
||||
if err := volume_info.SaveVolumeInfo(vifFile, volumeInfo); err != nil {
|
||||
glog.Warningf("Failed to create .vif file: %v", err)
|
||||
} else {
|
||||
shardFiles["vif"] = vifFile
|
||||
if info, err := os.Stat(vifFile); err == nil {
|
||||
t.GetLogger().WithFields(map[string]interface{}{
|
||||
"file_type": "vif",
|
||||
"file_path": vifFile,
|
||||
"size_bytes": info.Size(),
|
||||
}).Info("Volume info file generated")
|
||||
}
|
||||
return nil, fmt.Errorf("write %s: %w", vifFile, err)
|
||||
}
|
||||
vifInfo, err := os.Stat(vifFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stat %s for distribution: %w", vifFile, err)
|
||||
}
|
||||
shardFiles["vif"] = vifFile
|
||||
t.GetLogger().WithFields(map[string]interface{}{
|
||||
"file_type": "vif",
|
||||
"file_path": vifFile,
|
||||
"size_bytes": vifInfo.Size(),
|
||||
}).Info("Volume info file generated")
|
||||
|
||||
// Add the generation-0 bitrot checksum sidecar so it is distributed with
|
||||
// the shards (DistributeEcShards only ships files present in shardFiles).
|
||||
// Best-effort like the sidecar write above: if it is absent the holders
|
||||
// are simply unprotected rather than failing the encode.
|
||||
// Strict when protection is enabled and the encoder produced a manifest —
|
||||
// the write above already failed the encode otherwise — so the holders
|
||||
// cannot end up with shards whose checksums stayed behind on the worker.
|
||||
ecsumFile := erasure_coding.BitrotSidecarPath(baseName, 0)
|
||||
if info, err := os.Stat(ecsumFile); err == nil {
|
||||
ecsumInfo, ecsumErr := os.Stat(ecsumFile)
|
||||
if ecsumErr != nil && erasure_coding.BitrotProtectionEnabled && ecBitrot != nil {
|
||||
return nil, fmt.Errorf("stat %s for distribution: %w", ecsumFile, ecsumErr)
|
||||
}
|
||||
if ecsumErr == nil {
|
||||
shardFiles["ecsum"] = ecsumFile
|
||||
t.GetLogger().WithFields(map[string]interface{}{
|
||||
"file_type": "ecsum",
|
||||
"file_path": ecsumFile,
|
||||
"size_bytes": info.Size(),
|
||||
"size_bytes": ecsumInfo.Size(),
|
||||
}).Info("EC bitrot checksum sidecar generated")
|
||||
}
|
||||
|
||||
@@ -768,6 +791,20 @@ func (t *ErasureCodingTask) verifyEcShardsBeforeDelete(ctx context.Context) erro
|
||||
"per_server": summary,
|
||||
}).Warning("EC shard set incomplete but recoverable; proceeding with source deletion")
|
||||
}
|
||||
|
||||
// Before anything irreversible: every holder that answered must report
|
||||
// serving the layout these shards were encoded in. A holder too old to
|
||||
// know the uniform layout mounts them as legacy and returns wrong bytes,
|
||||
// and the source volume is the only remaining correct copy.
|
||||
if err := erasure_coding.RequireAgreedBlockLayout(t.volumeID, t.encodedBlockSize, perServer); err != nil {
|
||||
t.GetLogger().WithFields(map[string]interface{}{
|
||||
"volume_id": t.volumeID,
|
||||
"per_server": summary,
|
||||
"error": err.Error(),
|
||||
}).Error("EC holders disagree on the shard block layout — source volume will be kept")
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user