mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-18 21:26:56 +00:00
2ec899bdee9fd456bdb4729c93c09577ed8d07c8
1127
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
505049a4de |
volume: skip directory fsync on Windows, report a failed makeupDiff (#10572)
* volume: skip directory fsync on Windows * ci: run the windows jobs for the whole vacuum path Both windows jobs start the same weed mini cluster, so both exercise the volume server's vacuum path, but only one of them watched a single file in it. Cover the compact, reconcile and load files in both. * volume: report a failed makeupDiff instead of discarding it The cleanup removes assigned to the same err the makeupDiff failure was held in, so an aborted compaction returned nil once both removes succeeded. The master then recorded the vacuum as committed and the volume reloaded against the discarded generation. * volume: correct the fsyncDir comments after the windows skip Both comments described the old shape, where windows fell through to a sync whose error was swallowed. * volume: keep the makeupDiff failure ahead of its cleanup errors A failed remove of .cpd/.cpx outranked the failure that abandoned the compaction, so the caller saw the cleanup error instead of the cause. Log it and return the original, matching the Rust do_commit_compact. A leftover temp file is rolled back by reconcile on the next start. |
||
|
Chris LuandGitHub
|
312cfe5ae1 |
Fix volume.merge corrupting every needle it copies (#10565)
* Give volume.merge the needle size the target actually indexes by needleBlobFromNeedle returned the size Append reports, which is Size(n.DataSize) - payload bytes only. The .dat header, the needle map and WriteNeedleBlobRequest.Size all use n.Size, which additionally covers the flags, name, mime and lastModified fields. Every needle volume.merge copied therefore landed with a too-small size. The target indexed it at that length, so every later read failed the header check in ReadBytes with a size mismatch, and on v3 the fresh AppendAtNs stamp landed NeedleHeaderSize+DataSize+NeedleChecksumSize into the blob - exactly on the flags byte - overwriting flags, name size, mime size and the first mime bytes with the top of a timestamp. Needles came back with flags 0x18, no name, no mime and a phantom TTL parsed from two arbitrary timestamp bytes; the ones that decoded as expired 404 and vacuum would drop them. Since merge rebuilds every replica from the merged copy, no clean replica survives. Return n.Size, which Append fills in as it serializes, matching what the normal write path stores via nm.Put. * Reject needle blobs whose size disagrees with their own header WriteNeedleBlob trusts the caller's size for two destructive things: it is what goes into the needle map, and it is where the v3 AppendAtNs stamp is written inside the caller's buffer. A caller passing the payload-only DataSize convention corrupts both, and nothing surfaces until the needle is read back - by which point every replica may already have been rebuilt from it. Parse the blob's own header and refuse the write when the two disagree. Mirrored in the Rust volume server. |
||
 Mohit TalniyaandGitHub
|
0815ad78f6 |
fix(volume): persist the leveldb needle map watermark at batch boundaries (#10557)
levelDbWrite persists the replay watermark when its updateWatermark
argument is true. Put and Delete passed "watermark == 0", which is true
on exactly the writes that carry no checkpoint and false on the batch
boundary that carries one. The two cases were inverted:
recordCount % watermarkBatchSize != 0 -> watermark 0, flag true
-> re-persists a zero on 9999 of every 10000 writes
recordCount % watermarkBatchSize == 0 -> watermark N, flag false
-> drops the only value worth saving
The stored watermark therefore never left 0. Recovery stayed correct,
because replaying .idx from offset 0 is a superset of replaying from N
and replay is idempotent, so this never surfaced as a failure. It only
meant generateLevelDbFile walked the entire index on every rebuild, and
every needle write paid a second leveldb Put to rewrite the same zero.
Pass "watermark != 0" so the boundary write checkpoints and the writes
in between leave the key alone.
Verified on a 25000-needle volume: the stored watermark now reads 20000
instead of 0, and a rebuild replays 5000 entries instead of 25000.
The new test drives a full batch of Puts and a full batch of Deletes to
cover both call sites.
|
||
|
Chris LuandGitHub
|
de00091765 |
test: random needles always carry at least one byte (#10523)
A zero-data needle lands in .dat as a size-0 record, byte-identical to a delete marker, so scans that walk .dat count it as deleted. Once in 1024 writes newRandomNeedle produced one, and the idx-head repair then skipped a row TestRepairIdxHeadTombstones_ReadOnlyVolume expected back. |
||
 Lisandro PinandGitHub
|
fa9f471f56 |
EC scrubbing: list shards for needles failing scrubs in the result output. (#10510)
This allows to pinpoint failures to a subset of shards, which can then be bisected and potentially reconstructed. |
||
|
Chris LuandGitHub
|
4dc1b70b2f |
test: pin that a .vif replication outranks the superblock (#10499)
* test: pin that a .vif replication outranks the superblock Store.ConfigureVolume rewrites the .vif and never the replica-placement byte in the .dat, so that byte keeps whatever the volume was created with for good. readSuperBlock reads it and then overrides it from the .vif, which is what makes a replication change take effect and survive a remount. Invert that and every replication change silently reverts on the next mount, while the .vif on disk still records what the operator asked for -- a durability setting quietly going back to its old value, with nothing to indicate it. Worth pinning rather than reading off the code, because the field beside it resolves the other way: version takes the superblock over the .vif. Two fields, one function, opposite precedence, each a line to invert wrongly. Covers the empty case too, since a .vif that declares no replication has to leave the superblock standing or a volume whose replication was never configured would be forced to whatever the zero value parses as. * test: drop the unreachable nil check on MaybeLoadVolumeInfo It initialises the returned pointer before the existence check and every return is naked, so it never yields nil. Guarding against it implied a contract the callee does not have. |
||
|
Chris LuandGitHub
|
13176b4edd |
volume: recover .idx rows overwritten by tiered deletes (#10474)
* volume: recover .idx rows overwritten by tiered deletes A delete on a read-only volume backed by a remote tier used to write its tombstone row at .idx offset 0 rather than appending it, so each delete overwrote one more row at the front and lost the Put rows indexing the first needles in .dat. Those needles 404 even though .dat still holds them, and rebuilding .idx with weed fix means stopping the server and pulling the whole .dat back from the tier. The damage has a fingerprint -- .idx opening with a run of offset-0 tombstones, which a healthy .idx never does -- and .idx and .dat grow in lockstep, so the lost rows indexed exactly the first N .dat records. Detect it at load and re-derive them from a header-only walk over the head of .dat, cheap even against a remote tier, appending only the keys the .idx no longer names. * rust volume: mirror the .idx head tombstone recovery Port the Go detection and repair: an .idx opening with a run of offset-0 tombstones lost the Put rows indexing the first needles in .dat, so re-derive them at load from a header-only walk over the head of .dat and append the keys the .idx no longer names. * volume: put recovered .idx rows back in front instead of appending Appending left the offset-0 tombstone run at the head, so every later load re-walked .idx to the tail to notice the volume was already recovered, and the rows for the head of .dat sat past the .dat-tail row -- costing CheckVolumeDataIntegrity its O(1) path and breaking the ascending append order BinarySearchByAppendAtNs assumes. Rewrite .idx as the recovered rows followed by its current contents, through a temp file and a rename. .idx is back in .dat append order, so a later load stops after reading one row. * volume: keep the .idx mode when the repair replaces it The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go) or whatever the umask allows (Rust) would silently widen an index an operator had locked down. Carry the mode off the file being replaced. |
||
|
Chris LuandGitHub
|
fee3fcb55a |
mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes. |
||
|
Chris LuandGitHub
|
be81b9d5d7 |
volume: fix EC decode/reconstruct index locality under -dir.idx (#10442)
* volume: fix EC decode/reconstruct index locality under -dir.idx EC->replicated decode failed under -dir.idx and on multi-disk with "volume not found on disk". The reconstruct rebuilds the .dat on the data disk but the on-demand VolumeMount scans only the data directory, matching on .idx/.vif; with the rebuilt .idx off in the index directory it matched the volume's leftover EC .vif and skipped the volume as EC metadata. - Resolve the EC .ecx local-first: prefer the copy co-located with the shards over the shared -dir.idx copy, with a non-empty preference so a 0-byte local stub still yields to a valid sibling (the cross-disk fallback). - Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the mount finds it; sweep .ecx/.ecj from both the data and index directories on Destroy so a stale copy cannot re-mount as a phantom EC volume. - Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the .idx/.sdx from the data disk back to the -dir.idx directory (copy fallback across filesystems), and remount. A no-op without -dir.idx. * volume: tests for EC index locality (local-first .ecx, sweep, consolidate) - NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a 0-byte local stub yields to a non-empty shared copy (the #9212 fallback). - Destroy sweeps .ecx/.ecj from both the data and index directories. - ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and keeps the volume mounted; no-op without a separate index dir. - RenameOrCopyFile moves a file and drops the source. * volume: relocate the decoded index in place, without a read gap ConsolidateVolumeIndex previously unmounted the volume, moved the index, and remounted it. Between the EC-shard delete and the remount the volume had neither a normal nor an EC form mounted, so a read landing in that window got a not-found (or was proxied away). Move the index in place instead: RelocateIndexTo takes the data-file write lock, closes the needle map and data backend, moves the .idx (and derived .sdx), then retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The volume never leaves the mounted set, so a concurrent read blocks briefly on the lock rather than failing. The test now writes a needle before consolidating and reads it back after, proving the in-place reload keeps the volume serving. * volume: address review — maintenance guard, no orphan on copy failure - VolumeConsolidateIndex now rejects the request under maintenance mode, like VolumeConfigure and the other mutating volume RPCs. - RenameOrCopyFile rolls the cross-device copy back when the source cannot be removed, so a failed move never leaves two divergent copies (the loader would keep the data-dir one while the idx-dir orphan goes stale). - RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing it, since that leaves the volume unusable until the next load. |
||
|
Chris LuandGitHub
|
2d9227747a |
volume: reject needle blob writes to read-only volumes (#10435)
* volume: reject needle blob writes to read-only volumes WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a read-only volume the needle map is a SortedFileNeedleMap whose Put always fails, so the append is never indexed and never rolled back. Nothing upstream stops this: volume.check.disk picks its targets from the master's cached topology, which goes stale the moment a volume server marks a replica read-only itself — a failed data integrity check at load, or an EIO quarantine. Each sync attempt then grows the .dat of a replica that is supposed to be frozen by one unindexed needle, and reports it as "invalid argument", the bare os.ErrInvalid the needle map returns. Check IsReadOnly before touching .dat, same as the upload path does. * volume: say which needle and volume failed to index An index write that fails surfaced as a bare errno with no volume, no needle and no file — "invalid argument" for a read-only needle map, or a plain ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged at V(4), so by default the operator saw only the errno the client got back. |
||
 
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris LuandGitHub
|
5a54beac80 |
EC decode: read shards with the encode-time block layout (#10385)
* erasure_coding: WriteDatFile takes the encode-time dat size for the shard block layout * volume server: derive EC decode layout from the encode-time dat size, not the live extent * erasure_coding: test decode after tail deletions shrink the live extent below a large-block row * seaweed-volume: write_dat_file_from_shards takes the encode-time dat size for the shard block layout * seaweed-volume: derive EC decode layout from the encode-time dat size, not the live extent * seaweed-volume: test decode after tail deletions shrink the live extent below a large-block row * erasure_coding: reject decoding with no data shards * worker: record the encode-time dat size in the .vif * erasure_coding: fall back to the shard-derived layout only when the encode-time dat size is missing * erasure_coding: reject an ambiguous shard-derived block layout * seaweed-volume: fall back to the shard-derived layout only when the encode-time dat size is missing * seaweed-volume: reject an ambiguous shard-derived block layout |
||
|
Chris LuandGitHub
|
1f8d0a9ccf |
volume server: fill in ModifiedAtSecond on the /status volume list (#10351)
Only the heartbeat path read the .dat mtime; collectStatForOneVolume left the field at 0, so /status consumers could not tell how long a volume had been idle. |
||
  
|
8a71327324 |
fix: report short S3 ReaderAt reads (#10345)
* fix: report short S3 ReaderAt reads Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data. Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read. Fix: Clear io.EOF only when the requested buffer was completely filled. Validation: go test ./weed/storage/backend/... Co-authored-by: Codex <noreply@openai.com> * fix: validate S3 ReaderAt requests Co-authored-by: Codex <noreply@openai.com> * s3 ReadAt: simplify negative offset error * s3 ReadAt: stop reading once the buffer is full Skips the extra Read that only collected the terminal EOF, and bounds the loop if the server ignores the Range header and returns more data than requested, where Read on an empty slice can spin forever. * s3 ReadAt: cover full reads that arrive with io.EOF --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
8bff3b3213 |
fix(volume): reject overflowing needle ID deltas (#10342)
* fix: reject overflowing needle ID deltas Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error. Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range. Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit. Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD Co-authored-by: Codex <noreply@openai.com> * fix: propagate needle ID delta parse errors Co-authored-by: Codex <noreply@openai.com> * print the needle id in hex in the delta overflow error * batch delete: keep processing after a cookie mismatch * rust volume: reject overflowing needle id deltas --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandGitHub
|
10cdaf3818 |
Introduce weed shell command ec.check.replication. (#10328)
* Introduce weed shell command `ec.check.replication`.
This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.
The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.
* shell: ec.check.replication reports mixed under+over-replication in both lists
|
||
|
Chris LuandGitHub
|
6f816c955d |
volume.fsck: fix orphan purge against the rust volume server (#10289)
* rust volume: accept odd-length needle id hex in file ids Go formats the needle id with strconv.FormatUint and parses it back with strconv.ParseUint, neither of which pads to an even number of hex digits. hex::decode rejected such file ids with "Odd number of digits", so volume.fsck could not purge orphans from a rust volume server. Parse the needle id and cookie with from_str_radix, matching Go's ParseNeedleId and ParseCookie. * storage: emit even-length needle id hex in NeedleId.FileId volume.fsck and volume.check_disk build purge file ids here with unpadded FormatUint hex, while every other fid formatter strips whole leading zero bytes. Pad to even length so the output matches the canonical fid format and strict hex parsers accept it. |
||
|
Chris LuandGitHub
|
0ae1fdcad2 |
volume: reload a remote-tiered volume without re-entering the data lock (#10266)
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does not race the heartbeat's DataBackend read). But load() also runs it, and CommitCompact calls load() while already holding that lock, so reloading a remote-tiered volume during a compaction commit re-enters the non-reentrant lock and deadlocks. Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked; the public LoadRemoteFile keeps taking the lock for the live tier-upload handler that does not hold it. |
||
|
Chris LuandGitHub
|
254c2a1024 |
volume: clear remote flag when tiering a volume back to local (#10262)
VolumeTierMoveDatFromRemote downloads the .dat, trims the .vif, and swaps the data backend to the local file, but left hasRemoteFile set. The volume.tier.download command masks this by unmounting and remounting right after, which reloads the flag from the trimmed .vif — but in the window before the remount the in-memory flag is wrong: doDeleteRequest would skip appending the tombstone to the freshly local .dat, and the phantom-.dat guard stays disabled. Give SwapDataBackend a hasRemoteFile argument so the backend swap and the flag move together under one lock, and route both tier directions through it: the tier-down download passes false, LoadRemoteFile passes true. The flag can no longer disagree with the live backend. |
||
|
Chris LuandGitHub
|
2d2fdeac3d |
volume: keep tier-uploaded volume reporting to master after volume.tier.upload (#10259)
volume: keep tier-uploaded volume reporting to master A live volume.tier.upload removes the local .dat and swaps the data backend to remote, but v.hasRemoteFile was only ever set when a volume is loaded from disk, so the running volume kept it false. The phantom .dat guard then saw fileCount>0, !HasRemoteFile, and a missing .dat, and stopped reporting the volume to the master. The volume vanished from the topology even though the upload succeeded and the data was in cloud storage. Set hasRemoteFile in LoadRemoteFile, the single point where a volume's backend becomes remote, so it is true both on disk-scan load and after an in-process tier upload. Route the backend reassignment through SwapDataBackend so it happens under dataFileAccessLock, closing the old backend and never racing the heartbeat's concurrent DataBackend read. Make the field atomic since the heartbeat now reads it concurrently with the tier-upload handler that writes it. |
||
 qzhelloandGitHub
|
1d8a6e832c | fix(ec): detect truncated .ecx instead of treating it as clean EOF (#10217) | ||
|
Chris LuandGitHub
|
b4a99b996d |
feat(ec): EC bitrot CHECKSUM scrub on the Rust volume server (#10154)
* proto: add EC bitrot checksum messages + CHECKSUM scrub mode Mirror weed/pb/volume_server.proto byte-for-byte (field numbers + types) so the .ecsum sidecar payload is wire-identical across the Go and Rust binaries: EcBitrotProtection / EcShardChecksums / ChecksumAlgorithm, VolumeScrubMode.CHECKSUM=4, and VolumeEcShardsCopyRequest.copy_ecsum_file. No code uses them yet — the .ecsum format, producer, mount-load, copy, and scrub land in following commits. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): port the .ecsum bitrot checksum module ec_bitrot.rs mirrors weed/storage/erasure_coding/ec_bitrot.go: the .ecsum sidecar format (14-byte big-endian ECSU header + CRC32C over a prost-serialized EcBitrotProtection payload), the per-shard per-block CRC32C producer (ShardChecksumBuilder), save/load with payload self-integrity, manifest validation, status resolution, and verify_shard_file_blocks for the CHECKSUM scrub. A byte-exact test pins the serialized bytes against the Go reference's identical constant so a format drift in either binary fails loudly. Producer wiring (encode/vacuum), mount-load, copy, and the mode-4 dispatch land in following commits. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * test(ec): pin .ecsum sidecar bytes for cross-binary interop Deterministic EcBitrotProtection -> exact on-disk bytes, asserted against a canonical constant on BOTH sides (this test and ec_bitrot.rs), so a format drift in either binary fails its own suite rather than silently desyncing a Go-written .ecsum from a Rust-written one. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): write the .ecsum bitrot sidecar during EC encode write_ec_files now feeds each shard's bytes through a per-shard ShardChecksumBuilder as it writes them, then persists the generation-0 sidecar (<base>.ecsum) alongside the shards — mirroring weed's WriteEcFiles + SaveBitrotSidecar. Best-effort: a failed sidecar write leaves the generation unprotected rather than failing the encode. A test confirms the produced sidecar validates and its per-block CRCs match every on-disk shard. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): load the .ecsum at mount + EcVolume::checksum_scrub EcVolume now loads and validates its generation-0 .ecsum sidecar at mount, caching the parsed protection + BitrotStatus (Off/On/Invalid), and exposes bitrot_protection() mirroring Go's EcVolume.BitrotProtection(). checksum_scrub() verifies every locally-held shard's raw bytes against the sidecar block CRCs — the only path that exercises cold parity shards — reporting mismatched shards without mutating anything; a wholesale mismatch beyond parity is flagged as a suspect sidecar rather than mass shard corruption. Mirrors Go's ChecksumScrub. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(scrub): dispatch EC CHECKSUM (mode 4) to checksum_scrub Accept VolumeScrubMode.CHECKSUM=4 and route it to EcVolume::checksum_scrub, accumulating blocks scanned + mismatched shards into the scrub response, plus the CHECKSUM scrub-mode metric label. Read-only bitrot verification over local shards. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): copy the .ecsum sidecar during VolumeEcShardsCopy Honor copy_ecsum_file: when set, copy the generation-0 .ecsum alongside the shards so protection travels with them, mirroring Go's non-2PC copy path. Tolerant of a missing source (empty stream) — the 0-byte file is dropped so mount sees no sidecar (protection off) rather than a truncated/invalid one. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * feat(ec): remove the .ecsum sidecars when destroying an EC volume remove_ec_volume_files now clears <base>.ecsum (and any versioned .ecsum.v<N>) from the data and idx dirs, so a vid reuse can't load a stale sidecar. Mirrors Go's removeBitrotSidecars. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * style(ec): align bitrot comments and test setup for merge-cleanliness Match the shared bitrot code (write_ec_files, encode_one_batch, checksum_scrub, the encode sidecar test) to the canonical wording/layout so the volume-server Rust port stays line-aligned across trees, keeping periodic merges conflict-free. No behavior change. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris LuandGitHub
|
bea1357d38 |
ec: skip physically near-full disks when placing EC shards (#10167)
EC placement scored destinations purely by free EC shard slots (derived from maxVolumeCount) and shard counts, blind to real disk fullness — the same defect as volume balancing. A disk that is physically full but still shows free EC slots kept being chosen, and EC shard bytes are captured by statfs free space yet not by any slot accounting, so the slot math is exactly the metric that can't see EC fullness. Treat a disk at/above 90% physical usage as having zero free EC slots at snapshot-build time, so every existing freeSlots>0 placement predicate excludes it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology) via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard recovery can still complete onto fuller disks. |
||
|
Chris LuandGitHub
|
77bf2a3ab0 |
volume.balance: gate on real physical disk usage (fixes #10160) (#10162)
* shell: add volume.balance -byDiskUsage to balance by actual data The default balancer ranks servers by slot density, dividing used volumes by MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold, a physically near-full server looks nearly empty and gets picked as the move target, so balancing drains less-full servers onto an already-full one. -byDiskUsage ranks servers by the actual data they hold (sum of volume sizes) instead, so the fullest-by-data server is treated as full and balancing drains it. It assumes comparable disk sizes per disk type and still respects each server's free volume slots. Default behavior is unchanged. * plumb physical disk usage into topology, gate volume.balance on it Volume servers now report each disk's filesystem total/free bytes in the heartbeat, and the master stores them in DiskInfo. volume.balance uses them to skip any move target whose disk is already near full (-maxDiskUsagePercent, default 90), so an over-configured maxVolumeCount can no longer make a physically full server look empty and get drained onto. The gate judges each server against its own disk, so heterogeneous disk sizes are fine; servers that do not report bytes fall back to slot-only behavior. Rust seaweed-volume mirrors the heartbeat reporting. * admin: report real physical disk capacity when volume servers provide it The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit, which overstates it when maxVolumeCount is set higher than the disk holds. Prefer the filesystem capacity now reported per disk, falling back to the estimate for servers that do not report it. * worker: gate automatic balance on physical disk fullness too The maintenance balance worker selects the least slot-utilized server as the move destination, so an over-configured maxVolumeCount makes a physically full server look empty and get drained onto — the same defect as the shell command. Now that DiskInfo carries real disk bytes, skip any destination whose disk is at/above 90% used (per server, against its own disk); a full server can still be a source. When every candidate destination is full, create no tasks. Servers that do not report disk bytes are not gated. * balance: share the physical-disk-fullness gate between shell and worker The shell volume.balance command and the maintenance balance worker each grew their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull) and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer (DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the two balancers can't drift. * balance: harden the physical-disk gate Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity report clear previously stored bytes (0 means "not reported" for bytes, unlike maxVolumeCount), so a server that stops reporting falls back to slot-only instead of trusting stale capacity. In the worker, charge each planned move's bytes to its destination within a detection cycle so the gate sees a target fill up rather than only its heartbeat-time free space. Note the per-location capacity summing assumes one location per filesystem (the used ratio the gate relies on stays correct regardless; absolute capacity can over-report). |
||
|
Chris LuandGitHub
|
41d6c821ba |
feat(topology): report empty disks (per-disk type + capacity in heartbeat) (#10166)
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
|
||
|
Lisandro PinandGitHub
|
cac83bb4a8 |
Fix scrubbing of deleted needles on EC volumes. (#10130)
EC volumes do not propagate deletions to all shard indexes, so it is possible to run scrubbing on a volume where a deleted needle is still present in the index, or a needle deleted from the index is still present on the volume. On either scenario, scrubbing will fail due to size mismatch errors. This PR reworks the scrubbing logic so needle size mismatches are ignored in such scenarios. Scrubbing can still be forced to check deleted needles (f.ex. to discover index inconsistencies); this option will be exposed in RPCs and `weed shell` on a follow-up PR. |
||
|
Chris LuandGitHub
|
acbb6f7550 |
fix(scrub): don't flag offset-0 logical tombstones in volume scrub (#10148)
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes: scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead (want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and flagged the first live needle as overlapping. Skip offset-0 logical tombstones from both the size reconcile and the overlap check; they are still counted for the index-size check. Local deletes (offset != 0) are unaffected. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(scrub): mirror offset-0 logical tombstone handling into Rust Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips offset-0 logical tombstones from total_read, and check_index_file excludes them from the overlap check (still counted for the index-size check). Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris LuandGitHub
|
c9f2ef9ef7 |
fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub (#10147)
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks it, but appended every error unconditionally. A needle the .ecx still reports live while its reassembled on-disk header carries size 0 (delete state disagrees between index and header) is not corruption — the LOCAL twin of the #10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case; genuine (non-zero) size mismatches and CRC/tail errors are still reported. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo * fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch whose on-disk header size is 0 against a live index entry is a delete-state disagreement, not corruption. Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo |
||
|
Chris LuandGitHub
|
2efc0e1656 |
ec: recover EC shards whose .ecx index lives only on a peer server (#10108)
* ec: recover EC shards whose .ecx index lives only on a peer server A volume server that boots with EC shard files on disk but no .ecx index on any local disk cannot mount the shards, so the master never learns about them. ec.rebuild works off master-registered shards, so it sees the volume as short and gives up even though the shard data is intact. Add an operator-triggered recovery: VolumeEcShardsMount gains a recover_missing_index flag that makes the volume server fetch the missing .ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards. ec.rebuild runs this across the cluster before planning, so orphaned shards register and the rebuild sees the true shard set. .ecx is an immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal that differs across holders, so the recovered node adopts the source peer's deletion view, like a balanced or rebuilt shard does. * ec: mirror missing-index recovery into the Rust volume server Port the #10104 recovery to seaweed-volume so the Rust volume server self-heals the same layout: EC shards on disk with the .ecx index only on a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards to the store, recover_missing_ec_indexes (master LookupEcVolume + peer CopyFile fetch + mount) to the server, and the recover_missing_index flag on VolumeEcShardsMount. .ecx is the immutable encode-time index, identical on every holder. .ecj is a per-holder deletion journal, so the recovered node adopts the source peer's deletion view, matching the Go path. |
||
|
Chris LuandGitHub
|
bc257fe72e |
volume: detect phantom volumes held open as deleted FDs (#10011)
* volume: detect phantom volumes held open as deleted FDs
Add disk-file validation in heartbeat collection to prevent reporting
phantom volumes that exist in memory but are deleted from disk. This
unblocks re-replication when files are unlinked while the volume server
holds them open via file descriptors.
Cache disk checks per-volume with 30-second TTL to avoid syscall overhead.
Implement in both Go and Rust volume servers.
* volume: make last_disk_check_ns field public for heartbeat access
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes when size > 0
Skip phantom volume detection for zero-size volumes (e.g., test volumes).
Phantom volumes only occur when disk files are deleted while the process
holds them open via FDs - which requires the volume to have had actual data.
Test volumes with zero size should not trigger disk file existence checks.
* volume: only check for phantom volumes if file_count > 0
Use file_count as the indicator for whether a volume held actual data,
rather than volume size. Phantom volumes only occur when a volume that
had files is deleted while the process holds open file descriptors.
Test volumes with no file count won't trigger the phantom detection check.
* volume: stat the .dat with its extension when detecting phantom volumes
DataFileName()/IndexFileName() return the extensionless base path, so os.Stat
saw every volume's files as missing and dropped it from the heartbeat, leaving
the master with no locations and breaking deletes/lookups. Stat FileName(".dat")
instead, skip remote-tiered volumes whose .dat lives in cloud storage, and
re-check a missing file every heartbeat rather than caching the negative.
|
||
|
Chris LuandGitHub
|
5e8152b81c |
storage: register tier backends at the binary composition root (#9989)
The s3 and rclone tiered-storage backends were registered via blank imports in weed/storage (volume_tier.go and volume_info/volume_info.go). That forced every library consumer of weed/storage -- weed/shell, and through it external tools -- to link aws-sdk-go and, under the rclone build tag, the full rclone backend set and its cloud-storage SDKs, even though those consumers never tier volumes. Move the registrations into a new weed/storage/backend/all aggregator and blank-import it once from weed/command, the binary's composition root. The weed binary still registers both backends; weed/storage and its library consumers no longer pull the backend SDKs into their dependency graph. |
||
|
Chris Lu
|
33df4fe2c4 |
storage: nil-safe ReplicaPlacement.String()
Guard a nil receiver like TTL.String() already does, so formatting a zero-value VolumeInfo can't depend on fmt's panic recovery. |
||
|
Chris LuandGitHub
|
9c10d64ae9 |
shell: show remote storage name/key in volume.list output (#9987)
VolumeInfo.String() dropped RemoteStorageName/RemoteStorageKey, which are useful when debugging volume tiering. Append them when the volume is remote. |
||
|
Chris LuandGitHub
|
7e608c877a |
refactor(ec_balance): make the balance planner per-volume ratio-capable (#9960)
* refactor(ec_balance): make the balance planner per-volume ratio-capable Thread a per-volume EC ratio through the balance planner: Plan resolves each volume's data/parity from a new Options.VolumeRatio (falling back to the collection Ratio, then the build default, when it reports 0), and keys the global phase's ratio maps by volume instead of collection. The shell and worker balance paths build the per-volume lookup from each shard's heartbeat via the new ecbalancer.VolumeShardRatio. In OSS this is behavior-preserving: VolumeShardRatio returns 0 because the per-volume data_shards/parity_shards heartbeat fields are an enterprise feature, so every volume falls back to the collection ratio -- the existing standard-scheme behavior. The refactor keeps the shared planner in sync with the enterprise fork, which overrides VolumeShardRatio to classify and spread a mixed-ratio collection by each volume's own data/parity split. * perf(ec_balance): hoist the collection ratio out of the per-volume loop The collection ratio is constant for every volume in a collection, so resolve it once per collection instead of per volume; a custom Ratio func may do map lookups or locking. Addresses a review comment. |
||
|
Chris LuandGitHub
|
138220b961 |
fix(ec): recover EC shards with the volume's own ratio, not the build default (#9958)
* fix(ec): recover EC shards with the volume's own ratio, not the build default recoverOneRemoteEcShardInterval rebuilt a missing shard with a hardcoded 10+4 Reed-Solomon matrix (and counted sufficiency / iterated shards against the 10+4 constants). For a custom-ratio volume (e.g. 9+3) that reconstructs with the wrong matrix and corrupts the recovered bytes, and cachedLookupEcShardLocations could wrongly reject a degraded but recoverable custom-ratio read. Use the volume's own ECContext (loaded from its .vif) for the encoder, the shard-iteration bound, and the data-shard sufficiency checks. In OSS the ratio is always 10+4 so this is a no-op; it brings the Go volume server in line with the Rust one, which already reconstructs with the volume's ratio. * fix(ec): close data races in the EC read-recovery path Address review: the freshness check in cachedLookupEcShardLocations read ecVolume.ShardLocations / ShardLocationsRefreshTime without the lock while recover goroutines mutate them via forgetShardId -- snapshot both under ShardLocationsLock.RLock(). The recover goroutines also wrote the shared is_deleted return concurrently -- collect it via an atomic and fold it in after they join. Also size availableShards/missingShards by the volume's ECContext ratio rather than the 10+4 constants. |
||
|
Chris LuandGitHub
|
ef5fee6c28 |
fix(storage): delete/unmount every copy of a duplicate volume id (#9954)
* fix(storage): delete and unmount every copy of a duplicate volume id NewStore has no cross-disk duplicate guard (unlike the Rust volume server, which refuses to start in that state), so a stale twin of a volume id can mount on a second disk after a disk repair. DeleteVolume and UnmountVolume returned after the first matching disk, leaving the twin to survive and re-register as the volume's content. Walk every disk and act on all copies, emitting one heartbeat delta per copy. * fix(storage): surface partial delete/unmount failures across duplicate copies Address review: if removing one copy of a duplicate volume id fails with a real error (disk IO, permissions), the loop logged it and could still return success once another copy was removed -- leaving the stale copy to re-register, the exact divergence this guards against. DeleteVolume and UnmountVolume now accumulate such errors and return them (still attempting every disk), so a copy left behind is never reported as success. Add a DeleteVolume duplicate-copies regression test. |
||
|
Chris LuandGitHub
|
284796c7b6 |
fix(ec): fence stale-worker EC shard cleanup by encode generation (#9953)
* feat(ec): add encode_ts_ns to the EC task params, shard-unmount, and shard-delete RPCs The generation fence for stale EC-worker cleanup needs the encode generation on three messages: ErasureCodingTaskParams (admin issues it), VolumeEcShardsUnmountRequest, and VolumeEcShardsDeleteRequest (the worker carries it to the volume server). Additive fields only; 0 preserves the existing unfenced behavior. Mirror the two volume-server fields in the Rust volume server's proto copy. * feat(ec): issue the EC encode generation from the admin and carry it on the worker Stamp each EC proposal's encode_ts_ns from the admin's per-cycle DetectionSequence (a single-clock value) so generations are globally ordered even though detection runs on a rotating worker. The worker writes that generation into the distributed .vif and passes it on its shard unmount/delete RPCs; it falls back to a local timestamp for the .vif only on the unfenced legacy/shell path (keeping the read guard on). * fix(ec): fence the stale-worker EC shard unmount and teardown by generation A reaped-but-still-running EC worker's cleanupStaleEcShards issued a generation-blind unmount + full teardown that could unmount and then overwrite a newer run's live shards on a shared node. Both RPCs now carry the encode generation: the volume server unmounts/deletes a disk only when its .vif generation is strictly older than the request, and preserves a same-or-newer generation, a generation-0 (recovered or pre-upgrade) volume, and an unreadable .vif. Unload is per-disk, never node-wide. Request generation 0 keeps the blanket teardown for the shell pre-encode cleanup and pre-upgrade callers. Mirrored in the Rust volume server. * test(ec): cover the generation-fenced teardown and unmount End-to-end volume-server tests: a fenced FullTeardown wipes a strictly- older generation, preserves a newer one, preserves a generation-0 volume, and blanket-wipes on request generation 0; the gen-aware unmount preserves a same-or-newer mounted generation; and the .vif generation reader handles present/absent/no-config cases. * test(ec): pin the fenced .vif==teardown generation and the unreadable-.vif preserve A fenced run must stamp the admin generation verbatim into the .vif so it matches the generation sent on the teardown RPCs; add a regression test that sets the task generation and asserts the .vif carries it exactly. Also cover the present-but-unparseable .vif case (reads as generation 0, preserved) and correct the readEcGenerationTsNs docstring accordingly. * fix(ec): surface EC full-teardown filesystem errors in the Rust volume server remove_ec_volume_files(_full_teardown) discarded every fs::remove_file error, so a teardown that failed on permissions or a full disk still returned full_teardown_done=true and left stale artifacts to collide with the next encode. Return io::Result, ignore NotFound, propagate the first real error, and have the teardown RPC surface it -- matching the Go contract. The best-effort reconcile/load-cleanup callers keep ignoring it. * refactor(ec): reuse the EC volume lookup on unmount and short-circuit the gen read Address review: the Rust unmount fence reuses the ec_vol it already fetched instead of a second find_ec_volume; the Go .vif generation reader breaks out of the data/idx loop early when the two dirs are the same. |
||
|
Chris LuandGitHub
|
da243b9423 |
fix(ec): group orphan-source completeness by encode generation (topology encode_ts_ns) (#9952)
* feat(ec): carry the encode generation through the topology heartbeat Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and populate it from each EC volume's .vif identity. The volume server emits it on the full and incremental heartbeats; the master stores it on EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker layer can see which encode run produced each shard set. Field 14 avoids the enterprise fork's reserved 10-13. Mirror the proto field and both heartbeat emit sites in the Rust volume server. * fix(ec): group orphan-source shard completeness by encode generation countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two interrupted encode runs whose shard sets overlap unioned into a false-complete set -- triggering the orphaned-source delete while no single generation was actually complete. Group shards by encode_ts_ns and return the largest single generation's count, so the trigger fires only when one run holds the full set. Shards from pre-upgrade servers (encode_ts_ns==0) form their own bucket. The heartbeat carries one encode_ts_ns per (volume, disk), so this separates generations on different disks; same-disk mixing is prevented upstream by the pre-encode artifact wipe and the cross-run read guard. * fix(ec): guard against a nil Ec shard info entry in the generation count Defensive: a manually-constructed or corrupted topology could carry a nil entry in EcShardInfos. Skip it rather than dereference. * fix(ec): carry the encode generation on the EC shard unmount delta The mount delta sets EncodeTsNs; the unmount deletion delta left it 0. Populate it from the Ec volume before unloading so both incremental deltas are consistent (the Rust volume server already does this via its snapshot diff). |
||
|
Chris LuandGitHub
|
26754fca4d |
fix(ec): don't fabricate a stub .vif when mounting an EC volume (#9951)
When an EC volume's .vif was missing, NewEcVolume wrote a stub holding only the version. That stub implies the default 10+4 ratio with DatFileSize=0 and no encode identity, which the custom-ratio resolver and the startup credibility checks then read as an authoritative config -- masking the real ratio of a custom-ratio volume and defeating the byte-exact .vif gate. Mount with in-memory defaults instead and leave the real .vif to the encoder or a recovery tool. The Rust volume server already behaves this way. |
||
|
Chris LuandGitHub
|
94357ac6a9 |
[volume] preserve compression state during replication (#9946)
* preserve compression state during replication * explain why ParseUpload skips compression for replica writes * fix data race on err result in FetchAndWriteNeedle The local-write and replica-write goroutines all wrote the named err return under an unsynchronized err==nil check. Give each goroutine its own error slot and combine after wg.Wait(): local error wins, then the first replica failure. * skip redundant decompression of compressed needles during replication doUploadData decompressed a compressed input only to report the clear-data length on UploadResult.Size, which both replication callers discard. Skip the decompress when IsReplication. |
||
|
Chris LuandGitHub
|
1e858d8af0 |
fix(ec): make ec.decode write-path crash-safe and atomic (#9949)
* fix(ec): check decode .idx writes and fsync decoded .dat/.idx WriteIdxFileFromEcIndex silently dropped io.Copy and Write errors, so a short or failed write of the reconstructed .idx went unnoticed and the caller proceeded to delete the source EC shards. Propagate those errors. Also fsync the decoded .dat and .idx before returning, so the bytes are durable before the shards that produced them are removed cluster-wide. Mirror the .idx fsync into the Rust volume server (its .dat already syncs and its writes already propagate errors). * fix(ec): publish decoded .dat/.idx atomically via temp file and rename WriteDatFile and WriteIdxFileFromEcIndex wrote in place at the final name with O_TRUNC. A crash mid-write left a truncated .dat/.idx at the final name beside the still-present EC shards; on restart that partial file could be mounted as the live volume even though the shards held the real data. Write to a .tmp file, fsync it, then rename into place and fsync the directory, so the final name is only ever absent or complete. A failed decode removes its own temp file rather than leaking it. Add util.FsyncDir as the shared directory-fsync primitive and reuse the Rust volume server's fsync_dir for the mirrored change. * fix(ec): propagate .ecj read errors in the Rust decoder Path::exists returned false for any error (permission denied, transient IO), silently skipping the deletion journal and resurrecting deleted needles as live. Read the journal directly and treat only NotFound as absent, propagating other errors. The Go decoder already behaves this way (FileExists returns false only for IsNotExist, then the open surfaces other errors). * fix(ec): remove rename destination on Windows in the Rust decoder publish std::fs::rename does not replace an existing file on every Windows version. Remove the destination first under a Windows guard before the atomic publish rename, matching the compaction commit path. |
||
|
Chris LuandGitHub
|
4fb3e22a01 |
fix(tiering): never delete a shared remote object while replicas still reference it (#9942)
* tiering: stop a shared remote object being deleted while replicas still point at it A remote-tiered volume's .dat content lives only in one cloud object that all N replica .vif files point at. Deleting that object while destroying any one replica, or before a downloaded replica is durable, bricks the survivors. - volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so surviving replicas keep the shared object. Document why the alreadyPlaced anchor needs no replica sync (same-object replicas are byte-identical). - VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the containing directory, trims the .vif (fsynced) and swaps to the local DiskFile BEFORE deleting the remote object, on both the keep-remote and delete paths. Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote download leaves the replica served from local disk rather than the shared object, and a crash before delete merely leaks the object. - volume.tier.download keeps the shared object for every replica except the last, which deletes it. - s3 and rclone download paths fsync the .dat before close. * storage: swap the volume data backend under the data lock The tier-download swap closed v.DataBackend and assigned the new local DiskFile without holding dataFileAccessLock, racing concurrent reads/writes (use of a closed file / nil deref). Add an exported Volume.SwapDataBackend that performs the close-and-replace under the lock, and call it from the tier download. * server: skip directory fsync on Windows in the tier download path os.Open(dir).Sync() is unsupported on Windows and returns an error, which would fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on Windows, matching how the storage-side helper tolerates the unsupported case. * shell: make multi-replica tier.download resilient to already-local replicas If a multi-replica download is interrupted and retried, a replica made local in the prior attempt returns "already on local disk", which aborted the whole command and left the remaining remote replicas dangling. Treat that case as a skip-and-continue so a retry completes the rest. * server: assert downloaded .dat content, not just length, in the tier test A length-only check passes even if the bytes are corrupted; compare the full content of the local .dat against the original. |
||
|
Chris LuandGitHub
|
339a597e7e |
fix(vacuum): crash-safe compaction commit with a durable .cpc marker, fsync-before-rename, and a reload fence (#9944)
* storage: make vacuum/compaction commit crash-safe with a durable .cpc marker A crash mid-compaction-commit could lose or corrupt volume data. The two-rename commit (.cpd->.dat, .cpx->.idx) was not atomic, fsync results were discarded before renaming over a healthy .dat, a stale .ldb could poison the needle map, and a duplicate/late commit could delete the live .dat/.idx outright. Introduce a durable .cpc commit marker so the swap is atomic across a crash: - CommitCompact writes and fsyncs the .cpc marker after makeupDiff fsyncs the .cpd/.cpx, then runs applyCompactSwap: an existence-guarded rename of .cpd->.dat and .cpx->.idx, a directory fsync, removal of the stale .ldb/.rdb, and finally removal of the marker. - reconcileCompactState recovers an interrupted commit on load: roll forward (finish the renames) when the marker is present, roll back (delete the orphan .cpd/.cpx) when it is absent. It runs from a directory pre-pass keyed on .cpd/.cpc existence, since the per-volume loader is keyed on .idx/.vif and misses the marker-only and already-renamed-.idx states. - applyCompactSwap verifies BOTH .cpd and .cpx exist before touching the live files, so a stale-state commit (including the Windows RemoveAll-then-rename path) errors without deleting anything. - Error-check the fsyncs that gate the swap: the .cpd close-fsync and .cpx fsync in copyDataBasedOnIndexFile, the makeupDiff .idx fsync, and MemDb.SaveToIdx. - generateLevelDbFile rebuilds from offset 0 when the stored watermark sits past the end of the .idx, instead of replaying zero entries and poisoning the needle map. - removeVolumeFiles and cleanupCompact sweep the .cpc marker; cleanup refuses to unlink the temp files while a marker is present. Mirror the commit-marker, fsync-before-rename, guard, and load/reconcile logic in the Rust volume server. * storage: don't reconcile an already-loaded volume's compaction state on reload reconcileCompactStates runs in loadExistingVolumes, which is re-invoked at runtime on SIGHUP (Store.LoadNewVolumes). For a volume that is already loaded and mid-vacuum, its .cpd/.cpx are live temp files, not crash leftovers -- rolling them back would clobber the in-flight compaction (and remove a live .ldb out from under an open handle). Skip any vid already present in the volume map; genuine startup recovery runs before any volume is loaded, so the map is empty then. Mirrored in the Rust volume server. Also drop the .note keepVif change that crept into this branch; it belongs to the replica-copy/verify workstream and is restored to master's behavior here so the two changes don't collide. * storage: roll a compaction commit forward per-file, not all-or-nothing A crash after the .cpd->.dat rename but before .cpx->.idx leaves .cpd gone, .cpx and .cpc present, and a stale .idx. The roll-forward required BOTH temp files, so it skipped the swap and cleared the marker, pairing the fresh .dat with the stale .idx (index corruption). Finish whichever temp file remains: extract finishCompactSwap to rename .cpd->.dat and/or .cpx->.idx independently; applyCompactSwap keeps the both-present guard for the normal commit. Existence in the Rust mirror is checked robustly so a transient error never skips the swap. * seaweed-volume: propagate directory fsync failures on the compaction commit path fsync_dir dropped every sync_all error, so the commit could proceed with an undurable marker or rename and a later restart could recover the wrong generation. Return the error and check it at the commit call sites (marker write and the swap), matching the Go fsyncDir which already propagates. Directory fsync stays a no-op on Windows, where it is unsupported. * storage: overflow-safe stale-watermark check when rebuilding the leveldb index watermark*NeedleMapEntrySize can overflow uint64 for a corrupted watermark and wrap below the file size, defeating the stale-.ldb guard. Compare in entries (watermark > size/NeedleMapEntrySize) instead, which is equivalent and cannot overflow. LevelDb-backed needle map is Go-only; no Rust mirror. * storage: propagate idxFile.Close error when writing the compacted index SaveToIdx writes the .cpx that is renamed to .idx at commit; a discarded Close error (buffered data not flushed) could leave a partially-written index after a crash. Surface it in the same durability gate as the fsync. |
||
|
Chris LuandGitHub
|
c2591b4395 |
fix(replication): verify-before-destroy in VolumeCopy, check.disk, and over-replication trim (#9943)
* volume: verify before destroy in VolumeCopy and replication repair Four data-safety fixes around copy/repair paths that could destroy or resurrect data before verifying the source or survivors. (a) VolumeCopy no longer deletes a pre-existing local replica up front. The delete is deferred until ReadVolumeFileStatus on the source succeeds, so a transient source outage (or a retry after one) can no longer wipe a healthy destination replica. Gated on source readability only; size/count comparisons are intentionally not used because they invert legitimately after divergent vacuum/compaction. Mirrored in the Rust volume server. (b) volume.check.disk no longer resurrects vacuumed-deleted needles. A key present-and-live on the source but entirely absent on the target is ambiguous: it may be a genuine missing write, or a needle deleted on the target and then vacuumed (its index entry and any tombstone are gone). An individual needle AppendAtNs has no monotonic relation to a vacuum watermark, so the old cutoff heuristic could not tell them apart. Without positive proof the absence is a missing write, the safe default is to NOT push it back. Tradeoff: a real missing write may go unrepaired until a tombstone-aware path exists, but we never raise back deleted data. (c) Over-replication trim no longer resurrects needles or removes the wrong replica. The pre-delete sync now runs read-only (divergence check only) instead of writing the doomed replica's needles into the survivor. pickOneReplicaToDelete only ever removes the smallest of multiple healthy writable replicas; it refuses the trim when doing so would leave only read-only/integrity-flagged survivors, since file_count>0 alone cannot prove the survivor's .dat is readable. (d) Incomplete-volume (.note) cleanup keeps the shared .vif when an .ecx for the same vid coexists on the disk, so removing an interrupted regular copy cannot strip a coexisting EC volume's info file. VolumeCopy now surfaces .note write/remove errors instead of ignoring them. In the Rust volume server (where a persisting note is actually reachable) the .note check moves below the empty-stub sweep and EC validation, keeps the .vif on EC coexistence, and the mount path fails when a .note still persists. * shell: scope the over-replication writable-survivor guard to the trim path only The writable-survivor guard (never trim down to a read-only survivor) lived inside the shared pickOneReplicaToDelete, so it also gated the misplaced-volume relocation via pickOneMisplacedVolume -- a misplaced read-only volume (e.g. a full one) would silently stop being rebalanced. Extract pickSmallestReplica for the relocation path (which deletes-and-recreates and must act on read-only replicas), and keep the writable-survivor guard only in pickOneReplicaToDelete used by the over-replication trim. * seaweed-volume: recompute keep_vif after invalid-EC cleanup in the .note path keep_vif used the pre-validation ecx_exists snapshot, so when the EC-validation step above removed the invalid .ecx/shards, the .note cleanup still preserved a now-orphaned .vif. Re-check .ecx existence at cleanup time, matching the Go hasEcxFile re-check. * shell: keep placement when picking an over-replication victim to delete The trim picked the smallest writable replica without regard to placement, so it could delete the only replica in a required failure domain (e.g. with "100" and replicas dc1 + two in dc2, deleting dc1 leaves both survivors in dc2). Prefer a writable replica whose removal still satisfies placement, falling back to the smallest writable only when none does. |
||
|
Chris LuandGitHub
|
f724828bcb |
fix(ec): never delete recoverable EC shards on startup/reconcile (the non-empty-.dat sibling of the stub bug) (#9941)
* fix(ec): never delete recoverable shards on startup/reconcile (size-direction + byte-exact .dat)
EC startup validation and the cross-disk reconcile could delete the only
copy of distributed-EC shards whenever a non-empty .dat sat beside them.
This is the same data-loss class as the empty-.dat-stub fix, now for a
real (non-empty) stale or partial .dat.
validateEcVolume: the discriminating signal is the shard size relative to
the .dat's full encode, not the shard count.
- shards smaller than expected: an interrupted local encode left partial
shards and the .dat is the complete source -> reclaim the .dat.
- shards equal to expected: a valid (or still-distributing) EC volume ->
keep; the shards may be the only copy.
- shards larger than expected: the .dat is the stale/partial side (e.g. an
interrupted decode left a half-written .dat next to the real shards) ->
keep.
Previously any size mismatch, a low shard count beside a .dat, or a
transient stat error returned "delete", wiping sole-copy shards. Now every
ambiguity (size mismatch in either direction, inconsistent shard sizes,
transient I/O error, partial shard set) keeps the data; only a credible
full source .dat with no partial set to lose is reclaimed.
handleFoundEcxFile: a shard load failure (corrupt/locked .ecx, EMFILE
during a mass restart, transient I/O) no longer deletes the EC files when a
.dat exists -- it only unloads and keeps the files for retry. All deletion
authority now flows through validateEcVolume.
pruneIncompleteEcWithSiblingDat: count shards NODE-WIDE (a set split across
sibling disks summing to >= dataShards is independently recoverable and is
left alone), and require the sibling .dat to byte-exactly match the size
.vif recorded at encode time before deleting -- the prior "at least this
big, or bigger than a superblock" gate could trust a stale .dat and wipe
sole-copy shards. EC encode records the source size in .vif, so this gate
works for real volumes; older volumes without it fail safe (kept).
Rust volume server mirrors all of the above: size-direction + keep-on-
ambiguity in validate_ec_volume, keep-on-load-failure in
handle_found_ecx_file, and the node-wide + byte-exact gate in the prune.
The Rust validate/prune paths now resolve the data-shard count from the
volume's own .vif instead of hardcoding 10+4, so custom-ratio volumes are
not mis-sized and wrongly deleted on reboot.
Existing tests that encoded the old (unsafe) "delete on low count / size
mismatch" behavior are updated to the safe expectation, and new regression
tests cover the partial-decode-.dat-keeps-shards and transient-error-keeps
cases (Go and Rust); they fail on the pre-fix code.
* fix(ec): record DatFileSize in planted EC .vif for the prune test; trim comments
The multi-disk lifecycle e2e test planted a partial EC leftover with an
empty .vif, so the byte-exact prune gate (which a real encoded volume
satisfies via its recorded source size) kept it instead of cleaning up.
Record DatFileSize + the EC ratio in the planted .vif, matching production.
Also condense the verbose comments added in this change to the repo's
concise style.
|
||
|
Chris LuandGitHub
|
18cdb3819b |
fix(ec): crash-safe ecx-journal fold and shard rebuild (fsync before publish, no short-read-as-success) (#9938)
* fix(ec): make ecx-journal fold and shard rebuild crash-safe Two EC rebuild paths could silently lose or corrupt data: RebuildEcxFile folded the .ecj deletion journal into .ecx (in-place WriteAt tombstones) and then unlinked the journal without flushing the .ecx writes first. A crash could persist the unlink ahead of the tombstones, resurrecting deleted needles on the next load. It also read journal records with a bare n!=size break, so a torn tail silently dropped the remaining tombstones before the unlink. Now: read records with io.ReadFull (io.EOF ends cleanly, a torn tail aborts and leaves .ecj in place for retry), fsync .ecx before removing the journal. rebuildEcFiles treated a zero/short ReadAt as a clean end-of-input and discarded the read error, so a truncated or unreadable input shard produced truncated regenerated shards that were then published as restored redundancy; the regenerated shards were also never fsynced on the no-sidecar path. Now: derive the expected shard size from the present inputs up front (rejecting a divergent/zero-size input), drive the loop by that size, fail on any short read or short write, and fsync every regenerated shard before it is mounted/renamed. Rust volume server mirrors the rebuild fix: rebuild_ec_files now checks the read_at byte count (it previously discarded it, the same truncation bug). The Rust ecx fold already synced .ecx before removing the journal. Custom EC ratios are unaffected: the shard size derives from the input shards and the loop uses the .vif-resolved data/parity counts, never a hardcoded 10+4. * storage: close ecx journal files via defer in RebuildEcxFile Per review: a single deferred Close per file replaces the per-error-path manual closes, so new early returns cannot leak descriptors. The journal is still closed explicitly before its unlink since Windows cannot delete an open file; the deferred second Close is a harmless no-op. |
||
|
Chris LuandGitHub
|
34f9b91d69 |
fix(storage): never let an empty .dat delete healthy distributed EC shards (#9930)
* fix(storage): never let an empty .dat delete healthy distributed EC shards A leftover empty .dat stub (a phantom from the pre-fix loader; zero needles) next to a distributed EC volume's local shards made startup classify the volume as an interrupted local encode: validateEcVolume requires >= dataShards local shards when a .dat is present, fails with the 1-2 shards a distributed volume keeps per disk, and the cleanup deletes those shards -- the only copies of that part of the volume. Repeated across restart waves this destroys enough shards cluster-wide to make the volume unrecoverable. Go: - loadExistingVolume: hoist the empty-stub sweep above the EC presence checks. Previously the .vif-next-to-.ecx guard returned before the sweep ever ran, so exactly the dangerous layout (stub + .ecx + local shards) kept its stub and then lost its shards in loadAllEcShards. - validateEcVolume / checkDatFileExists: treat a .dat <= a superblock (zero needles) as absent. An empty .dat cannot be the encode source, so it must never gate shard deletion; this also covers stubs without a .vif, which the sweep cannot prove are EC leftovers. Rust mirror (seaweed-volume): the same gate in validate_ec_volume and check_dat_file_exists (the Rust sweep already ran before validation); the volume-load skip keeps a plain existence check so fresh, needle-less volumes still load. Regression tests in Go and Rust reproduce the production layout (a zero-byte .dat beside .ecx/.ecj and two shards of a 10+4 volume, with and without a .vif) and fail without the fix with the shards deleted. * fix(ec): gate source volume deletion on a recoverable shard set After EC encode, the shell command and the (plugin) worker task refused to delete the source volume unless every shard was present, and aborted otherwise -- leaving the source .dat next to live shards, exactly the mixed state the startup cleanup mishandles. Replace the full-set requirement with a recoverability gate shared by both callers (RequireRecoverableShardSet): deleting a non-empty source .dat requires at least dataShards distinct shards cluster-wide. Below that the source is kept and the encode fails as before. A degraded but recoverable set (>= dataShards, < total) now proceeds with a warning instead of aborting: the missing shards can be rebuilt from the survivors, while keeping the source would preserve the dangerous mixed state. Empty stub replicas are still swept unguarded (OnlyEmpty) -- an empty .dat has nothing to lose. dataShards/totalShards stay parameters so enterprise custom EC ratios share the helper verbatim. * test(ec): use recoverable shard verification gate |
||
|
Chris LuandGitHub
|
4f8af455bf |
feat(storage): sweep leftover empty EC .dat stubs on volume server startup (#9927)
* feat(storage): sweep leftover empty EC .dat stubs on volume server startup An EC volume keeps no local .dat. The pre-fix loader left empty 8-byte superblock .dat stubs next to EC metadata (one per lone .vif). Left in place each loads as a phantom empty volume, and the same vid's stub on two disks of one server blocks Rust startup via the duplicate-vid check in Store::add_location -- the prior fix stops creating new stubs but does not clean up existing ones. On startup, when a .dat is empty (<= a superblock, i.e. zero needles) and its .vif marks the volume erasure-coded, remove the stub (+ empty .idx) instead of loading it. The real data is in the EC shards, so the empty stub holds nothing to lose. Non-EC empty .dat files (e.g. freshly allocated volumes) are left alone. Done in both Rust (load_existing_volumes) and Go (loadExistingVolume), with regression tests that fail without the sweep. * refactor(storage): extract empty EC .dat stub sweep into its own function Move the startup stub-sweep into remove_empty_ec_dat_stub (Rust) and removeEmptyEcDatStub + vifIsEcVolume (Go) for clearer logic, and look up the .vif in both the data and idx directories (each read at most once) so a stub is still found when -dir.idx is configured. Adds direct tests for the idx-directory lookup on both engines. |
||
|
Chris LuandGitHub
|
79ac279fe1 |
fix(ec): don't mix EC shards from different encode runs (#9880)
* feat(ec): add encode_ts_ns to EC shard metadata and the shard read RPC EcShardConfig and VolumeEcShardReadRequest gain an int64 encode_ts_ns (encode time in unix nanos). It rides in .vif and the read request so a read can be scoped to the encode run that produced the index. * fix(ec): stamp each encode and reject cross-run shard reads Generate stamps EncodeTsNs into the volume's .vif. Reads carry it to the shard's owning volume (resolved together via FindEcVolumeWithShard, so a multi-disk server validates the disk that actually serves the bytes) and reject a shard from a different encode run, recovering from parity. A zero on either side (pre-upgrade volume) skips the guard. * fix(ec): stamp the encode identity on the worker-generated .vif The worker-local encode path now writes EncodeTsNs (and the resolved EC ratio) into the .vif, so the read guard is not silently off for volumes encoded by the maintenance worker. * fix(ec): wipe stale EC artifacts before re-encoding VolumeEcShardsGenerate evicts any in-memory EcVolume for the volume and removes its on-disk shard/index/sidecar files before writing fresh ones, so a retried encode never builds on a partial prior run and the unlink frees the inodes instead of leaving open fds serving old bytes. * fix(ec): unmount EC shards across all disks UnmountEcShards walked only the first disk holding the shard, leaving a duplicate copy mounted on a sibling disk (split-disk reconciled volumes) still serving and heartbeating. Traverse every disk and emit one deletion delta per disk. * fix(ec): delete orphan shards without a local .ecx deleteEcShardIdsForEachLocation gated shard-file removal on a local .ecx, so it could not clean an orphan .ecNN left by a failed copy on a disk with no index. Delete the requested shard files unconditionally; the index-file (.ecx/.ecj/.vif) routing stays gated as before. * fix(ec): clear stale EC shards cluster-wide before re-encoding ec.encode unmounts and deletes EC shards for the target volumes on every node before regenerating: fatal for the shards the topology reports (mounted leftovers), best-effort for the rest (a sweep that catches unmounted failed-copy orphans). A down node is a no-op. * fix(ec): don't nil EC fds on close so reads can't race eviction A reader resolves an EcVolume/shard under the lock then reads after it is released, so an eviction that nils ecxFile/ecdFile would race that read and panic. Close the fds without nilling the fields: the field is now write-once (no data race) and a concurrent read hits a closed fd, getting a clean error that the caller recovers from parity. * fix(ec): wipe stale EC artifacts on every disk and surface failures The pre-encode wipe only deleted beside the source volume, so a stale shard on a sibling disk survived and could be mounted against the new index at reconcile. Sweep every disk. Removal also ignored os.Remove errors, reporting a failed cleanup as success and letting a stale shard join the next generation; surface the first real failure (treating already-gone as success) from removeStaleEcArtifacts and the shard delete. * fix(ec): log when a local shard is skipped for a different encode run The cross-run guard returned errShardNotLocal, indistinguishable in logs from a genuinely-absent shard. Add a V(1) line naming both EncodeTsNs so operators can tell "wrong encode generation" from "shard not here". * fix(ec): surface metadata removal failures in the shard delete path deleteEcShardIdsForEachLocation still dropped os.Remove errors on the .ecx/.ecj/.vif/sidecar cleanup. A surviving stale .ecx is the orphan-index condition this path prevents, so route those through removeFileIfExists and return the first real failure instead of reporting cleanup as success. * fix(ec): fail orphan cleanup when a reachable node's delete fails The pre-encode orphan sweep swallowed every error for unreported (node, volume) pairs. That is only safe for an unreachable node, which cannot receive this encode's new generation. A reachable node whose delete genuinely failed (permission/IO) keeps an orphan shard that a later copy re-stamps with the new run's volume-level .vif identity, so the read guard would accept stale data. Surface those; stay best-effort only for unreachable nodes (gRPC Unavailable / no status). * fix(ec): guard ecjFile under its lock in the EC delete path EcVolume.Close nils ecjFile under ecjFileAccessLock; a delete that resolved its .ecx lookup before a concurrent eviction (the generate-time UnloadEcVolume) could then reach the journal append with a nil fd. Bail with a clear "volume closed" error under the lock instead. * fix(ec): reject an unstamped shard when the caller has an encode identity The read guard required both identities nonzero, so a current (stamped) caller accepted a holder with identity 0 and could be served a stale pre-upgrade shard. Reject when the caller is stamped and the holder differs (including unstamped); stay lenient only when the caller itself has no identity (pre-upgrade reader). A skipped shard recovers from parity. * fix(ec): full-teardown delete so cluster cleanup wipes a whole generation The pre-encode cluster sweep deleted only the listed canonical shards on remote nodes, leaving index/sidecar (and, on builds with versioned generations, those too) behind. Add a full_teardown flag to VolumeEcShardsDelete that evicts the volume and wipes every EC artifact for it on every disk via removeStaleEcArtifacts; the shell and worker pre-encode cleanup paths set it. Other delete callers (balance/decode/repair) are unchanged. * fix(ec): take ecjFileAccessLock before the nil-check in Sync and Close Sync and Close read ev.ecjFile before acquiring ecjFileAccessLock while Close nils it under the lock, a data race on the field. Take the lock first, then nil-check inside, in both. * fix(ec): acknowledge full_teardown so a pre-upgrade server can't fake success An old volume server silently ignores full_teardown and returns success for an ordinary delete, so the caller wrongly believes the generation was wiped and copies a fresh gen-0 onto an unwiped node. Echo full_teardown_done in the response; the worker destination cleanup fails when it is absent, and the shell cluster sweep fails for a reported (mounted) leftover while staying best-effort for an unreported node. encode_ts_ns stays an accepted transient (an old server just skips the new read guard, no regression). * fix(ec): fail the pre-encode sweep for any reachable node that can't ack teardown A reachable pre-upgrade server ignores full_teardown and returns success without wiping an orphan, which a later copy then folds into the new generation. Treat a missing full_teardown_done ack as fatal for every reachable node (best-effort only for a gRPC-unreachable one), not just for topology-reported pairs. * fix(ec): return the served shard identity and validate it client-side The encode identity was only enforced server-side, so a pre-upgrade server ignored the request field and served bytes unchecked. Echo the served shard's EncodeTsNs on every read response chunk and have the client reject a mismatch (including 0 from an old server), so the guard holds regardless of server version; a rejected read recovers from parity. * fix(ec): reject a short/empty remote shard read instead of serving zeros doReadRemoteEcShardInterval accepted an immediate EOF or a short stream and returned success with a partly zero-filled, unvalidated buffer (the server stamps the identity only on chunks that carry bytes). A non-deleted interval must arrive whole: require n == len(buf), exempting the is_deleted short-circuit (n=0), matching readLocalEcShardInterval's local check. A short read now fails so the caller recovers from parity. * test(ec): fake volume server echoes the full_teardown acknowledgement The worker now fails a teardown delete that isn't acknowledged (so a pre-upgrade server can't silently skip the wipe). The fake server's no-op VolumeEcShardsDelete returned an empty response, which the worker read as a skipped teardown and aborted the encode. Echo full_teardown_done. * feat(ec): mirror the encode-run identity guard + full_teardown into the Rust volume server The Go volume server stamps an encode-run identity (encode_ts_ns) into the .vif and rejects a read served from a shard of a different run; full_teardown wipes a whole generation and acknowledges it. The Rust volume server had none of it. Mirror the shared logic: load encode_ts_ns from the .vif onto the EcVolume, stamp it on every read response, and reject a request/response mismatch on both the server and the distributed-read client (recovering from parity); handle full_teardown by evicting the volume and wiping every EC artifact on each disk, echoing full_teardown_done so the caller can detect a server that ignored it. * fix(ec): remove a stale .vif on full teardown of a shard-only node A shard copy installs shards + .ecx before .vif, so an interrupted copy after a teardown could mount the new files under the previous run's identity / version / shard ratio / dat_file_size carried by the surviving .vif. Remove .vif during full teardown, gated on .idx absence so a source-volume holder keeps its live .vif. In Rust this lives in a teardown-only helper so the reconcile / load- fallback paths (which share the base removal) still preserve .vif. * fix(ec): treat a missing teardown ack as fatal, not as an unreachable node isNodeUnreachable returned true for any non-gRPC-status error, so a reachable pre-upgrade server's missing full_teardown_done ack (a plain error) was classified unreachable and the unreported pair was silently skipped. Classify only a real codes.Unavailable as unreachable, and wrap the missing ack in a sentinel the sweep treats as fatal regardless. A genuinely down node still surfaces as Unavailable from the RPC and stays best-effort. * fix(ec): reject a short shard read in the local EC needle reader read_ec_shard_needle ignored the byte count from shard.read_at and appended the whole pre-sized buffer, so a truncated shard's zero-filled tail passed the later length check and parsed as garbage. Require n == buf.len() per interval, erroring on a short read like the local interval reader already does. * fix(ec): probe reachability before skipping a node that returns Unavailable The pre-encode sweep skipped any node whose teardown delete returned codes.Unavailable, but a reachable volume server in maintenance mode also returns that code for the maintenance-gated delete, so its stale EC files were left behind on a node that can still receive the new generation. Confirm with a non-maintenance-gated empty-target Ping: skip only when the node fails the probe too (genuinely unreachable). * fix(ec): use try_exists for the teardown .vif .idx guard The teardown-only .vif removal gated on Path::exists(), which returns false on a permission/IO stat error, so a stat failure on a present .idx would read as a shard-only node and delete the live source volume's .vif. Gate on try_exists() == Ok(false) instead, preserving the sidecar on any stat error. * fix(ec): only skip a sweep node when a Ping confirms it is transport-down The pre-encode sweep skipped a node whenever its teardown delete and a liveness Ping both failed, but it treated ANY Ping error as down — an application-level Internal/ResourceExhausted, or Unimplemented from a pre-Ping server, left a reachable node's stale generation in place. Classify the Ping tri-state and skip only when it transport-fails with codes.Unavailable; a reachable or inconclusive node stays fatal. * fix(ec): exclude sweep-skipped nodes from the encode's rebalance The pre-encode sweep skips a genuinely-down node best-effort, but the rebalance then recollected the current topology — a node that recovered between the two could become a copy target and receive the new generation while still holding its stale, never-cleared shards. Have the sweep return the skipped set and exclude those nodes from the rebalance for this encode, so a node we could not clean cannot receive the new generation. Standalone ec.balance is unaffected. * fix(ec): re-sweep recovered nodes before generation so they aren't stranded A node skipped as down by the pre-encode sweep is excluded from the rebalance, but it can recover and become the generation host — mounting all shards locally, then being excluded from distribution. Union-only verification accepts all shards on one node and deletes the originals: a single point of failure. Re-sweep the skipped nodes just before generation; one whose teardown now succeeds leaves the skipped set and rebalances normally, while a node still down stays skipped. * fix(ec): abort the encode if a selected source is still skipped after re-sweep The re-sweep un-skips a recovered node, but the source was selected before it and a node can stay down through the re-sweep then recover just in time to be the generation host — mounting all shards locally while still excluded from the rebalance, which union-only verification accepts before deleting the originals. Abort the encode when a selected source remains skipped after the re-sweep. * fix(ec): batch delete returns retriable 503 when a volume became EC mid-batch If a volume is not EC at the batch-delete classification but is encoded to EC and its .dat deleted before the regular-volume mutation, the mutation returns an exact "not found" that the filer chunk-GC treats as completed, dropping the delete. Recheck EC presence under the mutation lock and return a retriable 503 with the "try again" token so the filer requeues it onto the EC path. * fix(ec): recheck EC state before the regular batch-delete mutation ec.encode mounts EC shards (copied from the .dat) before deleting the originals, so a volume can be EC while its .dat still exists. The batch delete only rechecked EC after a NotFound, so a successful regular-volume delete in that window wrote a tombstone to the soon-removed .dat — the delete was lost and the needle resurrected from the pre-tombstone shards. Recheck has_ec_volume under the write lock before delete_volume_needle and return a retriable 503 so the filer requeues onto the EC path. * fix(volume): make the metrics push test independent of test order test_push_metrics_once asserted the pushed body contains the request-counter family without ever touching the counter — a CounterVec with no children emits nothing, so the assertion only held when another test had already created a labelset in the shared registry. Create one in the test itself. |
||
 Bruce ZouandGitHub
|
1dd292fb84 | batch drain delta heartbeat messages (#9914) |