* feat(ec): carry the encode generation through the topology heartbeat
Add encode_ts_ns (field 14) to VolumeEcShardInformationMessage and
populate it from each EC volume's .vif identity. The volume server emits
it on the full and incremental heartbeats; the master stores it on
EcVolumeInfo and re-emits it via GetTopologyInfo, so the admin/worker
layer can see which encode run produced each shard set. Field 14 avoids
the enterprise fork's reserved 10-13. Mirror the proto field and both
heartbeat emit sites in the Rust volume server.
* fix(ec): group orphan-source shard completeness by encode generation
countExistingEcShardsForVolume ORed EcIndexBits across every disk, so two
interrupted encode runs whose shard sets overlap unioned into a
false-complete set -- triggering the orphaned-source delete while no
single generation was actually complete. Group shards by encode_ts_ns and
return the largest single generation's count, so the trigger fires only
when one run holds the full set. Shards from pre-upgrade servers
(encode_ts_ns==0) form their own bucket.
The heartbeat carries one encode_ts_ns per (volume, disk), so this
separates generations on different disks; same-disk mixing is prevented
upstream by the pre-encode artifact wipe and the cross-run read guard.
* fix(ec): guard against a nil Ec shard info entry in the generation count
Defensive: a manually-constructed or corrupted topology could carry a nil
entry in EcShardInfos. Skip it rather than dereference.
* fix(ec): carry the encode generation on the EC shard unmount delta
The mount delta sets EncodeTsNs; the unmount deletion delta left it 0.
Populate it from the Ec volume before unloading so both incremental
deltas are consistent (the Rust volume server already does this via its
snapshot diff).
* admin: report file and delete counts for EC volumes
The admin bucket size fix (#9058) left object counts at zero for
EC-encoded data because VolumeEcShardInformationMessage carried no file
count. Billing/monitoring dashboards therefore still under-report
objects once a bucket is EC-encoded.
Thread file_count and delete_count end-to-end:
- Add file_count/delete_count to VolumeEcShardInformationMessage (proto
fields 8 and 9) and regenerate master_pb.
- Compute them lazily on volume servers by walking the .ecx index once
per EcVolume, cache on the struct, and keep the cache in sync inside
DeleteNeedleFromEcx (distinguishing live vs already-tombstoned
entries so idempotent deletes do not drift the counts).
- Populate the new proto fields from EcVolume.ToVolumeEcShardInformationMessage
and carry them through the master-side EcVolumeInfo / topology sync.
- Aggregate in admin collectCollectionStats, deduping per volume id:
every node holding shards of an EC volume reports the same counts, so
summing across nodes would otherwise multiply the object count by the
number of shard holders.
Regression tests cover the initial .ecx walk, live/tombstoned delete
bookkeeping (including idempotent and missing-key cases), and the admin
dedup path for an EC volume reported by multiple nodes.
* ec: include .ecj journal in EcVolume delete count
The initial delete count only reflected .ecx tombstones, missing any
needle that was journaled in .ecj but not yet folded into .ecx — e.g.
on partial recovery. Expand initCountsLocked to take the union of
.ecx tombstones and .ecj journal entries, deduped by needle id, so:
- an id that is both tombstoned in .ecx and listed in .ecj counts once
- a duplicate .ecj entry counts once
- an .ecj id with a live .ecx entry is counted as deleted (not live)
- an .ecj id with no matching .ecx entry is still counted
Covered by TestEcVolumeFileAndDeleteCountEcjUnion.
* ec: report delete count authoritatively and tombstone once per delete
Address two issues with the previous EcVolume file/delete count work:
1. The delete count was computed lazily on first heartbeat and mixed
in a .ecj-union fallback to "recover" partial state. That diverged
from how regular volumes report counts (always live from the needle
map) and had drift cases when .ecj got reconciled. Replace with an
eager walk of .ecx at NewEcVolume time, maintained incrementally on
every DeleteNeedleFromEcx call. Semantics now match needle_map_metric:
FileCount is the total number of needles ever recorded in .ecx
(live + tombstoned), DeleteCount is the tombstones — so live =
FileCount - DeleteCount. Drop the .ecj-union logic entirely.
2. A single EC needle delete fanned out to every node holding a replica
of the primary data shard and called DeleteNeedleFromEcx on each,
which inflated the per-volume delete total by the replica factor.
Rewrite doDeleteNeedleFromRemoteEcShardServers to try replicas in
order and stop at the first success (one tombstone per delete), and
only fall back to other shards when the primary shard has no home
(ErrEcShardMissing sentinel), not on transient RPC errors.
Admin aggregation now folds EC counts correctly: FileCount is deduped
per volume id (every shard holder has an identical .ecx) and DeleteCount
is summed across nodes (each delete tombstones exactly one node). Live
object count = deduped FileCount - summed DeleteCount.
Tests updated to match the new semantics:
- EC volume counts seed FileCount as total .ecx entries (live +
tombstoned), DeleteCount as tombstones.
- DeleteNeedleFromEcx keeps FileCount constant and increments
DeleteCount only on live->tombstone transitions.
- Admin dedup test uses distinct per-node delete counts (5 + 3 + 2)
to prove they're summed, while FileCount=100 is applied once.
* ec: test fixture uses real vid; admin warns on skewed ec counts
- writeFixture now builds the .ecx/.ecj/.ec00/.vif filenames from the
actual vid passed in, instead of hardcoding "_1". The existing tests
all use vid=1 so behaviour is unchanged, but the helper no longer
silently diverges from its documented parameter.
- collectCollectionStats logs a glog warning when an EC volume's summed
delete count exceeds its deduped file count, surfacing the anomaly
(stale heartbeat, counter drift, etc.) instead of silently dropping
the volume from the object count.
* ec: derive file/delete counts from .ecx/.ecj file sizes
seedCountsFromEcx walked the full .ecx index at volume load, which is
wasted work: .ecx has fixed-size entries (NeedleMapEntrySize) and .ecj
has fixed-size deletion records (NeedleIdSize), so both counts are pure
file-size arithmetic.
fileCount = ecxFileSize / NeedleMapEntrySize
deleteCount = ecjFileSize / NeedleIdSize
Rip out the cached counters, countsLock, seedCountsFromEcx, and the
recordDelete helper. Track ecjFileSize directly on the EcVolume struct,
seed it from Stat() at load, and bump it on every successful .ecj append
inside DeleteNeedleFromEcx under ecjFileAccessLock. Skip the .ecj write
entirely when the needle is already tombstoned so the derived delete
count stays idempotent on repeat deletes. Heartbeats now compute counts
in O(1).
Tests updated: the initial fixture pre-populates .ecj with two ids to
verify the file-size derivation end-to-end, and the delete test keeps
its idempotent-re-delete / missing-needle invariants (unchanged
externally, now enforced by the early return rather than a cache guard).
* ec: sync Rust volume server with Go file/delete count semantics
Mirror the Go-side EC file/delete count work in the Rust volume server
so mixed Go/Rust clusters report consistent bucket object counts in
the admin dashboard.
- Add file_count (8) and delete_count (9) to the Rust copy of
VolumeEcShardInformationMessage (seaweed-volume/proto/master.proto).
- EcVolume gains ecj_file_size, seeded from the journal's metadata on
open and bumped inside journal_delete on every successful append.
- file_and_delete_count() returns counts derived in O(1) from
ecx_file_size / NEEDLE_MAP_ENTRY_SIZE and
ecj_file_size / NEEDLE_ID_SIZE, matching Go's FileAndDeleteCount.
- to_volume_ec_shard_information_messages populates the new proto
fields instead of defaulting them to zero.
- mark_needle_deleted_in_ecx now returns a DeleteOutcome enum
(NotFound / AlreadyDeleted / Tombstoned) so journal_delete can skip
both the .ecj append and the size bump when the needle is missing
or already tombstoned, keeping the derived delete_count idempotent
on repeat or no-op deletes.
- Rust's EcVolume::new no longer replays .ecj into .ecx on load. Go's
RebuildEcxFile is only called from specific decode/rebuild gRPC
handlers, not on volume open, and replaying on load was hiding the
deletion journal from the new file-size-derived delete counter.
rebuild_ecx_from_journal is kept as dead_code for future decode
paths that may want the same replay semantics.
Also clean up the Go FileAndDeleteCount to drop unnecessary runtime
guards against zero constants — NeedleMapEntrySize and NeedleIdSize
are compile-time non-zero.
test_ec_volume_journal updated to pre-populate the .ecx with the
needles it deletes, and extended to verify that repeat and
missing-id deletes do not drift the derived counts.
* ec: document enterprise-reserved proto field range on ec shard info
Both OSS master.proto copies now note that fields 10-19 are reserved
for future upstream additions while 20+ are owned by the enterprise
fork. Enterprise already pins data_shards/parity_shards at 20/21, so
keeping OSS additions inside 8-19 avoids wire-level collisions for
mixed deployments.
* ec(rust): resolve .ecx/.ecj helpers from ecx_actual_dir
ecx_file_name() and ecj_file_name() resolved from self.dir_idx, but
new() opens the actual files from ecx_actual_dir (which may fall back
to the data dir when the idx dir does not contain the index). After a
fallback, read_deleted_needles() and rebuild_ecx_from_journal() would
read/rebuild the wrong (nonexistent) path while heartbeats reported
counts from the file actually in use — silently dropping deletes.
Point idx_base_name() at ecx_actual_dir, which is initialized to
dir_idx and only diverges after a successful fallback, so every call
site agrees with the file new() has open. The pre-fallback call in
new() (line 142) still returns the dir_idx path because
ecx_actual_dir == dir_idx at that point.
Update the destroy() sweep to build the dir_idx cleanup paths
explicitly instead of leaning on the helpers, so post-fallback stale
files in the idx dir are still removed.
* ec: reset ecj size after rebuild; rollback ecx tombstone on ecj failure
Two EC delete-count correctness fixes applied symmetrically to Go and
Rust volume servers.
1. rebuild_ecx_from_journal (Rust) now sets ecj_file_size = 0 after
recreating the empty journal, matching the on-disk truth.
Previously the cached size still reflected the pre-rebuild journal
and file_and_delete_count() would keep reporting stale delete
counts. The Go side has no equivalent bug because RebuildEcxFile
runs in an offline helper that does not touch an EcVolume struct.
2. DeleteNeedleFromEcx / journal_delete used to tombstone the .ecx
entry before writing the .ecj record. If the .ecj append then
failed, the needle was permanently marked deleted but the
heartbeat-reported delete_count never advanced (it is derived from
.ecj file size), and a retry would see AlreadyDeleted and early-
return, leaving the drift permanent.
Both languages now capture the entry's file offset and original
size bytes during the mark step, attempt the .ecj append, and on
failure roll the .ecx tombstone back by writing the original size
bytes at the known offset. A rollback that itself errors is
logged (glog / tracing) but cannot re-sync the files — this is
the same failure mode a double disk error would produce, and is
unavoidable without a full on-disk transaction log.
Go: wrap MarkNeedleDeleted in a closure that captures the file
offset into an outer variable, then pass the offset + oldSize to the
new rollbackEcxTombstone helper on .ecj seek/write errors.
Rust: DeleteOutcome::Tombstoned now carries the size_offset and a
[u8; SIZE_SIZE] copy of the pre-tombstone size field. journal_delete
destructures on Tombstoned and calls restore_ecx_size on .ecj append
failure.
* test(ec): widen admin /health wait to 180s for cold CI
TestEcEndToEnd starts master, 14 volume servers, filer, 2 workers and
admin in sequence, then waited only 60s for admin's HTTP server to come
up. On cold GitHub runners the tail of the earlier subprocess startups
eats most of that budget and the wait occasionally times out (last hit
on run 24374773031). The local fast path is still ~20s total, so the
bump only extends the timeout ceiling, not the happy path.
* test(ec): fork volume servers in parallel in TestEcEndToEnd
startWeed is non-blocking (just cmd.Start()), so the per-process fork +
mkdir + log-file-open overhead for 14 volume servers was serialized for
no reason. On cold CI disks that overhead stacks up and eats into the
subsequent admin /health wait, which is how run 24374773031 flaked.
Wrap the volume-server loop in a sync.WaitGroup and guard runningCmds
with a mutex so concurrent appends are safe. startWeed still calls
t.Fatalf on failure, which is fine from a goroutine for a fatal test
abort; the fail-fast isn't something we rely on for precise ordering.
* ec: fsync ecx before ecj, truncate on failure, harden rebuild
Four correctness fixes covering both volume servers.
1. Durability ordering (Go + Rust). After marking the .ecx tombstone
we now fsync .ecx before touching .ecj, so a crash between the two
files cannot leave the journal with an entry for a needle whose
tombstone is still sitting in page cache. Once the fsync returns,
the tombstone is the source of truth: reads see "deleted",
delete_count may under-count by one (benign, idempotent retries)
but never over-reports. If the fsync itself fails we restore the
original size bytes and surface the error. The .ecj append is then
followed by its own Sync so the reported delete_count matches the
on-disk journal once the write returns.
2. .ecj truncation on append failure. write_all may have extended the
journal on disk before sync_all / Sync errors out, leaving the
cached ecj_file_size out of sync with the physical length and
drifting delete_count permanently after restart. Both languages
now capture the pre-append size, truncate the file back via
set_len / Truncate on any write or sync failure, and only then
restore the .ecx tombstone. Truncation errors are logged — same-fd
length resets cannot realistically fail — but cannot themselves
re-sync the files.
3. Atomic rebuild_ecx_from_journal (Rust, dead code today but wired
up on any future decode path). Previously a failed
mark_needle_deleted_in_ecx call was swallowed with `let _ = ...`
and the journal was still removed, silently losing tombstones.
We now bubble up any non-NotFound error, fsync .ecx after the
whole replay succeeds, and only then drop and recreate .ecj.
NotFound is still ignored (expected race between delete and encode).
4. Missing-.ecx hardening (Rust). mark_needle_deleted_in_ecx used to
return Ok(NotFound) when self.ecx_file was None, hiding a closed or
corrupt volume behind what looks like an idempotent no-op. It now
returns an io::Error carrying the volume id so callers (e.g.
journal_delete) fail loudly instead.
Existing Go and Rust EC test suites stay green.
* ec: make .ecx immutable at runtime; track deletes in memory + .ecj
Refactors both volume servers so the sealed sorted .ecx index is never
mutated during normal operation. Runtime deletes are committed to the
.ecj deletion journal and tracked in an in-memory deleted-needle set;
read-path lookups consult that set to mask out deleted ids on top of
the immutable .ecx record. Mirrors the intended design on both Go and
Rust sides.
EcVolume gains a `deletedNeedles` / `deleted_needles` set seeded from
.ecj in NewEcVolume / EcVolume::new. DeleteNeedleFromEcx /
journal_delete:
1. Looks the needle up read-only in .ecx.
2. Missing needle -> no-op.
3. Pre-existing .ecx tombstone (from a prior decode/rebuild) ->
mirror into the in-memory set, no .ecj append.
4. Otherwise append the id to .ecj, fsync, and only then publish
the id into the set. A partial write is truncated back to the
pre-append length so the on-disk journal and the in-memory set
cannot drift.
FindNeedleFromEcx / find_needle_from_ecx now return
TombstoneFileSize when the id is in the in-memory set, even though
the bytes on disk still show the original size.
FileAndDeleteCount:
fileCount = .ecx size / NeedleMapEntrySize (unchanged)
deleteCount = len(deletedNeedles) (was: .ecj size / NeedleIdSize)
The RebuildEcxFile / rebuild_ecx_from_journal decode-time helpers
still fold .ecj into .ecx — that is the one place tombstones land in
the physical index, and it runs offline on closed files. Rust's
rebuild helper now also clears the in-memory set when it succeeds.
Dead code removed on the Rust side: `DeleteOutcome`,
`mark_needle_deleted_in_ecx`, `restore_ecx_size`. Go drops the
runtime `rollbackEcxTombstone` path. Neither helper was needed once
.ecx stopped being a runtime mutation target.
TestEcVolumeSyncEnsuresDeletionsVisible (issue #7751) is rewritten
as TestEcVolumeDeleteDurableToJournal, which exercises the full
durability chain: delete -> .ecj fsync -> FindNeedleFromEcx masks
via the in-memory set -> raw .ecx bytes are *unchanged* -> Close +
RebuildEcxFile folds the journal into .ecx -> raw bytes now show
the tombstone, as CopyFile in the decode path expects.
* feat: pass expected_data_size from clients for size-aware assignment
Add expected_data_size field to AssignRequest (master proto) and
AssignVolumeRequest (filer proto) so clients can hint how large the
data will be. The master uses this instead of the 1MB default when
tracking pending volume sizes for weighted assignment.
- Add expected_data_size to master.proto AssignRequest
- Add expected_data_size to filer.proto AssignVolumeRequest
- Wire through filer AssignVolume handler
- Wire through HTTP submit handler (uses actual upload size)
- Add ExpectedDataSize to VolumeAssignRequest in operation package
- Topology.PickForWrite accepts optional expectedDataSize parameter
* fix: guard integer conversions in expected_data_size path
- common.go: clamp OriginalDataSize to non-negative before uint64 cast
- topology.go: cap expectedDataSize at math.MaxInt64 before int64 cast
* fix: parse dataSize hint in HTTP /dir/assign and test non-zero expectedDataSize
- HTTP /dir/assign now parses optional "dataSize" query parameter
and passes it to PickForWrite instead of hardcoded 0
- Add test assertion for PickForWrite with non-zero expectedDataSize
* dlm: replace modulo hashing with consistent hash ring
Introduce HashRing with virtual nodes (CRC32-based consistent hashing)
to replace the modulo-based hashKeyToServer. When a filer node is
removed, only keys that hashed to that node are remapped to the next
server on the ring, leaving all other mappings stable. This is the
foundation for backup replication — the successor on the ring is
always the natural takeover node.
* dlm: add Generation and IsBackup fields to Lock
Lock now carries IsBackup (whether this node holds the lock as a backup
replica) and Generation (a monotonic fencing token that increments on
each fresh acquisition, stays the same on renewal). Add helper methods:
AllLocks, PromoteLock, DemoteLock, InsertBackupLock, RemoveLock, GetLock.
* dlm: add ReplicateLock RPC and generation/is_backup proto fields
Add generation field to LockResponse for fencing tokens.
Add generation and is_backup fields to Lock message.
Add ReplicateLock RPC for primary-to-backup lock replication.
Add ReplicateLockRequest/ReplicateLockResponse messages.
* dlm: add async backup replication to DistributedLockManager
Route lock/unlock via consistent hash ring's GetPrimaryAndBackup().
After a successful lock or unlock on the primary, asynchronously
replicate the operation to the backup server via ReplicateFunc
callback. Single-server deployments skip replication.
* dlm: add ReplicateLock handler and backup-aware topology changes
Add ReplicateLock gRPC handler for primary-to-backup replication.
Revise OnDlmChangeSnapshot to handle three cases on topology change:
- Promote backup locks when this node becomes primary
- Demote primary locks when this node becomes backup
- Transfer locks when this node is neither primary nor backup
Wire up SetupDlmReplication during filer server initialization.
* dlm: expose generation fencing token in lock client
LiveLock now captures the generation from LockResponse and exposes it
via Generation() method. Consumers can use this as a fencing token to
detect stale lock holders.
* dlm: update empty folder cleaner to use consistent hash ring
Replace local modulo-based hashKeyToServer with LockRing.GetPrimary()
which uses the shared consistent hash ring for folder ownership.
* dlm: add unit tests for consistent hash ring
Test basic operations, consistency on server removal (only keys from
removed server move), backup-is-successor property (backup becomes
new primary when primary is removed), and key distribution balance.
* dlm: add integration tests for lock replication failure scenarios
Test cases:
- Primary crash with backup promotion (backup has valid token)
- Backup crash with primary continuing
- Both primary and backup crash (lock lost, re-acquirable)
- Rolling restart across all nodes
- Generation fencing token increments on new acquisition
- Replication failure (primary still works independently)
- Unlock replicates deletion to backup
- Lock survives server addition (topology change)
- Consistent hashing minimal disruption (only removed server's keys move)
* dlm: address PR review findings
1. Causal replication ordering: Add per-lock sequence number (Seq) that
increments on every mutation. Backup rejects incoming mutations with
seq <= current seq, preventing stale async replications from
overwriting newer state. Unlock replication also carries seq and is
rejected if stale.
2. Demote-after-handoff: OnDlmChangeSnapshot now transfers the lock to
the new primary first and only demotes to backup after a successful
TransferLocks RPC. If the transfer fails, the lock stays as primary
on this node.
3. SetSnapshot candidateServers leak: Replace the candidateServers map
entirely instead of appending, so removed servers don't linger.
4. TransferLocks preserves Generation and Seq: InsertLock now accepts
generation and seq parameters. After accepting a transferred lock,
the receiving node re-replicates to its backup.
5. Rolling restart test: Add re-replication step after promotion and
assert survivedCount > 0. Add TestDLM_StaleReplicationRejected.
6. Mixed-version upgrade note: Add comment on HashRing documenting that
all filer nodes must be upgraded together.
* dlm: serve renewals locally during transfer window on node join
When a new node joins and steals hash ranges from surviving nodes,
there's a window between ring update and lock transfer where the
client gets redirected to a node that doesn't have the lock yet.
Fix: if the ring says primary != self but we still hold the lock
locally (non-backup, matching token), serve the renewal/unlock here
rather than redirecting. The lock will be transferred by
OnDlmChangeSnapshot, and subsequent requests will go to the new
primary once the transfer completes.
Add tests:
- TestDLM_NodeDropAndJoin_OwnershipDisruption: measures disruption
when a node drops and a new one joins (14/100 surviving-node locks
disrupted, all handled by transfer logic)
- TestDLM_RenewalDuringTransferWindow: verifies renewal succeeds on
old primary during the transfer window
* dlm: master-managed lock ring with stabilization batching
The master now owns the lock ring membership. Instead of filers
independently reacting to individual ClusterNodeUpdate add/remove
events, the master:
1. Tracks filer membership in LockRingManager
2. Batches rapid changes with a 1-second stabilization timer
(e.g., a node drop + join within 1 second → single ring update)
3. Broadcasts the complete ring snapshot atomically via the new
LockRingUpdate message in KeepConnectedResponse
Filers receive the ring as a complete snapshot and apply it via
SetSnapshot, ensuring all filers converge to the same ring state
without intermediate churn.
This eliminates the double-churn problem where a rapid drop+join
would fire two separate ring mutations, each triggering lock
transfers and disrupting ownership on surviving nodes.
* dlm: track ring version, reject stale updates, remove dead code
SetSnapshot now takes a version parameter from the master. Stale
updates (version < current) are rejected, preventing reordered
messages from overwriting a newer ring state. Version 0 is always
accepted for bootstrap.
Remove AddServer/RemoveServer from LockRing — the ring is now
exclusively managed by the master via SetSnapshot. Remove the
candidateServers map that was only used by those methods.
* dlm: fix SelectLocks data race, advance generation on backup insert
- SelectLocks: change RLock to Lock since the function deletes map
entries, which is a write operation and causes a data race under RLock.
- InsertBackupLock: advance nextGeneration to at least the incoming
generation so that after failover promotion, new lock acquisitions
get a generation strictly greater than any replicated lock.
- Bump replication failure log from V(1) to Warningf for production
visibility.
* dlm: fix SetSnapshot race, test reliability, timer edge cases
- SetSnapshot: hold LockRing lock through both version update and
Ring.SetServers() so they're atomic. Prevents a concurrent caller
from seeing the new version but applying stale servers.
- Transfer window test: search for a key that actually moves primary
when filer4 joins, instead of relying on a fixed key that may not.
- renewLock redirect: pass the existing token to the new primary
instead of empty string, so redirected renewals work correctly.
- scheduleBroadcast: check timer.Stop() return value. If the timer
already fired, the callback picks up latest state.
- FlushPending: only broadcast if timer.Stop() returns true (timer
was still pending). If false, the callback is already running.
- Fix test comment: "idempotent" → "accepted, state-changing".
* dlm: use wall-clock nanoseconds for lock ring version
The lock ring version was an in-memory counter that reset to 0 on
master restart. A filer that had seen version 5 would reject version 1
from the restarted master.
Fix: use time.Now().UnixNano() as the version. This survives master
restarts without persistence — the restarted master produces a
version greater than any pre-restart value.
* dlm: treat expired lock owners as missing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: reject stale lock transfers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: order replication by generation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dlm: bootstrap lock ring on reconnect
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: auto-disable master vacuum when plugin vacuum worker is active
When a vacuum-capable plugin worker connects to the admin server, the
admin server calls DisableVacuum on the master to prevent the automatic
scheduled vacuum from conflicting with the plugin worker's vacuum. When
the worker disconnects, EnableVacuum is called to restore the default
behavior. A safety net in the topology refresh loop re-enables vacuum
if the admin server disconnects without cleanup.
* rename isAdminServerConnected to isAdminServerConnectedFunc
* add 5s timeout to DisableVacuum/EnableVacuum gRPC calls
Prevents the monitor goroutine from blocking indefinitely if the
master is unresponsive.
* track plugin ownership of vacuum disable to avoid overriding operator
- Add vacuumDisabledByPlugin flag to Topology, set when DisableVacuum
is called while admin server is connected (i.e., by plugin monitor)
- Safety net only re-enables vacuum when it was disabled by plugin,
not when an operator intentionally disabled it via shell command
- EnableVacuum clears the plugin flag
* extract syncVacuumState for testability, add fake toggler tests
Extract the single sync step into syncVacuumState() with a
vacuumToggler interface. Add TestSyncVacuumState with a fake
toggler that verifies disable/enable calls on state transitions.
* use atomic.Bool for isDisableVacuum and vacuumDisabledByPlugin
Both fields are written by gRPC handlers and read by the vacuum
goroutine, causing a data race. Use atomic.Bool with Store/Load
for thread-safe access.
* use explicit by_plugin field instead of connection heuristic
Add by_plugin bool to DisableVacuumRequest proto so the caller
declares intent explicitly. The admin server monitor sets it to
true; shell commands leave it false. This prevents an operator's
intentional disable from being auto-reversed by the safety net.
* use setter for admin server callback instead of function parameter
Move isAdminServerConnected from StartRefreshWritableVolumes
parameter to Topology.SetAdminServerConnectedFunc() setter.
Keeps the function signature stable and decouples the topology
layer from the admin server concept.
* suppress repeated log messages on persistent sync failures
Add retrying parameter to syncVacuumState so the initial
state transition is logged at V(0) but subsequent retries
of the same transition are silent until the call succeeds.
* clear plugin ownership flag on manual DisableVacuum
Prevents stale plugin flag from causing incorrect auto-enable
when an operator manually disables vacuum after a plugin had
previously disabled it.
* add by_plugin to EnableVacuumRequest for symmetric ownership tracking
Plugin-driven EnableVacuum now only re-enables if the plugin was
the one that disabled it. If an operator manually disabled vacuum
after the plugin, the plugin's EnableVacuum is a no-op. This
prevents the plugin monitor from overriding operator intent on
worker disconnect.
* use cancellable context for monitorVacuumWorker goroutine
Replace context.Background() with a cancellable context stored
as bgCancel on AdminServer. Shutdown() calls bgCancel() so
monitorVacuumWorker exits cleanly via ctx.Done().
* track operator and plugin vacuum disables independently
Replace single isDisableVacuum flag with two independent flags:
vacuumDisabledByOperator and vacuumDisabledByPlugin. Each caller
only flips its own flag. The effective disabled state is the OR
of both. This prevents a plugin connect/disconnect cycle from
overriding an operator's manual disable, and vice versa.
* fix safety net to clear plugin flag, not operator flag
The safety net should call EnableVacuumByPlugin() to clear only
the plugin disable flag when the admin server disconnects. The
previous call to EnableVacuum() incorrectly cleared the operator
flag instead.
* admin: seed admin_script plugin config from master maintenance scripts
When the admin server starts, fetch the maintenance scripts configuration
from the master via GetMasterConfiguration. If the admin_script plugin
worker does not already have a saved config, use the master's scripts as
the default value. This enables seamless migration from master.toml
[master.maintenance] to the admin script plugin worker.
Changes:
- Add maintenance_scripts and maintenance_sleep_minutes fields to
GetMasterConfigurationResponse in master.proto
- Populate the new fields from viper config in master_grpc_server.go
- On admin server startup, fetch the master config and seed the
admin_script plugin config if no config exists yet
- Strip lock/unlock commands from the master scripts since the admin
script worker handles locking automatically
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review comments on admin_script seeding
- Replace TOCTOU race (separate Load+Save) with atomic
SaveJobTypeConfigIfNotExists on ConfigStore and Plugin
- Replace ineffective polling loop with single GetMaster call using
30s context timeout, since GetMaster respects context cancellation
- Add unit tests for SaveJobTypeConfigIfNotExists (in-memory + on-disk)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: apply maintenance script defaults in gRPC handler
The gRPC handler for GetMasterConfiguration read maintenance scripts
from viper without calling SetDefault, relying on startAdminScripts
having run first. If the admin server calls GetMasterConfiguration
before startAdminScripts sets the defaults, viper returns empty
strings and the seeding is silently skipped.
Apply SetDefault in the gRPC handler itself so it is self-contained.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Revert "fix: apply maintenance script defaults in gRPC handler"
This reverts commit 068a506330.
* fix: use atomic save in ensureJobTypeConfigFromDescriptor
ensureJobTypeConfigFromDescriptor used a separate Load + Save, racing
with seedAdminScriptFromMaster. If the descriptor defaults (empty
script) were saved first, SaveJobTypeConfigIfNotExists in the seeding
goroutine would see an existing config and skip, losing the master's
maintenance scripts.
Switch to SaveJobTypeConfigIfNotExists so both paths are atomic. Whichever
wins, the other is a safe no-op.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: fetch master scripts inline during config bootstrap, not in goroutine
Replace the seedAdminScriptFromMaster goroutine with a
ConfigDefaultsProvider callback. When the plugin bootstraps
admin_script defaults from the worker descriptor, it calls the
provider which fetches maintenance scripts from the master
synchronously. This eliminates the race between the seeding
goroutine and the descriptor-based config bootstrap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* skip commented lock unlock
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
* reduce grpc calls
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add volume dir tags to topology
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add preferred tag config for EC
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Prioritize EC destinations by tags
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add EC placement planner tag tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor EC placement tests to reuse buildActiveTopology
Remove buildActiveTopologyWithDiskTags helper function and consolidate
tag setup inline in test cases. Tests now use UpdateTopology to apply
tags after topology creation, reusing the existing buildActiveTopology
function rather than duplicating its logic.
All tag scenario tests pass:
- TestECPlacementPlannerPrefersTaggedDisks
- TestECPlacementPlannerFallsBackWhenTagsInsufficient
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Consolidate normalizeTagList into shared util package
Extract normalizeTagList from three locations (volume.go,
detection.go, erasure_coding_handler.go) into new weed/util/tag.go
as exported NormalizeTagList function. Replace all duplicate
implementations with imports and calls to util.NormalizeTagList.
This improves code reuse and maintainability by centralizing
tag normalization logic.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add PreferredTags to EC config persistence
Add preferred_tags field to ErasureCodingTaskConfig protobuf with field
number 5. Update GetConfigSpec to include preferred_tags field in the
UI configuration schema. Add PreferredTags to ToTaskPolicy to serialize
config to protobuf. Add PreferredTags to FromTaskPolicy to deserialize
from protobuf with defensive copy to prevent external mutation.
This allows EC preferred tags to be persisted and restored across
worker restarts.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add defensive copy for Tags slice in DiskLocation
Copy the incoming tags slice in NewDiskLocation instead of storing
by reference. This prevents external callers from mutating the
DiskLocation.Tags slice after construction, improving encapsulation
and preventing unexpected changes to disk metadata.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add doc comment to buildCandidateSets method
Document the tiered candidate selection and fallback behavior. Explain
that for a planner with preferredTags, it accumulates disks matching
each tag in order into progressively larger tiers, emits a candidate
set once a tier reaches shardsNeeded, and finally falls back to the
full candidates set if preferred-tag tiers are insufficient.
This clarifies the intended semantics for future maintainers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply final PR review fixes
1. Update parseVolumeTags to replicate single tag entry to all folders
instead of leaving some folders with nil tags. This prevents nil
pointer dereferences when processing folders without explicit tags.
2. Add defensive copy in ToTaskPolicy for PreferredTags slice to match
the pattern used in FromTaskPolicy, preventing external mutation of
the returned TaskPolicy.
3. Add clarifying comment in buildCandidateSets explaining that the
shardsNeeded <= 0 branch is a defensive check for direct callers,
since selectDestinations guarantees shardsNeeded > 0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix nil pointer dereference in parseVolumeTags
Ensure all folder tags are initialized to either normalized tags or
empty slices, not nil. When multiple tag entries are provided and there
are more folders than entries, remaining folders now get empty slices
instead of nil, preventing nil pointer dereference in downstream code.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix NormalizeTagList to return empty slice instead of nil
Change NormalizeTagList to always return a non-nil slice. When all tags
are empty or whitespace after normalization, return an empty slice
instead of nil. This prevents nil pointer dereferences in downstream
code that expects a valid (possibly empty) slice.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add nil safety check for v.tags pointer
Add a safety check to handle the case where v.tags might be nil,
preventing a nil pointer dereference. If v.tags is nil, use an empty
string instead. This is defensive programming to prevent panics in
edge cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add volume.tags flag to weed server and weed mini commands
Add the volume.tags CLI option to both the 'weed server' and 'weed mini'
commands. This allows users to specify disk tags when running the
combined server modes, just like they can with 'weed volume'.
The flag uses the same format and description as the volume command:
comma-separated tag groups per data dir with ':' separators
(e.g. fast:ssd,archive).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This PR implements logic load/save persistent state information for storages
associated with volume servers, and reporting state changes back to masters
via heartbeat messages.
More work ensues!
See https://github.com/seaweedfs/seaweedfs/issues/7977 for details.
* proto: add RaftLeadershipTransfer RPC for forced leader change
Add new gRPC RPC and messages for leadership transfer:
- RaftLeadershipTransferRequest: optional target_id and target_address
- RaftLeadershipTransferResponse: previous_leader and new_leader
This enables graceful leadership transfer before master maintenance,
reducing errors in filers during planned maintenance windows.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7527
* proto: regenerate Go files for RaftLeadershipTransfer
Generated from master.proto changes.
* master: implement RaftLeadershipTransfer gRPC handler
Add gRPC handler for leadership transfer with support for:
- Transfer to any eligible follower (when target_id is empty)
- Transfer to a specific server (when target_id and target_address are provided)
Uses hashicorp/raft LeadershipTransfer() and LeadershipTransferToServer() APIs.
Returns the previous and new leader in the response.
* shell: add cluster.raft.leader.transfer command
Add weed shell command for graceful leadership transfer:
- Displays current cluster status before transfer
- Supports auto-selection of target (any eligible follower)
- Supports targeted transfer with -id and -address flags
- Provides clear feedback on success/failure with troubleshooting tips
Usage:
cluster.raft.leader.transfer
cluster.raft.leader.transfer -id <server_id> -address <grpc_address>
* master: add unit tests for raft gRPC handlers
Add tests covering:
- RaftLeadershipTransfer with no raft initialized
- RaftLeadershipTransfer with target_id but no address
- RaftListClusterServers with no raft initialized
- RaftAddServer with no raft initialized
- RaftRemoveServer with no raft initialized
These tests verify error handling when raft is not configured.
* shell: add tests for cluster.raft.leader.transfer command
Add tests covering:
- Command name and help text validation
- HasTag returns false for ResourceHeavy
- Validation of -id without -address
- Argument parsing with unknown flags
* master: clarify that leadership transfer requires -raftHashicorp
The default raft implementation (seaweedfs/raft, a goraft fork) does not
support graceful leadership transfer. This feature is only available when
using hashicorp raft (-raftHashicorp=true).
Update error messages and help text to make this requirement clear:
- gRPC handler returns specific error for goraft users
- Shell command help text notes the requirement
- Added test for goraft case
* test: use strings.Contains instead of custom helper
Replace custom contains/containsHelper functions with the standard
library strings.Contains for better maintainability.
* shell: return flag parsing errors instead of swallowing them
- Return the error from flag.Parse() instead of returning nil
- Update test to explicitly assert error for unknown flags
* test: document integration test scenarios for Raft leadership transfer
Add comments explaining:
- Why these unit tests only cover 'Raft not initialized' scenarios
- What integration tests should cover (with multi-master cluster)
- hashicorp/raft uses concrete types that cannot be easily mocked
* fix: address reviewer feedback on tests and leader routing
- Remove misleading tests that couldn't properly validate their
documented behavior without a real Raft cluster:
- TestRaftLeadershipTransfer_GoraftNotSupported
- TestRaftLeadershipTransfer_ValidationTargetIdWithoutAddress
- Change WithClient(false) to WithClient(true) for RaftLeadershipTransfer
RPC to ensure the request is routed to the current leader
* Improve cluster.raft.transferLeader command
- Rename command from cluster.raft.leader.transfer to cluster.raft.transferLeader
- Add symmetric validation: -id and -address must be specified together
- Handle case where same leader is re-elected after transfer
- Add test for -address without -id validation
- Add docker compose file for 5-master raft cluster testing
* pb: add id field to Heartbeat message for stable volume server identification
This adds an 'id' field to the Heartbeat protobuf message that allows
volume servers to identify themselves independently of their IP:port address.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* storage: add Id field to Store struct
Add Id field to Store struct and include it in CollectHeartbeat().
The Id field provides a stable volume server identity independent of IP:port.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* topology: support id-based DataNode identification
Update GetOrCreateDataNode to accept an id parameter for stable node
identification. When id is provided, the DataNode can maintain its identity
even when its IP address changes (e.g., in Kubernetes pod reschedules).
For backward compatibility:
- If id is provided, use it as the node ID
- If id is empty, fall back to ip:port
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* volume: add -id flag for stable volume server identity
Add -id command line flag to volume server that allows specifying a stable
identifier independent of the IP address. This is useful for Kubernetes
deployments with hostPath volumes where pods can be rescheduled to different
nodes while the persisted data remains on the original node.
Usage: weed volume -id=node-1 -ip=10.0.0.1 ...
If -id is not specified, it defaults to ip:port for backward compatibility.
Fixes https://github.com/seaweedfs/seaweedfs/issues/7487
* server: add -volume.id flag to weed server command
Support the -volume.id flag in the all-in-one 'weed server' command,
consistent with the standalone 'weed volume' command.
Usage: weed server -volume.id=node-1 ...
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* topology: add test for id-based DataNode identification
Test the key scenarios:
1. Create DataNode with explicit id
2. Same id with different IP returns same DataNode (K8s reschedule)
3. IP/PublicUrl are updated when node reconnects with new address
4. Different id creates new DataNode
5. Empty id falls back to ip:port (backward compatibility)
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* pb: add address field to DataNodeInfo for proper node addressing
Previously, DataNodeInfo.Id was used as the node address, which worked
when Id was always ip:port. Now that Id can be an explicit string,
we need a separate Address field for connection purposes.
Changes:
- Add 'address' field to DataNodeInfo protobuf message
- Update ToDataNodeInfo() to populate the address field
- Update NewServerAddressFromDataNode() to use Address (with Id fallback)
- Fix LookupEcVolume to use dn.Url() instead of dn.Id()
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* fix: trim whitespace from volume server id and fix test
- Trim whitespace from -id flag to treat ' ' as empty
- Fix store_load_balancing_test.go to include id parameter in NewStore call
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* refactor: extract GetVolumeServerId to util package
Move the volume server ID determination logic to a shared utility function
to avoid code duplication between volume.go and rack.go.
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* fix: improve transition logic for legacy nodes
- Use exact ip:port match instead of net.SplitHostPort heuristic
- Update GrpcPort and PublicUrl during transition for consistency
- Remove unused net import
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
* fix: add id normalization and address change logging
- Normalize id parameter at function boundary (trim whitespace)
- Log when DataNode IP:Port changes (helps debug K8s pod rescheduling)
Ref: https://github.com/seaweedfs/seaweedfs/issues/7487
2025-12-02 22:08:11 -08:00
Chris LuGitHubCopilotgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* show ec shards on dashboard, show max in its own column
* master collect shard size info
* master send shard size via VolumeList
* change to more efficient shard sizes slice
* include ec shard sizes into volume server info
* Eliminated Redundant gRPC Calls
* much more efficient
* Efficient Counting: bits.OnesCount32() uses CPU-optimized instructions to count set bits in O(1)
* avoid extra volume list call
* simplify
* preserve existing shard sizes
* avoid hard coded value
* Update weed/storage/erasure_coding/ec_volume_info.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/admin/dash/volume_management.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update ec_volume_info.go
* address comments
* avoid duplicated functions
* Update weed/admin/dash/volume_management.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* simplify
* refactoring
* fix compilation
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-02 02:16:49 -07:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* initial design
* added simulation as tests
* reorganized the codebase to move the simulation framework and tests into their own dedicated package
* integration test. ec worker task
* remove "enhanced" reference
* start master, volume servers, filer
Current Status
✅ Master: Healthy and running (port 9333)
✅ Filer: Healthy and running (port 8888)
✅ Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready
* generate write load
* tasks are assigned
* admin start wtih grpc port. worker has its own working directory
* Update .gitignore
* working worker and admin. Task detection is not working yet.
* compiles, detection uses volumeSizeLimitMB from master
* compiles
* worker retries connecting to admin
* build and restart
* rendering pending tasks
* skip task ID column
* sticky worker id
* test canScheduleTaskNow
* worker reconnect to admin
* clean up logs
* worker register itself first
* worker can run ec work and report status
but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.
* move ec task logic
* listing ec shards
* local copy, ec. Need to distribute.
* ec is mostly working now
* distribution of ec shards needs improvement
* need configuration to enable ec
* show ec volumes
* interval field UI component
* rename
* integration test with vauuming
* garbage percentage threshold
* fix warning
* display ec shard sizes
* fix ec volumes list
* Update ui.go
* show default values
* ensure correct default value
* MaintenanceConfig use ConfigField
* use schema defined defaults
* config
* reduce duplication
* refactor to use BaseUIProvider
* each task register its schema
* checkECEncodingCandidate use ecDetector
* use vacuumDetector
* use volumeSizeLimitMB
* remove
remove
* remove unused
* refactor
* use new framework
* remove v2 reference
* refactor
* left menu can scroll now
* The maintenance manager was not being initialized when no data directory was configured for persistent storage.
* saving config
* Update task_config_schema_templ.go
* enable/disable tasks
* protobuf encoded task configurations
* fix system settings
* use ui component
* remove logs
* interface{} Reduction
* reduce interface{}
* reduce interface{}
* avoid from/to map
* reduce interface{}
* refactor
* keep it DRY
* added logging
* debug messages
* debug level
* debug
* show the log caller line
* use configured task policy
* log level
* handle admin heartbeat response
* Update worker.go
* fix EC rack and dc count
* Report task status to admin server
* fix task logging, simplify interface checking, use erasure_coding constants
* factor in empty volume server during task planning
* volume.list adds disk id
* track disk id also
* fix locking scheduled and manual scanning
* add active topology
* simplify task detector
* ec task completed, but shards are not showing up
* implement ec in ec_typed.go
* adjust log level
* dedup
* implementing ec copying shards and only ecx files
* use disk id when distributing ec shards
🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned
* Delete original volume from all locations
* clean up existing shard locations
* local encoding and distributing
* Update docker/admin_integration/EC-TESTING-README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* check volume id range
* simplify
* fix tests
* fix types
* clean up logs and tests
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
the calculation of `unclaimedSpaces` only needs to subtract `unusedSpace` when `preallocate` is not enabled.
Signed-off-by: LHHDZ <shichanglin5@qq.com>
Chris LuandGitHub
Lisandro PinandGitHub
chrislu
LHHDZandGitHub
Konstantin LebedevandGitHub
Guo LeiandGitHub
guol-fnstandguol-fnst
chrislu